Fabio routing to wrong back end

We are experiencing multiple instances of Fabio routing requests to the wrong backend service. Once it starts happening it persists, typically until some change is made to the routing table by a service restart or similar. It's quite catastrophic because once it starts happening it's as if a whole bunch of URL/endpoints just disappear and start returning 404 not found because the requests are landing on a backend that doesn't serve them.
We have traced using tcpdump the requests coming in and out of Fabio and have proved beyond doubt that it is making the wrong routing decisions.
The attached dump shows a request coming into fabio for d2mx-prod-admin.dionglobal.com and then leaving fabio destined for a service on port 30398 but the routing table/consul indicates that the service on 30398 is analyser.sequoiadirect.com.au.

tcpro-tcpro-tcpro       passing    <IP_ADDRESS>     30398          [urlprefix-analyser.sequoiadirect.com.au:9999/, urlprefix-analyser.boursedata.com.au:9999/, urlprefix                                                                                -analyser.d2mx.com.au:9999/
We were initially running on fabio-1.5.0-go1.8.3-linux_amd64 but moved to fabio-1.5.4-go1.9.2-linux_amd64 to see if fixed the problem, but it hasnt.

Thank's Frank. What's the best way or format to capture the routing table? The table doesn't actually change very often, and AFAIK, the table as it is now will be the same as when the failure starts to occur.

@craigday try curl localhost:9998/api/routes?raw or set log.routes.format = all

@craigday Is this still an issue?

Yep. Have sent you the routing table just now.

This hit us again this morning. Is there any further info we can provide? Can you enumerate any possible theories or code paths that might be suspect, so we can help with the analysis?

Looking.

@craigday I'm awake now and DM'ing you on Twitter.

Got hit by this as well running fabio 1.5.7
Requests to all our services started to get routed intermittent to our hash-ui service. Causing lots of weirdness. Stopping/Purging the service and starting it again made the problem dissapear for now.

@atillamas FYI we believe we know what is causing this issue. Websocket requests that fail upgrade are left open and connected to their original back end. Nnginx out front is pooling these and sending requests straight through to this backend, completely bypassing the Fabio routing. Our workaround, for now, was to isolate the websocket onto an isolated fabio cluster.
I believe Fabio should be detecting these failed upgrades and closing the connections.

The way fabio currently handles websockets (via a raw tcp connection) makes detection a bit difficult. We can go back to a protocol proxy which relays WS messages instead. However, a first attempt is to inject a Connection: close header into the upgrade request since with HTTP/1.1 all connections are permanent unless declared otherwise.

@craigday Can you try that patch and see if it works? I'll add an integration test to simulate that behavior later.


Fix for issue #41 (md-button ignores type prop)

Hi Marcus,
Here is a fix for issue 41.
Congratulations for your excelent work!
Regards,
Paulo

Sounds good for me. Thanks a lot!


BUG Automatic install of classes fails

First, I have corrected in the benchmark solver files of benchmark_logreg_l2 in my branch FIX_utils_import
from benchopt.util import safe_import_context

by
from benchopt.utils.safe_import import safe_import_context

to solve a first bug which should be raised in an issue in all concerned benchmarks.
Then I still get this error because lightning cannot be installed automatically by the Benchmark class:
$ benchopt run ../benchmarks/benchmark_logreg_l2
Installing Lightning in None:... failed
/home/nidham/phd/benchOpt/benchopt/benchmark.py:116: UserWarning: Some solvers were not successfully installed, and will thus be ignored. Use 'export BENCHO_RAISE_INSTALL_ERROR=true' to stop at any installation failure and print the traceback.
  warnings.warn(
/tmp/tmp02ch3slw: line 2: benchopt.benchmark.Benchmark: No such file or directory

You should use the master version of the benchmark where I fixed all the import error: https://github.com/benchopt/benchmark_logreg_l2/commit/4fc90c20c5a1a2dc3471601eb8b10a242d7e2335
Also, if you want to see the install error, you can call BENCHO_RAISE_INSTALL_ERROR=1 benchopt run path/bench/logreg_l2.
Finally, it seems that you are calling the benchmark with an existing conda env that does not have the latest version of benchopt. You should use the -r flag that will reinstall everything to get rid of the line 2: benchopt.benchmark.Benchmark: No such file or directory error.

Ok It was a bug on our side actually.
This should have been fixed by #108


Failed to install provider due to expired OpenPGP key

Description:
I encountered an error while running terraform init with the ischluff/keepass provider v1.0.0. The error message indicates that the installation failed due to an expired OpenPGP key.
╷
│ Error: Failed to install provider
│
│ Error while installing ischluff/keepass v1.0.0: error checking signature:
│ openpgp: key expired
╵

Steps to Reproduce:

Run terraform init with the following configuration

provider "keepass" {
  database = "***"
  password = ***
}

Terraform Version: v1.6.0
Operating System: Linux
Provider Version: ischluff/keepass v1.0.0
Could you please update the OpenPGP key or provide guidance on how to resolve this issue?

Please try again, I have extended the signing key

I tried running terraform init with version 1.0.1, but I am still getting:
│ Error: Failed to install provider
│
│ Error while installing ischluff/keepass v1.0.1: error checking signature:
│ openpgp: key expired

Ok this hole was deeper than I expected.
Apparently the checking of signing key expirations was an unintended change by the terraform developers. See https://github.com/hashicorp/terraform/issues/33984
You are probably using an affected terraform version and may fix this by upgrading to atleast terraform-1.6.1 or downgrading to 1.5.7
Concerning the expired key, I have published an updated version to the ubuntu key servers.
Apparently this check is run against the gpg public key provided by the terraform registry. I have no way to actually update the key used by the registry, I just hope that they eventually refresh the keys they use there.

Thank you for the research and help. I managed to download the new 1.0.1 version with terraform 1.6.1.
In our case, we made the decision to transition to locally installed providers. For those encountering challenges with locally installed providers and Terraform/Terragrunt, here are some key points to consider:

Terraform writes provider information in the state file. Even if you remove any registry references to a particular provider from your configuration files, you will still need to do a terraform state replace-provider.

You will have to run terragrunt init first to install the local provider, before failing on the PGP Key check for the remote provider. The state replace-provider cmd needs both providers to be initialized for the specific module.

Since terragrunt supports inter-module dependencies, running terragrunt init within a dependent module failed to initialize the dependency module's local providers. The solution was to run a terragrunt run-all init --terragrunt-include-external-dependencies for it to be able to install the local providers.

Hope the above helps someone ;)


Support client side handshaker handoff fd

This is similar to what we do on the server side in chttp2_server.cc's handshake done code. It is just we never needed to support handoff on the client side.

CLA bot seems stuck. Closing and reopening.

Test failures:
ios: #20751
Bazel Opt: #20928
Bazel Tsan: #19726
Bazel Windows Debug: #20922
Interop: #20890


Switched off execution of the node tests.

This change is

Hi @jareksl, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by Microsoft and real humans are currently evaluating your PR.
TTYL, MSBOT;


How to reset session when entering a new space?

I've recently found that ARSession.state does not directly correspond to whether the device pose is "tracked".
 In this phase, hands and device are tracked correctly (but in an arbitrary offset to the XR Origin). Meshes (ARMeshManager) and QR codes do not seem to function.
Device is moved back to original location, recognizes space (device pose == tracked, ARSession.state == SessionTracking)

In many cases, this makes sense: objects are anchored to the original space, meshes created, QR codes located, etc.  However, this causes problems if the user actually wants to continue using the app in the new space.
I'd like to forcefully reset the session (start in a new arbitrary coordinate system) after (3) so that the app can continue in the new space, building new spatial meshes etc..
I've found that ARSession.Reset() does not seem to resolve this; the state just remains as SessionInitializing and AR features do not work until the original space is rediscovered. XRInputSubsystem.TryRecenter() returns true but does not fix the ARSession state.
So questions are:

Is this expected behaviour or is something more complex going on in my setup?
How do I go about resetting the session / allowing the app to continue in the new location?

Setup
Unity 2022.3.8f1 (Windows)
HoloLens 2
MRTK3 pre-16
Mixed Reality OpenXR Plugin 1.7.2
Unity OpenXR Plugin 1.8.2
ARFoundation 5.0.7

This does not reproduce for me with the sample repo as of today (with Unity 2021.3.x)
In this sample repo I observe in the Mesh example scene:

Running AR Mesh example scene as normal
Cover sensors, move to new space
Uncover sensors. Subsystems (meshing) resume as normal.

Compared to my configuration (details above):

Running scene as normal (meshes visible)
Cover sensors, move to new space.
Uncover sensors. Subsystems do not resume.
Return to original space. Subsystems resume.

Any ideas on what would cause the difference in behaviour?  If not, I can try to reconstruct the sample scene with newer ARFoundation and Unity for comparison.

This does reproduce in a blank/minimal template project: . To reproduce:
Create minimal project:

Create new Unity project in 2022.3.9f1
Add XR packages
XR Plugin Management 4.3.3
OpenXR Plugin 1.8.2
Mixed Reality OpenXR Plugin 1.8.1 (via scoped registry)
Enable OpenXR + Microsoft HoloLens feature group in XR Plugin Management
Add XR Origin, Tracked Pose Driver
Add an ARMeshManager with visuals (i.e. VR/SpatialMesh shader)
Accept fixes (permissions) for OpenXR project validation

Run Project

On HoloLens, "Remove All Holograms" to clear tracking data
Build and deploy
Observe normal behaviour at launch - spatial meshes visible, SessionTracking
Cover HoloLens sensors to cause tracking loss
With sensors covered, move to another room
Uncover sensors
Observe that tracking does not recover. Spatial meshes not visible, SessionInitializing
Return to main space and allow device to relocalize
Observe that session tracking recovers and meshes reappear

I've found the source of this issue. It doesn't present in the samples because it's an MRTK configuration rather than an OpenXR configuration. https://learn.microsoft.com/en-us/windows/mixed-reality/develop/unity/camera-in-unity?tabs=xr

Note that this option doesn't exist in MRTK3, and will likely result in confusion for developers, particularly as the behaviour is situational.
Configuring via code with xrInputSubsystem.TrySetTrackingOriginMode(TrackingOriginModeFlags.Unbounded); resolved the issue and produced behaviour the same as the sample.

For completeness, this is what worked for me (without MRTK)
/// <summary>
    /// Configures the input system to use a specified tracking mode on start.
    /// See https://github.com/microsoft/OpenXR-Unity-MixedReality-Samples/issues/165
    /// </summary>
    [RequireComponent(typeof(XROrigin))]
    public class XRCustomTrackingOrigin : MonoBehaviour
    {
        [SerializeField, Tooltip("Overrides the tracking origin mode from the XR Origin.")]
        private TrackingOriginModeFlags trackingOriginMode = TrackingOriginModeFlags.Unbounded;

        private async void Start()
        {
            Debug.Assert(GetComponent<XROrigin>().RequestedTrackingOriginMode == XROrigin.TrackingOriginMode.NotSpecified, "Tracking origin of the XROrigin must be set to NotSpecified.");

            List<XRInputSubsystem> subs = new List<XRInputSubsystem>();
            subs.Clear();
            bool initialized = false;

            while (!initialized)
            {
                await UniTask.DelayFrame(1);
                SubsystemManager.GetSubsystems(subs);
                foreach (XRInputSubsystem sub in subs)
                {
                    if (sub != null)
                    {
                        initialized = true;

                        if (sub.GetSupportedTrackingOriginModes().HasFlag(trackingOriginMode))
                        {
                            sub.trackingOriginUpdated -= OnInputSubsystemTrackingOriginUpdated;
                            sub.trackingOriginUpdated += OnInputSubsystemTrackingOriginUpdated;
                            SetTrackingMode(sub, trackingOriginMode);
                        }
                    }
                }
            }
        }

        private void OnInputSubsystemTrackingOriginUpdated(XRInputSubsystem subsystem)
        {
            TrackingOriginModeFlags old = subsystem.GetTrackingOriginMode();
            if (old != trackingOriginMode)
            {
                Debug.Log($"Attempting to change tracking origin mode from {old} to {trackingOriginMode}");
                SetTrackingMode(subsystem, trackingOriginMode);
            }
        }

        private void SetTrackingMode(XRInputSubsystem subsystem, TrackingOriginModeFlags trackingOriginMode)
        {
            if (subsystem.TrySetTrackingOriginMode(trackingOriginMode))
            {
                Debug.Log("Set input system tracking mode to " + trackingOriginMode + " for " + subsystem.subsystemDescriptor.id);
            }
            else
            {
                Debug.LogWarning("Failed to set input tracking mode for " + subsystem.subsystemDescriptor.id);
            }

        }
    }


add translation RU,UA,PL

@Ost268 wouldn't say google translate did good, at least with russian it did not

@Ost268 wouldn't say google translate did good, at least with russian it did not
I will fixed with time, so that there are no stupid questions on the forums

@Ost268 Thank you a lot. Btw Docs are moved to
https://github.com/woa-msmnile/woa-msmnile.github.io

Hello, please reply ''finined'' here if you have finished, thanks

Sir Please withdraw your commit of Lahaina here

You should open a branch for pr and develop lahaina in another branch. Otherwise it will show here.
Create a new branch(eg lahaina) from your current main, then go back to main, checkout it to the end of
where it should be(e.g.  81f2d31), and force push to main.
If you do not do this, we'll close the pr. Please tell me if you meet any issue.

@sunflower2333 Ost268 i wonder if you do not check github emails?🤔

Please stop pushing commits related to 8350 here. This PR is about updating the documentation only. We will have to close this PR unfortunately if it continues.

Closed. Please create a new pr if you want to add support for another device. Please create a different pr for documentation update. But please do not stitch a ton of changes in the same pr and deviate from the original goal.


Update PlaceNL.py

Easier for multiple accounts, save accounts in accounts.txt

Can't you make that into a alternative argument? That would mean that current scripts/bots won't need a lot of work updating.

Yes that is a good alternative ofcourse, like --config accounts.txt ?

I'm working on a config file implementation 👍


[sstar] make local build on ubuntu 20 happy

Good afternoon!
We are very grateful for the work you have done.
In the near future, we will test your patch and enable it to work
Thank you again very much !

Dear ch999dev
Please pay attention to the feedback on your patch, you may need your help
https://github.com/OpenIPC/openipc-2.1/issues/23

Hello. There is nothing console related in this patch. It only fixes things related to Python 3. May be the previous version of the script didn't generate / apply the DTB at all?

@ch999dev I suspect that the problem might be in scripts/ms_bin_option_get_int.py where the lines parsing value were originally commented out but your patch replaces them with an un-commented version:
if __name__ == '__main__':

     name=sys.argv[2]
-#    if sys.argv[3].upper().startswith( '0X' ):
-#        value=long(sys.argv[3],16)
-#    else:
-#        value=long(sys.argv[3])
+    if sys.version_info.major == 2:
+        if sys.argv[3].upper().startswith( '0X' ):
+            value=long(sys.argv[3],16)
+        else:
+            value=long(sys.argv[3])
+    else:
+        if sys.argv[3].upper().startswith( '0X' ):
+            value=int(sys.argv[3],16)
+        else:
+            value=int(sys.argv[3])

I'm currently building a new kernel image to check this hypothesis.

There is also forgotten print in scripts/ms_bin_option_update_int.py.

1 Yes. Good catch! This was commented out in the original script
2 This is not a forgotten print :) Just left this to check on build.

Now I've checked the original scripts/ms_bin_option_get_int.py and the value isn't being used anywhere, so the commented/uncommented section should be irrelevant.
But still, if I build my kernel image with 02_infinity6b0_kernel_4.9.84_sdk-Ispahan-ILS00V020_python3_check.patch, the console is dead. If I build the kernel image without this patch, the console works fine. Strange...

@ch999dev Finally! I've got the devil. The problem is in the third file named ms_builtin_dtb_update.py. You've added a few lines that replace a built-in constant with whatever is passed as the third command line argument:
name='#MS_DTB#'

...

if sys.version_info.major == 2:
    name=sys.argv[2]
else:
    name=sys.argv[2].encode()

Removing these four lines fixes the issue.

As far as I remember, without these patches BR2 could not determine KERNEL_VERSION and stopped at the stage of finalizing the image. And the mstar DTB script just failed (but BR2 ignored the error).
(ubuntu 20 / python 3)
What versions are you building on?

I'm building it using Docker. The base system is Debian Buster. I've also got into the problem where BR2 could not finalize the image but it was fixed by this patch: 03_infinity6b0_kernel_4.9.84_sdk-Ispahan-ILS00V020_fix_makefile.patch.

This is where the build stopped without a patch on local Ubuntu 20 / Python3
https://github.com/OpenIPC/openipc-2.1/runs/2719774667?check_suite_focus=true#step:7:127

Yes, that's the same line where it stopped in my case but as I mentioned, 03_infinity6b0_kernel_4.9.84_sdk-Ispahan-ILS00V020_fix_makefile.patch fixes it. However, the changes in ms_builtin_dtb_update.py seem to be unrelated and, at least in my case, it breaks things.

What if the unpatched ms_builtin_dtb_update.py silently fails and the DTB is remains not patched? And because of this, you have a console? :)

What if the unpatched ms_builtin_dtb_update.py silently fails and the DTB is remains not patched? And because of this, you have a console? :)

I don't think so. I've added a few debug outputs into the ms_builtin_dtb_update.py to find out what you're actually putting into the name variable. The command line argument that you use equals to the path to the DTB (arch/arm/boot/dts/infinity6b0-ssc009a-s01a.dtb in my case). This string, obviously, cannot be found in the resulting kernel image. The script is supposed to find the "#MS_DTB#" placeholder in the kernel image and replace it with the actual size of the DTB. Your patch breaks this.
Finally, it really isn't about having or not-having console. It is about having a broken kernel image or a working one. :)
To sum it up:

03_infinity6b0_kernel_4.9.84_sdk-Ispahan-ILS00V020_fix_makefile.patch is fine. It fixes the image finalization.
02_infinity6b0_kernel_4.9.84_sdk-Ispahan-ILS00V020_python3_check.patch:
a. changes in ms_bin_option_get_int.py are fine (even though not necessary because the original lines were commented out and the value isn't being used anywhere),
b. changes in ms_bin_option_update_int.py are also good (though the extra print probably isn't necessary),
c. changes in ms_builtin_dtb_update.py are not good because they replace a built-in constant with a value from command line and it results in a broken DTB.

Ah. Ok. WRT 2/c
This should be something like (seems typo)
    if sys.version_info.major == 2:
        offset=fmap.find(name)
    else:
        offset=fmap.find(name.encode)

Yes. That seems more like it. Though you probably meant offset=fmap.find(name.encode()).

Yes, exactly

pushed #25


feat: Adding option for insecure mail servers

What kind of change does this PR introduce?
This feature allows user to use insecure mail servers (i.e. invalid TLS certificate)
What is the current behavior?
Currently, mailme doesn't send email triggering "x509: certificate signed by unknown authority" in logs.
What is the new behavior?
The mail should now be sent if the user sets the boolean parameter Insecure to TRUE
Additional context
This PR depends on mailme following PR : https://github.com/supabase/mailme/pull/7

hi @RayaneB75, thanks for contributing but i don't think this is something we want to add - emails sent by gotrue contain the user's email confirmation link or a one-time password (OTP) and we wouldn't want them to be sent through an insecure mail server.

Hi, I understand your concern, however, I know that some environments might not be able to implement TLS certificates verifications for mails, and that's my case.
That's why this setting should be set as FALSE by default, still, letting users choose to enable it or keep it secure.

might not be able to implement TLS certificates verifications for mails, and that's my case.

@RayaneB75 hmm can you elaborate more on why this is not possible? are you using a SMTP service like AWS SES / SendGrid / Resend or are you using an in-house implementation?

might not be able to implement TLS certificates verifications for mails, and that's my case.

@RayaneB75 hmm can you elaborate more on why this is not possible? are you using a SMTP service like AWS SES / SendGrid / Resend or are you using an in-house implementation?

In our in-house mail implementation, for several reasons I can't elaborate here, we ¨need¨ to contact our mail server with it's mail.intra.fqdn name, but this mail server provides a mail.fqdn x509 certificate (for external queries). Thus, GOTRUE sends a failure when I try to send the OTP mail because the domain name on the certificate doesn't mach the domain name with use to contact it.

@RayaneB75 unfortunately, i don't see a compelling reason why this change (which makes it insecure by default) should go in - if you really need to use an insecure mail server, you can do it by creating a "send email" hook (https://supabase.com/docs/guides/auth/auth-hooks/send-email-hook?queryGroups=language&language=http) and making a request to your SMTP server directly


Intellij Idea Code Running Problem After Upgrade Dart 3.0.1 in Windows

After Upgrading in Dart 3.0.1
Whenever i try to run any code on my Intellij Idea it shows error !
`Crash when compiling file:///C:/Users/HomePC/Desktop/Dart/Module_04/Live%20Class/exception_handling.dart:
Unexpected Kernel SDK Version 7a6514d137 (expected 1618a77599).
#0      BinaryBuilder._readAndVerifySdkHash (package:kernel/binary/ast_from_binary.dart:657:7)
#1      BinaryBuilder.readComponent. (package:kernel/binary/ast_from_binary.dart:689:7)
#2      Timeline.timeSync (dart:developer/timeline.dart:166:22)
#3      BinaryBuilder.readComponent (package:kernel/binary/ast_from_binary.dart:673:21)
#4      ProcessedOptions.loadComponent (package:front_end/src/base/processed_options.dart:480:10)
#5      ProcessedOptions.loadSdkSummary (package:front_end/src/base/processed_options.dart:426:13)

#6      generateKernelInternal. (package:front_end/src/kernel_generator_impl.dart:79:31)

#7      withCrashReporting (package:front_end/src/fasta/crash.dart:136:12)

#8      generateKernel. (package:front_end/src/kernel_generator_impl.dart:48:12)

#9      generateKernel (package:front_end/src/kernel_generator_impl.dart:47:10)

#10     kernelForModule (package:front_end/src/api_prototype/kernel_generator.dart:100:11)

#11     SingleShotCompilerWrapper.compileInternal (file:///C:/b/s/w/ir/x/w/sdk/pkg/vm/bin/kernel_service.dart:407:11)

#12     Compiler.compile. (file:///C:/b/s/w/ir/x/w/sdk/pkg/vm/bin/kernel_service.dart:215:45)

#13     _processLoadRequest (file:///C:/b/s/w/ir/x/w/sdk/pkg/vm/bin/kernel_service.dart:889:37)

Process finished with exit code 252`

Removing area-intellij as this is an issue with loading kernel files not IntelliJ.

I have this problem!

cc @jensjoha

The initial issue is many months old and I assume the reporter is no longer having this problem.
The two hashes in the error message is from 3.0.0 and 3.0.1, so I'll guess that the update didn't go well. It being Windows I'm guessing the file was in use and wasn't actually properly unpacked or something. But I'm guessing.
The comment from September, being quite a bit newer, probably isn't the same 3.0.0 vs 3.0.1 thing (so should probably have been a separate issue) -- but without further information I'm guessing it's the same issue (i.e. some install/update not going as it should).
As such I'll close this issue.


Add git repo manipulation for API server. - [closed]

In GitLab by @wknet123 on Aug 10, 2017, 10:33
Merges dev-apiserver-git -> dev

In GitLab by @wknet123 on Nov 13, 2019, 09:53
closed


Support  blocks from Rust docs in vscode hover

Set "supportHtml": true to support rendering <code> blocks in hovers.
e.g.  https://github.com/rust-lang/rust/blob/1bd4fdc943513e1004f498bbf289279c9784fc6f/library/std/src/fs.rs#L109

bors r+

bors r+

changelog fix (first contribution) support <code> blocks in hover


Block processes creations originating from PSExec and WMI commands and MECM (SCCM)

As Microsoft states that ASR Rule: "Block processes creations originating from PSExec and WMI commands" is not compatible with management through MECM (SCCM).
So this ASR Rule is incompatible and a reason was given as to why: "... because this rule blocks WMI commands the Configuration Manager client uses to function correctly."
But does not say which  setting should be set when using in conjunction with MECM. Is it recommended to set this ASR policy to: Not Configured, Warn or Audit? Or do we need to set it to: "Off"?
Second question is:
Are there any plans to make this work (compatible) with Microsoft's MECM?
Source:
https://learn.microsoft.com/en-us/microsoft-365/security/defender-endpoint/attack-surface-reduction-rules-reference?view=o365-worldwide#block-process-creations-originating-from-psexec-and-wmi-commands

@Dansimp Please help us in resolving this issue. Thanks

To make it easier for you to submit feedback on articles on learn.microsoft.com, we're transitioning our feedback system from GitHub Issues to a new experience.
As part of the transition, this GitHub Issue will be moved to a private repository. We're moving Issues to another repository so we can continue working on Issues that were open at the time of the transition. When this Issue is moved, you'll no longer be able to access it.
If you want to provide additional information before this Issue is moved, please update this Issue before December 15th, 2023.
With the new experience, you no longer need to sign in to GitHub to enter and submit your feedback. Instead, you can choose directly on each article's page whether the article was helpful. Then you can then choose one or more reasons for your feedback and optionally provide additional context before you select Submit.
Here's what the new experience looks like.
Note: The new experience is being rolled out across learn.microsoft.com in phases. If you don't see the new experience on an article, please check back later.
First, select whether the article was helpful:

Then, choose at least one reason for your feedback and optionally provide additional details about your feedback:

Article was helpful
Article was unhelpful

Finally, select Submit and you're done!


I got an exception issue

System.AggregateException: One or more errors occurred. (The given key 'redist.zip' was not present in the dictionary.)
[Inner Exception]
System.Collections.Generic.KeyNotFoundException: The given key 'redist.zip' was not present in the dictionary.

Yeah me as well

SAMEEE


Release please unable to generate release notes for all modules.

Release please is generating release notes only for a fraction of modules configured. The monorepo has ~134 modules but only 38 modules have their release notes generated in the release main PR.

It was unclear from the initial issue report, but the release notes (CHANGELOG.md files) are being generated for all the libraries, however, the PR body is being truncated.
This will cause issues at release tagging time as it release-please won't be able to parse all the libraries it needs to tag.

As a repo maintainer, you should be able to preview the upcoming release notes easily. We also need the ability to store/fetch the entire set of proposed release notes.
a. find an alternative way to store the extended release notes

additional comments on the PR
a gist with a link to the gist in the PR body
a file attached to the issue (not sure if this is possible via the API)
an external store (e.g. GCS)
b. rebuild the PR again as if we would propose the PR and use those parsed notes
c. look at the modified CHANGELOG.md files in the release PR and parse out the notes

We are going with pushing extended release notes to a markdown file on a separate branch.


Check horizontal scrollbar width and adjust inner scroll container

This PR checks the width of the scrollbar and shrinks the internal scroll containers width to remove scrollbarWidth. We avoid setting overflow and potentially cutting off content, a user can set overflow via custom className if they need to hide scrollbars in a fixed grid header scenario.
Alternate to #133
No scrollbars when all columns fit:

Scrollbars when columns don't fit:

Double line content:

Hey @ryanseddon,
Thanks for taking the time to submit a PR! :)
I'm a little confused by part of it and so I left a question. This also only seems to handle the horizontal axis (scrollbar width). Seems like any solution we should needs to handle vertical too.
I'm still a little unclear from your 2nd and 3rd screenshots how this differs from the approach currently in master. (Maybe you could show me a case when they'd render something differently?)

Hmm I think I may have under thought the issue and this PR doesn't really solve what's going on it just looked like it did in my screenshots.

No worries dude. It's easy to do. I think this is an issue with a lot of edge-cases.
I have an idea based on our discussion on the other ticket though that I might give a shot. At least it's better than it was at this point :)


Update @sentry/node

In project Dashboard, Sentry keeps asking me to update:
sentry.javascript.node @v5.30.0
Update to @v6.10.0
Package version
"adonis5-sentry": "^2.0.0",

Updated, please use<EMAIL_ADDRESS>


eval issue

Hi,
First of all, thank you very much for this awesome library :)
I am writing a markdown package called markdown-preview-enhanced for atom editor, and I am using your library to render dot language graph.
It was working fine with viz.js version 1.3.0, however after I upgraded it to 1.7.0, it started casuing issue like:

EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self'"

It seems that the eval function caused the issue, but version 1.3.0 doesn't have this problem.
Any idea how to solve this problem?
Thanks

I just tried modifying the first line
var e;e||(e=eval("(function() { try { return Module || {} } catch(e) { return {} } })()"));
to
var e;e||(e= Module || {});
and it seems to be working.
not sure if that's the reason.

That call to eval() is emitted by Emscripten. We can replace it in the generated code, but I would prefer to address this by changing Emscripten settings if possible.
Emscripten does provide a NO_DYNAMIC_EXECUTION setting which removes calls to eval(), etc. but if that's used it won't run Closure Compiler. That increases the size of viz.js by about 100KB. I'll look at ways to use that setting but also avoid the file size increase.

eval is pretty bad; modern sites with a content security policy will block its use anyway.
And now this problem is now spreading through your dependents (I found this issue after having trouble with graphql-voyager).
I would suggest providing an eval-free version by default, with a smaller version available for those with special requirements.


Memory leak in Antimony

I've been running a large number of roadrunenr instances by generating the SBML from Antimony then loading the SBML. I've discovered there is a significant memory leak in Antimony such that I end up using 10s of Gb of memory and on rate occasions the entire spyder IDE just vanishes I think due to an out of memroy error. I'm using 20,000 models at a time and are generated on the fly in antimony format.
You can monitor the memory usage using the following scripts, I just watch the task manager memory consumption. Memory consuption continuous increases over time. This was run outside spyder from the windows terminal to eliminate any potential sypder issues.
Using latest tellurium windows installer. As an example if you run the loop with 50,000 models, the memory usage is at 16 Gb, and climbing. You can imagine that everntually it runs out of memory.
As a check you can run the second script which bypasses antimony and there is no leak at all so the leak must be in antimony.
import tellurium as te
With leak:
astr = """
S1 -> S2; k1*S1
S2 -> S3; k2*S2;
S3 -> S4 + S5; k3*S3;

k1 = 0.1; k2 = 0.2; k3 = 0.4
S1 = 10
"""

for i in range (20000):
   r = te.loada(astr)
   print (i)

Without leak:
import roadrunner
import antimony

astr = """
S1 -> S2; k1*S1
S2 -> S3; k2*S2;
S3 -> S4 + S5; k3*S3;

k1 = 0.1; k2 = 0.2; k3 = 0.4
S1 = 10
"""
antimony.loadAntimonyString(astr)
sbmlstr = antimony.getSBMLString()

for i in range (20000):
   r = roadrunner.RoadRunner (sbmlstr)
   print (i)

This turned out to not be a 'leak' in the traditional sense, but improper use of the Antimony library inside Tellurium.  Because you can store and load models separately using libantimony, it keeps a record of everything you've stored so you can load it again, until you call 'clearPreviousLoads'.  In the past, tellurium used to call that function (as far as I recall), but somehow it got dropped in more recent versions.  I'll fix that, and create a new release.

Now fixed!


Unable to issue "system vmlicense upload" on Azure

I have built a FortiGate NGFW - Single VM on Azure using the marketplace and selecting the "BYOL 6.0.3" model. Following the build, I SSH in, and issue:
config system global
set admin-sport 443
end
exit

I then execute the following playbook:
---
- hosts: localhost
  vars:
    host_ip_address: <IP_ADDRESS>
    host_user: someuser
    host_pass: 123_Somepass

  tasks:
  - name: Issue license
    fortiosconfig:
      host: "{{ host_ip_address }}"
      username: "{{ host_user }}"
      password: "{{ host_pass }}"
      vdom: "{{ host_vdom | default('root') }}"
      https: true
      config: system vmlicense upload
      action: upload
      config_parameters:
        filename: "./somefile.lic"

I get this as output:
(azure) vagrant@ubuntu:/HostHome/Documents/01 Projects - Ansible on Azure/Azure/2018-12-14_FGT_License$ ansible-playbook test.yml
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] ********************************************************************************************************************************************************

TASK [Gathering Facts] **************************************************************************************************************************************************
ok: [localhost]

TASK [Issue license] ****************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Exception: can not get following login
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1545039137.76-129793567464365/AnsiballZ_fortiosconfig.py\", line 113, in <module>\n    _ansiballz_main()\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1545039137.76-129793567464365/AnsiballZ_fortiosconfig.py\", line 105, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1545039137.76-129793567464365/AnsiballZ_fortiosconfig.py\", line 48, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/tmp/ansible_fortiosconfig_payload_nSoxNj/__main__.py\", line 845, in <module>\n  File \"/tmp/ansible_fortiosconfig_payload_nSoxNj/__main__.py\", line 833, in main\n  File \"/tmp/ansible_fortiosconfig_payload_nSoxNj/__main__.py\", line 756, in fortigate_config_upload\n  File \"/tmp/ansible_fortiosconfig_payload_nSoxNj/__main__.py\", line 469, in login\n  File \"/home/spriggsj/azure/local/lib/python2.7/site-packages/fortiosapi/fortiosapi.py\", line 130, in login\n    raise Exception('can not get following login')\nException: can not get following login\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

PLAY RECAP **************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1

The verbose output from the failure step is as follows:
TASK [Issue license] ****************************************************************************************************************************************************
task path: /HostHome/Documents/01 Projects - Ansible on Azure/Azure/2018-12-14_FGT_License/test.yml:9
The full traceback is:
Traceback (most recent call last):
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1545041674.57-132789405945393/AnsiballZ_fortiosconfig.py", line 113, in <module>
    _ansiballz_main()
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1545041674.57-132789405945393/AnsiballZ_fortiosconfig.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1545041674.57-132789405945393/AnsiballZ_fortiosconfig.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_fortiosconfig_payload_nLR_hv/__main__.py", line 845, in <module>
  File "/tmp/ansible_fortiosconfig_payload_nLR_hv/__main__.py", line 833, in main
  File "/tmp/ansible_fortiosconfig_payload_nLR_hv/__main__.py", line 756, in fortigate_config_upload
  File "/tmp/ansible_fortiosconfig_payload_nLR_hv/__main__.py", line 469, in login
  File "/home/spriggsj/azure/local/lib/python2.7/site-packages/fortiosapi/fortiosapi.py", line 130, in login
    raise Exception('can not get following login')
Exception: can not get following login

fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1545041674.57-132789405945393/AnsiballZ_fortiosconfig.py\", line 113, in <module>\n    _ansiballz_main()\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1545041674.57-132789405945393/AnsiballZ_fortiosconfig.py\", line 105, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1545041674.57-132789405945393/AnsiballZ_fortiosconfig.py\", line 48, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/tmp/ansible_fortiosconfig_payload_nLR_hv/__main__.py\", line 845, in <module>\n  File \"/tmp/ansible_fortiosconfig_payload_nLR_hv/__main__.py\", line 833, in main\n  File \"/tmp/ansible_fortiosconfig_payload_nLR_hv/__main__.py\", line 756, in fortigate_config_upload\n  File \"/tmp/ansible_fortiosconfig_payload_nLR_hv/__main__.py\", line 469, in login\n  File \"/home/spriggsj/azure/local/lib/python2.7/site-packages/fortiosapi/fortiosapi.py\", line 130, in login\n    raise Exception('can not get following login')\nException: can not get following login\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

Does this mean that because the license isn't valid at the time we try to make the API call, we're unable to use the API to deploy a license? Thanks!

And it continues! 😁
I re-wrote the module to do an SSH connection to perform the license upload on the Azure image:
---
- hosts: localhost
  gather_facts: false
  vars:
    host: "<IP_ADDRESS>"
    username: someuser
    password: 123_Somepass
    vdom: root
    tftp_host: "<IP_ADDRESS>"
    tftp_file: "somefile.lic"
  tasks:
  - name: ssh cli cmd
    fortiosconfig:
      action: ssh
      host: "{{ host }}"
      username: "{{ username }}"
      password: "{{ password }}"
      commands: |
                execute restore vmlicense tftp {{ tftp_file }} {{ tftp_host }}
                y

This didn't work, so I replaced the commands: block with this:
      commands: |
                execute restore vmlicense tftp {{ tftp_file }} {{ tftp_host }}\ny\n

Also didn't work. Using the expect module works however (it's not very clean, the expect module keeps issuing the same statements over and over agin - but it gets there!)...
---
- hosts: localhost
  gather_facts: false
  vars:
    host: "<IP_ADDRESS>"
    username: someuser
    password: 123_Somepass
    vdom: root
    tftp_host: "<IP_ADDRESS>"
    tftp_file: "somefile.lic"
  tasks:
  - name: ssh cli cmd
    expect:
      command: "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null {{ username }}@{{ host }}"
      responses:
        (?i)password: "{{ password }}"
        \x23: "execute restore vmlicense tftp {{ file }} {{ tftp_host }}"
        (?i)do: "y"
    register: expect_output
    failed_when: False
    changed_when: False

Can you share the result of :
get system interface
on your fortigate please ?

Can you share the result of :
get system interface
on your fortigate please ?

fgt01fw01 # get system interface
== [ port1 ]
name: port1   mode: dhcp    ip: <IP_ADDRESS> <IP_ADDRESS>   status: up    netbios-forward: disable    type: physical   netflow-sampler: disable    sflow-sampler: disable    scan-botnet-connections: disable    src-check: enable    mtu-override: disable    wccp: disable    drop-overlapped-fragment: disable    drop-fragment: disable
== [ port2 ]
name: port2   mode: static    ip: <IP_ADDRESS> <IP_ADDRESS>   status: up    netbios-forward: disable    type: physical   netflow-sampler: disable    sflow-sampler: disable    scan-botnet-connections: disable    src-check: enable    mtu-override: disable    wccp: disable    drop-overlapped-fragment: disable    drop-fragment: disable
== [ ssl.root ]
name: ssl.root   ip: <IP_ADDRESS> <IP_ADDRESS>   status: up    netbios-forward: disable    type: tunnel   netflow-sampler: disable    sflow-sampler: disable    scan-botnet-connections: disable    src-check: enable    wccp: disable


[BUG]

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional logs
Paperback log files are a great help in identifying and resolving issues.
Log files contain sensitive information such as server address and API key,
so please send them via<EMAIL_ADDRESS>And it is recommended to change related API key.
You can get app logs at paperback settings > export app logs,
and extension logs at paperback settings > external sources > Kavya > share logs.

Since there is no detailed description of the issue, I will close this issue.


Development: Move repository from yarn to npm

As as well per Stephan's request, this PR updates all of Apollon Standlone's dependencies to a recent version, replaces yarn by npm as a package manager and replaces tslint with eslint.
At the moment, a lot of the eslint default rules are disabled to have eslint behave the same way as tslint behaved before. In the future, we should sequentially reenable these rules and fix the issues detected. To not make this PR unnecessarily large, I decided to not do this in the current PR.
Motivation and Context
A lot of Apollon Standlone's dependencies, especially tslint, were outdated.
Steps for Testing
The most sensible test for this change would be to test if running and building the application behaves as expected.

@loreanvictor Could you have another look? I changed the README and ensure quote enforcements. :)

Can we merge this PR?


Internal sealed class are exported

Internal sealed class are exported by kswift but won't be available in the library. Swift compilation failed because it can not find types.
internal sealed class Response<T> {
    data class Success<T>(val result: T) : Response<T>()
    data class Failure<T>(val error: String?) : Response<T>()
}

Internal sealed class can be hidden with the @KSwiftExclude annotation but it does not work for the sub class.
sealed class Test {
    data class Exported(val value: Int) : Test()
    @KSwiftExclude // does not work, nor internal keyword
    internal data class Hidden(val value: Int) : Test()
}

Such class will make swift fail compilation as well.

will be available in 0.4.0

Apparently this also doesn't work for sealed classes under another internal class. Example:
internal interface MyInterface {
    sealed class Test {} // still gets exported
}

Not sure if there is any way to fix that.


only one shortcut per command

It seems that currently the shortcuts only support one per command (hence https://github.com/jupyter/jupyterlab/issues/320 J/K in the notebook are not easily possible)
this commit would ideally add the shortcuts, but since they are higher than the original (arrow) ones, the arrow shortcuts get ignored.
Looks like the commands are a hash table... Alternatively I'm not adding things correctly (and should have added directly to the sequence somehow?)..

edit:
relevant part of the phosphide
https://github.com/phosphorjs/phosphide/blob/master/src/services/shortcutregistry.ts#L50

Hi @to266, this behavior is fixed in the new consildated phosphor repo, which we will be switching over to in the next week or so.

@blink1073 - done?

Yep, fixed by #591.


Performance regression for BernoulliLogit

I was just playing around a bit with https://github.com/torfjelde/TuringBenchmarking.jl and noticed a sudden change in the runtime described in the README (the example model is suddenly 16x slower for gradient evaluation for ReverseDiff with compiled mode).
I eventually narrowed it down to #1892  being the cause, i.e. the performance of the following model:
@model function irt(y, i, p; I = maximum(i), P = maximum(p))
    theta ~ filldist(Normal(), P)
    beta ~ filldist(Normal(), I)
    Turing.@addlogprob! sum(logpdf.(BernoulliLogit.(theta[p] - beta[i]), y))

    return (; theta, beta)
end

absolutely tanks for ReverseDiff when we use the implementation of BernoulliLogit from Distributions.jl :confused:
On<EMAIL_ADDRESS>┌ Info: Turing.jl
│   run(suite) =
│    2-element BenchmarkTools.BenchmarkGroup:
│      tags: []
│      "linked" => 3-element BenchmarkTools.BenchmarkGroup:
│         tags: []
│         "evaluation" => Trial(1.333 ms)
│         "Turing.Essential.ReverseDiffAD{true}()" => Trial(1.752 ms)
│         "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(174.759 ms)
│      "not_linked" => 3-element BenchmarkTools.BenchmarkGroup:
│         tags: []
│         "evaluation" => Trial(1.339 ms)
│         "Turing.Essential.ReverseDiffAD{true}()" => Trial(1.796 ms)
└         "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(169.376 ms)

while on<EMAIL_ADDRESS>┌ Info: Turing.jl
│   run(suite) =
│    2-element BenchmarkTools.BenchmarkGroup:
│      tags: []
│      "linked" => 3-element BenchmarkTools.BenchmarkGroup:
│         tags: []
│         "evaluation" => Trial(554.568 μs)
│         "Turing.Essential.ReverseDiffAD{true}()" => Trial(16.418 ms)
│         "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(140.508 ms)
│      "not_linked" => 3-element BenchmarkTools.BenchmarkGroup:
│         tags: []
│         "evaluation" => Trial(554.415 μs)
│         "Turing.Essential.ReverseDiffAD{true}()" => Trial(16.445 ms)
└         "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(139.849 ms)

Given that evaluation and ForwardDiff is faster in the latter case, it's clearly an "issue" with ReverseDiff, but at the same time this is such a significant perf hit that it makes me a bit uncomfortable to just "leave it in" there :confused:
Thoughts? @devmotion

I don't have an immediate answer but generally the implementation in Distributions is much more specialized and hence more efficient than the previous BernoulliLogit in Turing (which fell back to BinomialLogit). So it seems there's a more ReverseDiff-specific issue here... Is there some type instability somewhere? Some type inference issue?

Do you know of a good way to check this for ReverseDiff?

It seems strange to me since AFAIK ForwardDiff is also used for broadcasting in ReverseDiff, no? So it's weird that ForwardDiff perf improves but ReverseDiff doesn't.

That's what I was thinking too, so I tried the folllowing impl to no avail:
function Distributions.logpdf(d::BernoulliLogit, x::Real)
    return (1 - x) * Distributions.logfailprob(d) + x * Distributions.logsuccprob(d)
end

perf is still bad.

No, I meant without two calls of log1pexp. Ie. something like
logpdf(d::BernoulliLogit, x::Bool) = -log1pexp(x ? -d.logitp : d.logitp)
function logpdf(d::BernoulliLogit, x::Real)
    logitp = d.logitp
    z = -log1pexpx(x == 0 ? logitp : -logitp)
    return insupport(d, x) ? z : oftype(z, -Inf)
end

Unfortunately doesn't help :confused:

😥 What happens if you implement the gradient of the logpdf function for ReverseDiff?

Or simpler: If you do not go through Distributions but define the logpdf directly as a separate function and broadcast that one?

I guess, for debugging it could also be useful to inspect the tape that ReverseDiff creates with the different implementations.

Just commenting to let you know I've seen the comments and I'm planning on having a go at it at some point, but right now I have some more pressing TODOs so need to put this on the backlog for a bit :confused:

I guess, for debugging it could also be useful to inspect the tape that ReverseDiff creates with the different implementations.

I wrote a small package to check this (https://github.com/torfjelde/ReverseDiffDebugUtils.jl) and AFAIT, they're the same :confused:
<EMAIL_ADDRESS>(which runs in ~1.4ms):

Distributions@0.25.80(which runs in ~18ms):

So seems like it has to be something in the reverse pass?

Think I found a clue: with<EMAIL_ADDRESS>ReverseDiffAD{false} is just as fast as ReverseDiffAD{true}, i.e. compilation doesn't help for some reason!

While on<EMAIL_ADDRESS>there's a bit of a slow-down from ~1.4ms to ~1.9ms

Probably is a type-instability somewhere then?

Profiling it, it becomes clear that ReverseDiff.special_forward_exec!(inst) is the issue where inst is the ∇broadcast that is different between the two implementations.
The reverse pass (ReverseDiff.ReverseExecutor) takes up almost noting of the 17ms runtime.

Changing the model to:
@model function irt(y, i, p; I = maximum(i), P = maximum(p))
    theta ~ filldist(Normal(), P)
    beta ~ filldist(Normal(), I)
    tmp = BernoulliLogit.(theta[p] - beta[i])
    Turing.@addlogprob! sum(logpdf.(tmp, y))

    return (; theta, beta)
end

to avoid nested broadcasting, we get
julia> @benchmark $(LogDensityProblems.logdensity_and_gradient)($∂ℓ, $θ)
BenchmarkTools.Trial: 1323 samples with 1 evaluation.
 Range (min … max):  3.588 ms …   5.117 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     3.741 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   3.773 ms ± 166.545 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%

when compiling, which is muuuuch better than the 17ms from before.
Now, without compilation:
julia> @benchmark $(LogDensityProblems.logdensity_and_gradient)($∂ℓ, $θ)
BenchmarkTools.Trial: 453 samples with 1 evaluation.
 Range (min … max):   8.556 ms … 39.326 ms  ┊ GC (min … max):  0.00% … 71.77%
 Time  (median):      9.103 ms              ┊ GC (median):     0.00%
 Time  (mean ± σ):   11.044 ms ±  6.140 ms  ┊ GC (mean ± σ):  13.86% ± 16.86%

which is also better.

Finally, using
logpdf_bernoulli_logit(logitp, x) = x == 0 ? StatsFuns.logistic(-logitp) : StatsFuns.logistic(logitp)

logpdf_bernoulli_logit(logitp, x::Bool) = x ? StatsFuns.logistic(logitp) : StatsFuns.logistic(-logitp)

@model function irt(y, i, p; I = maximum(i), P = maximum(p))
    theta ~ filldist(Normal(), P)
    beta ~ filldist(Normal(), I)
    Turing.@addlogprob! sum(logpdf_bernoulli_logit.(theta[p] - beta[i], y))

    return (; theta, beta)
end

we get
julia> suite = TuringBenchmarking.make_turing_suite(
           model,
           adbackends = [TuringBenchmarking.ForwardDiffAD{40}(), TuringBenchmarking.ReverseDiffAD{true}()]
       );

julia> run(suite)
2-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "linked" => 3-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(256.576 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(457.752 μs)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(126.160 ms)
  "not_linked" => 3-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(256.936 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(457.365 μs)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(126.680 ms)

which is significantly better (it's also 3X the speed of stan).

The logpdf in your last comment is wrong though, isn't it?

Uhm yes :facepalm: I mixed the logpdf and pdf impls. With the correct one we're only at:
julia> run(suite)
2-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "linked" => 3-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(549.488 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(3.246 ms)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(133.501 ms)
  "not_linked" => 3-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(550.383 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(3.834 ms)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(134.578 ms)

Replacing LogExpFunctions.log1pexp without all the conditional statements, i.e. just log1p(exp(x)), results in
julia> run(suite)
2-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "linked" => 3-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(492.951 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(2.993 ms)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(136.426 ms)
  "not_linked" => 3-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(496.187 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(4.443 ms)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(135.160 ms)

so a slightl improvement but not sufficient.

https://github.com/TuringLang/Turing.jl/issues/1934#issuecomment-1377574540 doesn't matter either?

Nah :confused:
julia> run(suite)
2-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "linked" => 4-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(375.343 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(4.449 ms)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(125.817 ms)
          "Turing.Essential.ZygoteAD()" => Trial(33.744 ms)
  "not_linked" => 4-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(378.344 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(3.554 ms)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(120.604 ms)
          "Turing.Essential.ZygoteAD()" => Trial(34.406 ms)

Okay, I think I've found the issue, or at least the explanation.
In the one with<EMAIL_ADDRESS>the broadcast results in a simdloop, which makes it so that:

Forward pass is much faster.
Reverse pass is dirt cheap.

So I decided the only logical thing to do is to add more broadcasting in the hopes that this, for some reason, would trigger similar things after the LogitBernoulli change, i.e. I did this:
julia> logpdf_bernoulli_logit(logitp, x) = -log1pexp(x == 0 ? -logitp : logitp)
logpdf_bernoulli_logit (generic function with 2 methods)

julia> logpdf_bernoulli_logit(logitp, x::Bool) = -log1pexp(x ? logitp : -logitp)
logpdf_bernoulli_logit (generic function with 2 methods)

julia> # performant model
       @model function irt(y, i, p; I = maximum(i), P = maximum(p))
           theta ~ filldist(Normal(), P)
           beta ~ filldist(Normal(), I)
           Turing.@addlogprob! sum(logpdf_bernoulli_logit.(theta[p] .- beta[i], y))

           return (; theta, beta)
       end
irt (generic function with 2 methods)

julia> # Instantiate
       model = irt(y, i, p);

julia> suite = TuringBenchmarking.make_turing_suite(
           model,
           adbackends = [TuringBenchmarking.ForwardDiffAD{40}(), TuringBenchmarking.ReverseDiffAD{true}(), TuringBenchmarking.ZygoteAD()]
       );

julia> run(suite)
2-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "linked" => 4-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(380.079 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(840.097 μs)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(100.826 ms)
          "Turing.Essential.ZygoteAD()" => Trial(33.692 ms)
  "not_linked" => 4-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(379.018 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(833.076 μs)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(100.951 ms)
          "Turing.Essential.ZygoteAD()" => Trial(33.791 ms)

julia> # performant model
       @model function irt(y, i, p; I = maximum(i), P = maximum(p))
           theta ~ filldist(Normal(), P)
           beta ~ filldist(Normal(), I)
           Turing.@addlogprob! sum(logpdf_bernoulli_logit.(theta[p] - beta[i], y))  # dont' broadcast `-`

           return (; theta, beta)
       end
irt (generic function with 2 methods)

julia> # Instantiate
       model = irt(y, i, p);

julia> suite = TuringBenchmarking.make_turing_suite(
           model,
           adbackends = [TuringBenchmarking.ForwardDiffAD{40}(), TuringBenchmarking.ReverseDiffAD{true}(), TuringBenchmarking.ZygoteAD()]
       );

julia> run(suite)
2-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "linked" => 4-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(380.220 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(3.054 ms)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(124.792 ms)
          "Turing.Essential.ZygoteAD()" => Trial(33.673 ms)
  "not_linked" => 4-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(380.720 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(3.575 ms)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(124.956 ms)
          "Turing.Essential.ZygoteAD()" => Trial(33.465 ms)

i.e. not broadcasting over the Distribution but replace it with a simpler method and broadcasting over - results in an implementation even faster than the original one on<EMAIL_ADDRESS>Which is even more annoying! So just the right amount of broadcasting leads to great performance, but if you do too little or too much, you're screwed.

Yeah, broadcasting performance issues and gotchas are about the worst... Countless hours that went into these things in the SciML ecosystem as well.

But are there any "guidelines" or just general advice on how to:

Identify these.
Fix them.

? :confused:

The usual workflow I experienced was that someone notices performance issues and then one starts debugging and finally notices that's broadcasting related (e.g., there was (is?) also a limit after how many broadcasting operations performance completely degrades, I'll try to dig up the relevant issues).

place it with a simpler method

BTW that's what DistributionsAD does for many univariate distributions with flatten: https://github.com/TuringLang/DistributionsAD.jl/blob/master/src/flatten.jl It's used mainly/only in filldist but maybe it would be useful more generally. Even though in principle ideally it would not be needed.

BTW that's what DistributionsAD does for many univariate distributions with flatten

Woah, I was completely unaware of this! And yeah this might be very helpful.

That indeed does wonders:
julia> # Using `DistributionsAD.flatten` to address performance.
       using Distributions, DistributionsAD

julia> """
           get_logpdf_expr(Tdist)

       Return a flattened method for computing the logpdf of `Tdist`.
       """
       function get_logpdf_expr(Tdist)
           x = gensym()
           fnames = fieldnames(Tdist)
           func = Expr(:->,
                       Expr(:tuple, fnames..., x),
                       Expr(:block,
                            Expr(:call, :logpdf,
                                 Expr(:call, :($Tdist), fnames...),
                                 x,
                                 )
                            )
                       )
           return :(flatten(::Type{<:$Tdist}) = $func)
       end
get_logpdf_expr

julia> # 1. Use `flatten` to extract a, well, flattened `logpdf`.
       eval(get_logpdf_expr(BernoulliLogit))
flatten (generic function with 1 method)

julia> # 2. [OPTIONAL] Use `StructArrays.jl` to avoid the initial call to the constructor entirely.

       # 3. Define a "fast" logpdf method.
       @generated function fast_logpdf(
           dist::Product{V,D,<:StructVector{<:Any,<:NamedTuple{names}}},
           x::AbstractArray
       ) where {V,D<:UnivariateDistribution,names}
           # Get the flatten expression.
           f = flatten(D)

           args = [:(dist.v.$n) for n in names]
           return :(sum($f.($(args...), x)))
       end
fast_logpdf (generic function with 2 methods)

julia> # 4. Convenience method for constructing `StructArray` without
       function DistributionsAD.arraydist(::Type{D}, args...) where {D<:Distribution}
           return DistributionsAD.arraydist(D, NamedTuple{fieldnames(D)}(args))
       end

julia> DistributionsAD.arraydist(::Type{D}; args...) where {D<:Distribution} = DistributionsAD.arraydists(D, NamedTuple(args))

julia> function DistributionsAD.arraydist(::Type{D}, args::NamedTuple) where {D<:Distribution}
           # TODO: Use `purename`?
           return DistributionsAD.arraydist(StructArray{D}(args))
       end

julia> # 5. Type-piracy so we can make use of `~`.
       function Distributions.logpdf(dist::Product{<:Any,<:UnivariateDistribution,<:StructVector}, x::AbstractVector{<:Real})
           return fast_logpdf(dist, x)
       end

julia> @model function irt_vroom(y, i, p; I = maximum(i), P = maximum(p))
           theta ~ filldist(Normal(), P)
           beta ~ filldist(Normal(), I)
           y ~ arraydist(BernoulliLogit, theta[p] - beta[i])

           return (; theta, beta)
       end
irt_vroom (generic function with 2 methods)

julia> model = irt_vroom(y, i, p);

julia> suite = TuringBenchmarking.make_turing_suite(
           model,
           adbackends = [TuringBenchmarking.ForwardDiffAD{40}(), TuringBenchmarking.ReverseDiffAD{true}()]
       );

julia> run(suite)
2-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "linked" => 3-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(389.573 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(747.912 μs)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(127.035 ms)
  "not_linked" => 3-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(391.116 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(745.925 μs)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(126.951 ms)

This also improves Zygote-perf 20-fold.

But why is the constructor so slow? It's the most simple struct one can come up with: https://github.com/JuliaStats/Distributions.jl/blob/d21c5a3d2386910b586cd9da188721f313073570/src/univariate/discrete/bernoullilogit.jl#L19-L21 Is it just that ReverseDiff is inherently bad with handling anything else than scalars or arrays?

Is it just that ReverseDiff is inherently bad with handling anything else than scalars or arrays?

Well, I have no idea but from this it does seem that if you have a constructor in a broadcasting statement, then you want to hide this from ReverseDiff.jl. That is, make it broadcast (param, x) -> logpdf(dist(param), x) instead of logpdf.(dist.(param), x). I'm assuming these two are recorded differently on the tape, e.g. the former record a broadcast of the closure while the latter records a fused broadcast of all the methods involved?

Or rather than fully blaming ReverseDiff, maybe it's more the type-inference failing in these two scenarios once you involve TrackedArray and/or TrackedReal?

Because, as I said before, the two recorded tapes are the same with the exception of the one broadcast instruction.
I tried using both Cthulhu and JET and couldn't properly identify this failure of type-inference though (but maybe I should check again now that I'm a bit more knowledgeable about ReverseDiff's internals).

Nah, still no luck with Cthulhu and JET

So I used Infiltrator.jl to break at the end of ∇broadcast and I observe the following:
infil> @code_warntype broadcast(df, ReverseDiff.value.(targs)...)
MethodInstance for broadcast(::Df{Base.Broadcast.var"#10#12"{Base.Broadcast.var"#16#18"{Base.Broadcast.var"#13#14"{Base.Broadcast.var"#9#11"}, Base.Broadcast.var"#13#14"{Base.Broadcast.var"#15#17"}, Base.Broadcast.var"#23#24"{Base.Broadcast.var"#25#26"}, Base.Broadcast.var"#19#20"{Base.Broadcast.var"#21#22"}, UnionAll}, typeof(logpdf)}, DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}, Tuple{}, Val{(1, 2)}}, ::Vector{Float64}, ::Vector{Int64})
  from broadcast(f::Tf, As...) where Tf in Base.Broadcast at broadcast.jl:798
Static Parameters
  Tf = Df{Base.Broadcast.var"#10#12"{Base.Broadcast.var"#16#18"{Base.Broadcast.var"#13#14"{Base.Broadcast.var"#9#11"}, Base.Broadcast.var"#13#14"{Base.Broadcast.var"#15#17"}, Base.Broadcast.var"#23#24"{Base.Broadcast.var"#25#26"}, Base.Broadcast.var"#19#20"{Base.Broadcast.var"#21#22"}, UnionAll}, typeof(logpdf)}, DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}, Tuple{}, Val{(1, 2)}}
Arguments
  #self#::Core.Const(broadcast)
  f::Df{Base.Broadcast.var"#10#12"{Base.Broadcast.var"#16#18"{Base.Broadcast.var"#13#14"{Base.Broadcast.var"#9#11"}, Base.Broadcast.var"#13#14"{Base.Broadcast.var"#15#17"}, Base.Broadcast.var"#23#24"{Base.Broadcast.var"#25#26"}, Base.Broadcast.var"#19#20"{Base.Broadcast.var"#21#22"}, UnionAll}, typeof(logpdf)}, DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}, Tuple{}, Val{(1, 2)}}
  As::Tuple{Vector{Float64}, Vector{Int64}}
Body::Union{Vector, BitVector}
1 ─ %1 = Core.tuple(f)::Tuple{Df{Base.Broadcast.var"#10#12"{Base.Broadcast.var"#16#18"{Base.Broadcast.var"#13#14"{Base.Broadcast.var"#9#11"}, Base.Broadcast.var"#13#14"{Base.Broadcast.var"#15#17"}, Base.Broadcast.var"#23#24"{Base.Broadcast.var"#25#26"}, Base.Broadcast.var"#19#20"{Base.Broadcast.var"#21#22"}, UnionAll}, typeof(logpdf)}, DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}, Tuple{}, Val{(1, 2)}}}
│   %2 = Core._apply_iterate(Base.iterate, Base.Broadcast.broadcasted, %1, As)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, Df{Base.Broadcast.var"#10#12"{Base.Broadcast.var"#16#18"{Base.Broadcast.var"#13#14"{Base.Broadcast.var"#9#11"}, Base.Broadcast.var"#13#14"{Base.Broadcast.var"#15#17"}, Base.Broadcast.var"#23#24"{Base.Broadcast.var"#25#26"}, Base.Broadcast.var"#19#20"{Base.Broadcast.var"#21#22"}, UnionAll}, typeof(logpdf)}, DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}, Tuple{}, Val{(1, 2)}}, Tuple{Vector{Float64}, Vector{Int64}}}
│   %3 = Base.Broadcast.materialize(%2)::Union{Vector, BitVector}
└──      return %3

for the "slow" version, and for the "fast" version:
infil> @code_warntype broadcast(df, ReverseDiff.value.(targs)...)
MethodInstance for broadcast(::Df{var"#53#54", DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}, Tuple{}, Val{(1, 2)}}, ::Vector{Float64}, ::Vector{Int64})
  from broadcast(f::Tf, As...) where Tf in Base.Broadcast at broadcast.jl:798
Static Parameters
  Tf = Df{var"#53#54", DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}, Tuple{}, Val{(1, 2)}}
Arguments
  #self#::Core.Const(broadcast)
  f::Df{var"#53#54", DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}, Tuple{}, Val{(1, 2)}}
  As::Tuple{Vector{Float64}, Vector{Int64}}
Body::Vector{DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}}
1 ─ %1 = Core.tuple(f)::Tuple{Df{var"#53#54", DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}, Tuple{}, Val{(1, 2)}}}
│   %2 = Core._apply_iterate(Base.iterate, Base.Broadcast.broadcasted, %1, As)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, Df{var"#53#54", DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}, Tuple{}, Val{(1, 2)}}, Tuple{Vector{Float64}, Vector{Int64}}}
│   %3 = Base.Broadcast.materialize(%2)::Vector{DiffResults.ImmutableDiffResult{1, Float64, Tuple{StaticArraysCore.SVector{2, Float64}}}}
└──      return %3

i.e. type-instability when broadcasting df for "slow" version and type-stability for df for "fast" version.
Tried removing the closure, i.e. replacing df with a wrapper-struct, but it doesn't help.
Seems like the function f itself is type-unstable?
infil> @descend ReverseDiff.splatcall(f, ReverseDiff.SVector(ReverseDiff.value.(map(first, targs))), untracked, inds)
splatcall(f, x::StaticArraysCore.SVector{N}, utargs::T, ::Val{tinds}) where {N, T<:Tuple, tinds} in ReverseDiff at /home/tor/.julia/packages/ReverseDiff/YkVxM/src/derivatives/broadcast.jl:111
    ∘ ─ %0 = invoke splatcall(::#10#12{…},::SArray{…},::Tuple,::Val{…})::Any
111 1 ─ %1 = StaticArrays.getfield(x, :data)::Tuple{Float64, Float64}│╻╷  macro expansion
    │   %2 = Base.getfield(%1, 1, true)::Float64                     ││┃│  getindex
    │   %3 = StaticArrays.getfield(x, :data)::Tuple{Float64, Float64}││╻   getindex
    │   %4 = Base.getfield(%3, 2, true)::Float64                     │││╻   getindex
    │   %5 = Core.getfield(f, :makeargs)::Base.Broadcast.var"#16#18"{Base.Broadcast.var"#13#14"{Base.Broadcast.var"#9#11"}, Base.Broadcast.var"#13#14"{Base.Broadcast.var"#15#17"}, Base.Broadcast.var"#23#24"{Base.Broadcast.var"#25#26"}, Base.Broadcast.var"#19#20"{Base.Broadcast.var"#21#22"}, UnionAll}
    │   %6 = Core.getfield(%5, :f)::UnionAll                         │││╻   #16
    │   %7 = (%6)(%2)::Any                                           ││││
    │   %8 = (Distributions.logpdf)(%7, %4)::Any                     │││
    └──      return %8                                               ││

Seems like the fact that BernoulliLogit is a UnionAll causes issues. It's worth pointing out that in the "old" version BernoulliLogit is a function rather than a UnionAll.

Even "worse", just hiding BernoulliLogit behind a constructor resolves the type-inference issue (for ReverseDiff.jl; Zygote.jl still struggles):
julia> BernoulliLogitF(x) = BernoulliLogit(x)
BernoulliLogitF (generic function with 1 method)

julia> @model function irt(y, i, p; I = maximum(i), P = maximum(p))
           theta ~ filldist(Normal(), P)
           beta ~ filldist(Normal(), I)
           Turing.@addlogprob! sum(logpdf.(BernoulliLogitF.(theta[p] - beta[i]), y))

           return (; theta, beta)
       end
irt (generic function with 2 methods)

julia> model = irt(y, i, p);

julia> suite = TuringBenchmarking.make_turing_suite(
           model,
           adbackends = [TuringBenchmarking.ForwardDiffAD{40}(), TuringBenchmarking.ReverseDiffAD{true}(), TuringBenchmarking.ZygoteAD()]
       );

julia> run(suite)
2-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "linked" => 4-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(379.554 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(746.761 μs)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(122.954 ms)
          "Turing.Essential.ZygoteAD()" => Trial(78.837 ms)
  "not_linked" => 4-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(379.219 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(749.739 μs)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(126.728 ms)
          "Turing.Essential.ZygoteAD()" => Trial(79.091 ms)

Zygote.jl still benefits from the arraydist(BernoulliLogit, ...) approach though:
julia> suite = TuringBenchmarking.make_turing_suite(
           model_vroom,
           adbackends = [TuringBenchmarking.ForwardDiffAD{40}(), TuringBenchmarking.ReverseDiffAD{true}(), TuringBenchmarking.ZygoteAD()]
       );

julia> run(suite)
2-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "linked" => 4-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(388.298 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(748.069 μs)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(122.363 ms)
          "Turing.Essential.ZygoteAD()" => Trial(1.495 ms)
  "not_linked" => 4-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "evaluation" => Trial(388.359 μs)
          "Turing.Essential.ReverseDiffAD{true}()" => Trial(748.285 μs)
          "Turing.Essential.ForwardDiffAD{40, true}()" => Trial(121.801 ms)
          "Turing.Essential.ZygoteAD()" => Trial(1.493 ms)

A problem with the generated functions (https://github.com/JuliaDiff/ReverseDiff.jl/blob/d522508aa6fea16e9716607cdd27d63453bb61e6/src/derivatives/broadcast.jl#L111)? A missing type parameter, leading to non-specialization of Function or Type somewhere?

Tried adding type-parameter and it doesn't do anything :confused:

But can it specialize when BernolliLogit is a UnionAll? If we had something like BernoulliLogit{Float64} then it would probably be okay

Just to make things even "clearer":
infil> @code_warntype f.makeargs.f
MethodInstance for getproperty(::Base.Broadcast.var"#16#18"{Base.Broadcast.var"#13#14"{Base.Broadcast.var"#9#11"}, Base.Broadcast.var"#13#14"{Base.Broadcast.var"#15#17"}, Base.Broadcast.var"#23#24"{Base.Broadcast.var"#25#26"}, Base.Broadcast.var"#19#20"{Base.Broadcast.var"#21#22"}, UnionAll}, ::Symbol)
  from getproperty(x, f::Symbol) in Base at Base.jl:38
Arguments
  #self#::Core.Const(getproperty)
  x::Base.Broadcast.var"#16#18"{Base.Broadcast.var"#13#14"{Base.Broadcast.var"#9#11"}, Base.Broadcast.var"#13#14"{Base.Broadcast.var"#15#17"}, Base.Broadcast.var"#23#24"{Base.Broadcast.var"#25#26"}, Base.Broadcast.var"#19#20"{Base.Broadcast.var"#21#22"}, UnionAll}
  f::Symbol
Body::Any
1 ─      nothing
│   %2 = Base.getfield(x, f)::Any
└──      return %2

Brian Chen pointed out in Slack that the reason why Zygote fails this check: https://github.com/FluxML/Zygote.jl/blob/c2f1794ca9da3088a2f3bfb0144c8bfc4dd89d9a/src/lib/broadcast.jl#L198-L199
which causes the slow down.
In particular, the _dual_purefun fails when BernoulliLogit is used.

Regarding Zygote, I wonder if it's problematic for the compiler that the pullback is closed over the variable len of type Val in https://github.com/FluxML/Zygote.jl/blob/c2f1794ca9da3088a2f3bfb0144c8bfc4dd89d9a/src/lib/broadcast.jl#L206. One could check if making the pullback a callable struct with type parameter inclen(args) improves anything.

I think the fast Zygote version just hits  broadcast_forward though. And it was pointed out that BernoulliLogitF doesn't help Zygote because the combine_eltypes will not be a subtype of Union{Real,Complex} and so the check https://github.com/FluxML/Zygote.jl/blob/c2f1794ca9da3088a2f3bfb0144c8bfc4dd89d9a/src/lib/broadcast.jl#L198 is false.

Coincidentally, https://github.com/FluxML/Zygote.jl/issues/1359 was filed today about type instability in ∇broadcasted. The culprit is not len, but the compiler failing to const prop i into an ntuple callback. As Tor said though, ideally you'd want to avoid hitting this fallback path in the first place

Sure, the other path would be ideal - but if the fallback path could be improved, it might make it less of an issue.

Hence https://github.com/FluxML/Zygote.jl/pull/1360. I do think the fundamental issue is that we don't have a forward mode AD that understands complex structures like Distributions. Thus we're left with a half-dozen flattening/unflattening libraries and multiple reverse-mode ADs with the same performance cliffs in broadcasting.

It's nothing too exotic, just that one should be able to differentiate through code like Normal.(means, stds) efficiently. I see there was some attempt at making this work for ForwardDiff at https://github.com/JuliaDiff/ForwardDiff.jl/pull/307, but that PR hasn't been touched for almost 5 years :(

For generic inner type, I don't have a clear understanding of how to verify that it is "dualable" or "taylorable".

It's a good question and I don't have an answer. The only Julia AD I know of which can (potentially) do this right now is Enzyme, and that's pure SCT. Perhaps there's something you could do with type-level programming and internals like return_type to determine whether a particular struct is dualable/taylorable.


Bash script does nothing

When I run the one-liner bash script for XCode, my terminal instance refreshes but nothing else happens. Typing codesigndoc gives me zsh: command not found: codesigndoc obviously.
Please include the full (Terminal) output of codesigndoc -
in Terminal.app you can save the output into a file with Command+s
In addition to this, if codesigndoc generated an Xcode output log, please attach that too.

Hi @AceNorth,
Thank you for reporting the issue! I forwarded it to our tooling team and they'll get back to you if they find something!

Hi @AceNorth ,
which terminal app and which shell do you use?
As a workaround, you can download the binary from: https://github.com/bitrise-tools/codesigndoc/releases (codesigndoc-Darwin-x86_64 ) add executable permission to it: chmod +x full/path/to/the/binary and run: full/path/to/the/binary scan

The manual install does work. I don't use any plugins.

Do you have any idea how could i reproduce this issue? I'm using iTerm and zsh as well. But i never saw a similar issue, also none of our users reported it.

Nope! No idea. I'm not getting any errors or output whatsoever, just a reboot of my terminal. If nobody else is experiencing it, must just be a weird edge case.

If you would have any more info about the issue, please open a new one with the details and thanks for reporting!


Update documentation concerning shell script

User story

As a developer
I want / need instructions in the documentation to implement the shell script
So that I know how to implement it

Acceptance criteria

Documentation updated concerning the shell script

Feature DoD

Feature has been fully implemented
All acceptance criteria were met
All tests are passing
No known errors
Documentation updated
Bill of materials updated

Sprint Release DoD

Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Peer code review passed
Developers agreed to release
Documentation updated
Bill of materials updated

remark of @A-Sandrini : login as separate step necessary; @murali-kri5hna : manual step mandatory currently as user role is not stored with creds in the config file. Idea for future feature: store creds in config file or setup script


Generate random cookies

Checklist

[X] I'm reporting a feature request
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2022.04.08 (update instructions) or later (specify commit)
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Description
I'd like it if name and/or value of the cookie is not provided in InfoExtractor._set_cookie these parameters generated randomly.
Verbose log
No response

This is not needed just for https://github.com/yt-dlp/yt-dlp/pull/3624 imo. If such a function becomes useful for other extractors, it can eventually be moved into utils.
Also, there is no reason for this to be a separate issue. This is simply a implementation detail of #3624, and can just be discussed in that PR