Agora encryption is not working

Hi guys, we are trying to enable agora encryption but we got black/blue screens, support answered us that the problem comes from flutter, so SDK is not encrypting.
here is code snipped which we used. testing device: Samsung Galaxy S10
    await AgoraRtcEngine.create(...);
    await AgoraRtcEngine.setEncryptionMode('aes-128-ecb');
    await AgoraRtcEngine.setEncryptionSecret('ourSecret');

also we used other encryption modes too

what version of the plugin did you add? We fix iOS encryption bug in v1.0.10

I am using agora_rtc_engine: ^1.0.10

Android or iOS？

I am testing in my real device (Samsung Galaxy S10) also enabled production for agora, so we are using token + appid combination

what platform is another device

Web, (latest chrome)

has web set the same config?

yeah, the same one, also here is support answer

I will test it later.

It works for me. Could you pls share your web code?

here is dart code
    await AgoraRtcEngine.create(...);
    await AgoraRtcEngine.setEncryptionMode('aes-128-ecb');
    await AgoraRtcEngine.setEncryptionSecret('ourSecret');

    await AgoraRtcEngine.enableVideo();
    await AgoraRtcEngine.enableAudio();
    await AgoraRtcEngine.joinChannel(
      widget.videoCallTokenInfo.token,
      widget.videoCallTokenInfo.room,
      null,
      widget.videoCallTokenInfo.identity,
    );

and here is web
       const client = this.agoraService.createClient(
            {
                codec: Utils.isSafari() ? 'h264' : 'vp8',
                mode: 'rtc',
            },
            true,
        );
        client.setEncryptionMode('aes-128-ecb');
        client.setEncryptionSecret('ourSecret');

and then calling startCall function
    startCall({ token, room, identity }): void {
        this.agoraService.client.join(
            token,
            room,
            identity,
            (uid) => {
                this.localStream = this.agoraService.createStream({
                    streamID: uid,
                    audio: true,
                    cameraId: null,
                    microphoneId: null,
                    video: true,
                    screen: false,
                });

            },
            (err) => {
                console.error('Join channel failed', err);
            },
        );
    }

Where is client.init on the web?

we are using ngx-agora and createClient function inits itself, also currently all infrastructure is working good, it's not working only when I am enabling encryption

could you share the code of this.agoraService.createClient?

it's open-source lib, here is the link
https://github.com/drew-thompson/ngx-agora/blob/master/projects/ngx-agora/src/lib/ngx-agora.service.ts#L80

I use our web SDK the same version as ngx-agora, and it works for me.

could you try our demo and add encryption code?

We tested encryption in all web browsers in web and mobile, so the problem comes from agora flutter sdk, maybe it's not supporting token and encryption together

I mean you can use our flutter and web demo, add the encryption code, then test them, it works for me.

@NarHakobyan could you pls provide a channel name and time which you used to join and verified the encryption feature to be not working. we can check our backend to see what could be wrong.

Hi dear @plutoless, sure here is channel name cf4b11b0-8994-11ea-8004-1b8552d7be8f and approximately date was 2020-04-29

I mean you can use our flutter and web demo, add the encryption code, then test them, it works for me.

@NarHakobyan Have you try this?

@NarHakobyan i have checked the backend, it seems your web client is calling encryption api properly while android not. i think sth is wrong with your flutter setup, could you pls double check? as we have verified using our own demo and it's working.

@plutoless I am using the same code from example but with production, so we are not using only APP_ID also using token

i don't think so... token should only affect your channel join, as long as you can join the channel successfully it should not affect your rtmp streaming

i don't think so... token should only affect your channel join, as long as you can join the channel successfully it should not affect your rtmp streaming

I can send full code from mobile, if it will help

yes pls. we can try out

  @override
  void initState() {
    super.initState();
    WidgetsBinding.instance.addObserver(this);
    socketState.on('cancel-video-call', cancelVideoCallEvent);
    // initialize agora sdk
    initialize();
  }

  @override
  void dispose() {
    // clear users
    _users.clear();
    threadsState.newCreatedThread = null;
    // destroy sdk
    AgoraRtcEngine.leaveChannel();
    AgoraRtcEngine.destroy();
    socketState.off('cancel-video-call', cancelVideoCallEvent);
    super.dispose();
  }

  @override
  void didChangeAppLifecycleState(AppLifecycleState state) {
    //TODO fix bug when close app with muted settings open setstate;
    if (state == AppLifecycleState.paused ||
        state == AppLifecycleState.inactive ||
        state == AppLifecycleState.detached) {
      if (!videoCallMuted) {
        AgoraRtcEngine.muteLocalVideoStream(true);
        videoCallMuted = true;
      }
      if (!muted) {
        AgoraRtcEngine.muteLocalAudioStream(true);
        muted = true;
      }
    }
    if (state == AppLifecycleState.resumed) {
      if (videoCallMuted) {
        AgoraRtcEngine.muteLocalVideoStream(false);
        videoCallMuted = false;
      }
      if (muted) {
        AgoraRtcEngine.muteLocalAudioStream(false);
        muted = false;
      }
    }
  }

  void cancelVideoCallEvent(data) {
    _onCallEnd(context);
  }

  Future<void> initialize() async {
    await _initAgoraRtcEngine();
    _addAgoraEventHandlers();
    await AgoraRtcEngine.enableWebSdkInteroperability(true);
    await AgoraRtcEngine.setParameters(
        '''{\"che.video.lowBitRateStreamParameter\":{\"width\":320,\"height\":180,\"frameRate\":15,\"bitRate\":140}}''');
  }

  /// Create agora sdk instance and initialize
  Future<void> _initAgoraRtcEngine() async {
    await AgoraRtcEngine.create('78ce6db18fc24e8b8835a4b18644a76a');
    await AgoraRtcEngine.setEncryptionMode('aes-128-ecb');
    await AgoraRtcEngine.setEncryptionSecret('ourSecret');

    await PermissionHandler().requestPermissions([PermissionGroup.camera]);

    await AgoraRtcEngine.enableVideo();
    await AgoraRtcEngine.enableAudio();
    await AgoraRtcEngine.joinChannel(
      widget.videoCallTokenInfo.token,
      widget.videoCallTokenInfo.room,
      null,
      widget.videoCallTokenInfo.identity,
    );
  }

  /// Add agora event handlers
  void _addAgoraEventHandlers() {
    AgoraRtcEngine.onError = (code) {
      setState(() {
        final info = 'onError: $code';
        _infoStrings.add(info);
      });
    };

    AgoraRtcEngine.onJoinChannelSuccess = (
      String channel,
      int uid,
      int elapsed,
    ) {
      setState(() {
        final info = 'onJoinChannel: $channel, uid: $uid';
        _infoStrings.add(info);
      });
    };

    AgoraRtcEngine.onLeaveChannel = () {
      setState(() {
        _infoStrings.add('onLeaveChannel');
        _users.clear();
      });
    };

    AgoraRtcEngine.onUserJoined = (int uid, int elapsed) {
      setState(() {
        final info = 'userJoined: $uid';
        _infoStrings.add(info);
        _users.add(uid);
      });
    };

    AgoraRtcEngine.onUserOffline = (int uid, int reason) {
      setState(() {
        final info = 'userOffline: $uid';
        _infoStrings.add(info);
        _users.remove(uid);
      });
    };

    AgoraRtcEngine.onFirstRemoteVideoFrame = (
      int uid,
      int width,
      int height,
      int elapsed,
    ) {
      setState(() {
        final info = 'firstRemoteVideo: $uid ${width}x $height';
        _infoStrings.add(info);
      });
    };
  }

  /// Helper function to get list of native views
  List<Widget> _getRenderViews() {
    final List<AgoraRenderWidget> list = [
      AgoraRenderWidget(0, local: true, preview: true),
    ];
    _users.forEach((int uid) => list.add(AgoraRenderWidget(uid)));
    return list;
  }

@NarHakobyan run createClient, but the client is initing, you should setEncryptionMode after onSuccess callback
const client = this.agoraService.createClient(
            {
                codec: Utils.isSafari() ? 'h264' : 'vp8',
                mode: 'rtc',
            },
            true,
        );
        client.setEncryptionMode('aes-128-ecb');
        client.setEncryptionSecret('ourSecret');

 createClient(
    config: ClientConfig,
    autoInitializing: boolean = true,
    onSuccess?: () => void,
    onFailure?: (error: Error) => void
  ): AgoraClient {
    this.client = this.AgoraRTC.createClient(config);
    if (autoInitializing) {
      this.init(this.config.AppID, onSuccess, onFailure);
    }
    return this.client;
  }

@LichKing-2234 I have tried that approach also

@NarHakobyan By the way, if you call leaveChannel, you should setEncryptionMode again.

I have tried your code, if setEncryptionMode after onSuccess callback, it worked

but where is onSuccess callback in flutter?

I am sure it's a problem of flutter, I have tested web encryption in all browsers, all worked but flutter, it's showing a blue and black screen, also added onSuccess (in web) but the same

            this.agoraService.createClient(
                {
                    codec: Utils.isSafari() ? 'h264' : 'vp8',
                    mode: 'rtc',
                },
                true,
                // eslint-disable-next-line no-console
                () => {
                    console.log('create is created');
                    this.agoraService.client.setEncryptionMode('aes-128-ecb');
                    this.agoraService.client.setEncryptionSecret('ourSecret');
                },
                () => console.error("error client isn't created"),
            );

is it worked flutter with flutter? I think web encryption failed.

can you please check this channel? 41a88f28-fb3e-4ddb-a2dd-4099adc61e7e

@LichKing-2234 @plutoless any new guys? 😨 this is a very important part

Hi, I'm not sure if this is your problem, but it might be worth a try!
The Flutter documentation says to call setEncryptionSecret before setEncryptionMode (see https://github.com/Meherdeep/agora-flutter-documentation#setencryptionmode), so you could try changing the code from your comment to:
/// Create agora sdk instance and initialize
Future<void> _initAgoraRtcEngine() async {
    await AgoraRtcEngine.create('...');
    await AgoraRtcEngine.setEncryptionSecret('ourSecret');
    await AgoraRtcEngine.setEncryptionMode('aes-128-ecb');

    await PermissionHandler().requestPermissions([PermissionGroup.camera]);

    await AgoraRtcEngine.enableVideo();
    await AgoraRtcEngine.enableAudio();
    await AgoraRtcEngine.joinChannel(
      widget.videoCallTokenInfo.token,
      widget.videoCallTokenInfo.room,
      null,
      widget.videoCallTokenInfo.identity,
    );
  }

I'm not sure if it does actually make a difference but could be worth a try!

@LichKing-2234 @plutoless any new guys? 😨 this is a very important part

I will try to run the angular lib and test it

I have tried the angular demo and flutter demo, it works for me

here is the changed code

here is the changed code of flutter demo

if the encryption is not matching flutter, you can see the warning in the console log.
13:59:55:641 Agora-SDK [WARNING]: [82CEB] stream crypt error

Sorry guys for delay, here is logs

maybe you can try to add encryption after enbaleViedeo

The same, also I am getting 504 timeout and cors errors

Please refer to the diff code picture, which I have commented above. I test the sample code and it works fine.

I just did the same :)

I just did the same :)

The same code without any differences?

you can try to clone our flutter demo and the ngx-agora demo, then modify the code as same as mine. and test it.

In the demo there is only appID configuration, we are using also token

what about cors errors?

In the demo there is only appID configuration, we are using also token

okay, I will test it with the token.

I have joinChannel with the token, it works fine. please ignore web video is black because my mac camera is broken.

what about cors errors?

try ping webcollector-1.agora.io?

what about cors errors?

try ping webcollector-1.agora.io?

it's cors error, ping is working maybe in webcollector need to add * for cross-origin access.

guys, I am sure agora flutter SDK has issues, It's working nice on the web, We tested many times with both encrypted and encrypted with nonencrypted, so all cases working. We are getting errors on mobile

is it work the first time?

have you set encryption config when you rejoin the channel?
the encryption config will reset when you leave the channel.
https://docs.agora.io/en/Video/API Reference/java/classio_1_1agora_1_1rtc_1_1_rtc_engine.html#a3586017747a74d79ea6e7421af6b1a20

Yeah, it's not working for the first time

I have no idea about it,The best way is provide your project or make a demo, then I can test it.

Hey guys, sorry for so much disturbing you, that was really dummy issue, web and mobile used different appid, but I don't know why videocall worked, thanks a lot for help
Be safe.


RFC: repo-level config, per-category nav

Continues #217.
This project is quite hidden, based on the docs branch. I'd suggest eventually moving that branch to its own repo and optionally reading open graph config from libraries' repositories.
// docs.config.json
{
  // Directory to look for docs, defaults to repo root
  // "root": "docs",

  // OpenGraph meta
  "title": "React Three Fiber",
  "description": "React-three-fiber is a React renderer for three.js",
  "image": "docs/social-image.jpg",

  // Order categories by pathname, `nav` frontmatter will control page order per category
  categories: [
    "introduction",
    "getting-started"
  ]
}

We can substitute local, hard-coded config in this repo with a NPM-like repository path:
// data/libraries.ts
'react-three-fiber': 'pmndrs/react-three-fiber/master/docs.config.json',

Curious if we can substitute some of the open graph meta with those from GitHub. Would be nice to point to a repo and have the site find docs.config.json in addition to repo meta.
// data/libraries.ts
'react-three-fiber': 'pmndrs/react-three-fiber',

the repo has been renamed to pmndrs/docs and main is now the production branch deployed on vercel


Compile question about using  c++ API.

Hi,experts. I'd like to use this repo by c++ API. But I met  a question when I use the cmd to compile my code.
g++  test_matmul.cpp  -o main.exe  -I $intel_npu_acceleration_library_include_path -I $openvino_include_path -L $intel_npu_acceleration_library_lib -L $openvino_lib_path -lopenvino -lintel_npu_acceleration_library -mavx2 -mf16c.
There is test_matmul.cpp.
#include "nn.factory.h"
int main(){
   auto factory = ModelFactory("NPU",inC,outC,batch,true);
   // create model
   .......
}

Console output:

Your is most probably a linking issue. I just created a c++ example to show how you can use the c++ API: https://github.com/intel/intel-npu-acceleration-library/tree/main/examples/cpp

It works, thank you a lot.


Use kubernetes manifest instead of kubectl_manifest in the guides

The kubernetes provider now officially supports manifests. Could the getting started guides be adjusted to use that terraform resource instead of the kubectl_manifest?

There are some considerations that have to be made for the way the Kubernetes provider works. The main issue is that it uses server side apply to deal with CRDs which means that the Kubernetes cluster has to be created before running a plan. This means that it is not possible to create a Kubernetes cluster and bootstrap Terraform within the same state.
Would that be an issue for you?
I have nothing against adding it to the docs however, I have mostly been waiting for someone else to come along and figure out the details.

Creating kubernetes in a separate module is already a recommendation on the official kubernetes terraform provider, see excerpt below from https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#stacking-with-managed-kubernetes-cluster-resources:
WARNING
When using interpolation to pass credentials to the Kubernetes provider from other resources, these resources SHOULD NOT be created in the same Terraform module where Kubernetes provider resources are also used. This will lead to intermittent and unpredictable errors which are hard to debug and diagnose. The root issue lies with the order in which Terraform itself evaluates the provider blocks vs. actual resources.

So these docs are new to me, then again its been a while since I checked those docs. To be honest I can see their point however I do not agree with it from a practical position.
I have been creating Kubernetes resources in the same TF state as the cluster for years now without any major issues, and I am planning on continuing doing so for the foreseeable future. I do not know what the rest of the community is doing, so I cant really speak for how others are doing.
My personal opinion is that the Terraform project is taking the easy way out by giving these types of recommendations, instead of actually solving the underlying problems. No other provider comes with these types of recommendations and part of the provider design is that these types of dependencies should not even exist. Terraform and Kubernetes has already been a hit or miss experience sadly.
Having said that I welcome additions to the documentation. I do not have too much time over so I can't really promise when I can get this done. So if you want to contribute the documentation that would be highly appreciated.

I just want to lean into this a bit and flag it as semi-important for our use-case.  Independent of how you choose to modularize and bootstrap Kubernetes, this is a pretty common pattern for "All-in-one" deployment toolchains, and it was fairly effortless to achieve for fluxv1 via its Helm chart (yes, I know Helm is a no-go, but bear with me).  Having a multi-module Terraform app bootstrap EKS and then use the module credential outputs to pass to another module (without having to do a bizarre kubeconfig modification which is annoying for making these flows both user-and-automation portable) that bootstraps Flux for the rest of the configuration is fairly table-stakes for us.
I've been working on a prototype with this provider and the "kubernetes_manifest" resource, and it feels like we're riiiiiiiiight on the edge of that being doable.  Outside of bootstrap flows, though, there's one nagging little issue in that Fluxv2's controller CRDs publish with the top-level status key despite it not actually being used by the Kube API server on apply.  While this is relatively benign with generalized manifest deployment via kubectl, the "kubernetes_manifest" resource restricts its usage.  Outside of the obvious opinion around Terraform erroring on a key that Kubernetes itself disregards, a paved-path solution that doesn't require me parsing a bunch of YAML and removing status keys would be useful if we're going to take Fluxv2 seriously in our production workflows.

I think we had the topic of status key being published in places where it need not be republished came up in a Flux dev meeting only this week. It looks like you've identified a place where that status key is published and it shouldn't be 👍
If that's the only thing blocking you from using terraform provider in a novel way, I'm pretty positive that Flux team will be more than willing to accept a PR that changes the way they are published to conform! 🎉 It's not exactly clear to me what needs to change, but it is clear that status is a write-only field for the Kubernetes API and read-only for users, so should not usually be included in any manifests that are destined to be applied with Flux.

Hah, I actually only picked the one as an example.  I can actually probably expedite your research and I'm also happy to submit a PR.  My prototype logging spat out all the examples of CRDs hitting that 'status' error, and they're pretty easy to correlate to the codebase:
source.toolkit.fluxcd.io_buckets.yaml
source.toolkit.fluxcd.io_gitrepositories.yaml
source.toolkit.fluxcd.io_helmcharts.yaml
source.toolkit.fluxcd.io_helmrepositories.yaml
kustomize.toolkit.fluxcd.io_kustomizations.yaml
helm.toolkit.fluxcd.io_helmreleases.yaml
notification.toolkit.fluxcd.io_alerts.yaml
notification.toolkit.fluxcd.io_providers.yaml
notification.toolkit.fluxcd.io_receivers.yaml
image.toolkit.fluxcd.io_imagepolicies.yaml
image.toolkit.fluxcd.io_imagerepositories.yaml
image.toolkit.fluxcd.io_imageupdateautomations.yaml
It's actually surprisingly common with CRDs especially.  Most times I've seen it happen are part of CI/CD pipelines (or human pipelines) when CRD's are unit tested in dev environments, and then once confirmed as functional are pulled back out of the cluster with a GET.  The "status" appended by the cluster becomes part of the manifest and since the Kube API server drops it on subsequent applies, it goes mostly unnoticed and propagates either through pipeline automation or just a lack of a compelling reason or understanding to change it.  I'm happy to have Terraform be that compelling reason.  :-)

Just a brief update on this.  The PR's will take a little time and hit multiple projects, so I decided to test some hacky regex in the meantime that seems to do the trick:
resource "kubernetes_manifest" "flux_install" {
  for_each = data.kubectl_file_documents.flux_base_west.manifests
  manifest = yamldecode(replace(each.value, "/status:.*\\s +acceptedNames:.*\\s(.*\\s)*\\[\\]/", ""))
}

However, while the CRDs seem to install fine now, this uncovered a bizarre provider error that's resulting in the wrong CPU limit hitting (4) of the controller deployments.  I opened the above issue over at the Kubernetes provider to track it.

Our CRDs are generated by Kubernetes controller-gen, unless this changes in upstream we can’t remove the status from the controller repos. The status is there because without it, on older Kubernetes versions, the custom resource will not have that sub-resource, meaning that Flux will stop functioning as it needs to write to status. I think it’s ok to remove status as long as this is proven to work on all Kubernetes versions supported by Flux.

Ah, that explains a lot. Looks like there are a bunch of issues over there requesting removal, but even if they fix it now, every CRD generated by controller-gen for the last N years would have a status key.  That would make this issue prevalent enough that the kube_manifest provider probably shouldn't be blocking it. Might take the issue over there too and see if we can converge somewhere in the middle.

It would be much easier to use kubernetes_manifest if flux data sources provide Yaml documents not as a string but at least a list of strings. Would you be open to a PR from me for this? @phillebaba I guess this would enable using only kubernetes provider instead of kubectl provider

Sadly this seems like it will never work. Even with clever uses of regex to remove the status fields and break apart the multi-doc YAML (BTW, I put my version of the parsing/splitting logic at the bottom of this comment for reference), there is a blocking issue to do with Server-Side Applies and Custom Resources. Basically, you cannot apply the flux_install and flux_sync manifests in the same Terraform apply because flux_sync manifests require the GitRepository and Kustomization resource definitions to be available on the Kubernetes API at planning time. The maintainers of the hashicorp/kubernetes provider have made it fairly clear that they do not want to provide a route that does not use Server Side Applies (https://github.com/hashicorp/terraform-provider-kubernetes/issues/1367) so the only options are:

Have the Flux install and Flux sync as separately applied modules (note: you cannot separately define modules and then use them in the same project, you must run apply on two different projects one after another)
Use the kubectl provider, or another provider which does not use Server Side Applies (or allows you to bypass server-side applies or validation)
Preinstall the Custom Resource Definitions before running Terraform (note: I haven't tried this and this could cause other errors when applying the Flux install manifests)

locals {
  namespace_key = "v1/namespace/${data.flux_sync.this.namespace}"
  install_manifests = [
    for yaml in split(
      "\n---\n",
      "\n${replace(data.flux_install.this.content, "/(?m)^---[[:blank:]]*(#.*)?$/", "---")}\n"
    ) :
    replace(yaml, "/([^[:blank:]]|^)status:.*\\s(?: +[a-zA-Z]+: *.*\\s)+(?: +[a-zA-Z]+: *.*)/", "")
    if trimspace(replace(yaml, "/(?m)(^[[:blank:]]*(#.*)?$)+/", "")) != ""
  ]
  install_manifest_objects = { for value in [
    for yaml in local.install_manifests : yamldecode(yaml)
  ] : lower(join("/", compact(["${value["apiVersion"]}", "${value["kind"]}", lookup(value["metadata"], "namespace", ""), value.metadata.name]))) => value }
  install = { for key, obj in local.install_manifest_objects : key => obj if key != local.namespace_key }
  sync_manifests = [
    for yaml in split(
      "\n---\n",
      "\n${replace(data.flux_sync.this.content, "/(?m)^---[[:blank:]]*(#.*)?$/", "---")}\n"
    ) :
    replace(yaml, "/([^[:blank:]]|^)status:.*\\s(?: +[a-zA-Z]+: *.*\\s)+(?: +[a-zA-Z]+: *.*)/", "")
    if trimspace(replace(yaml, "/(?m)(^[[:blank:]]*(#.*)?$)+/", "")) != ""
  ]
  sync = { for value in [
    for yaml in local.sync_manifests : yamldecode(yaml)
  ] : lower(join("/", compact(["${value["apiVersion"]}", "${value["kind"]}", lookup(value["metadata"], "namespace", ""), value.metadata.name]))) => value }
  namespace = local.install_manifest_objects["${local.namespace_key}"]
}

@elliottpope I know that there have been so many issues with splitting yaml etc. Initially doing it this way seemed like a good idea as it required minimal code from my side. I am working on #332 which will hopefully solve all of these problems by removing all the horrible yaml decoding.

@elliottpope I know that there have been so many issues with splitting yaml etc. Initially doing it this way seemed like a good idea as it required minimal code from my side. I am working on #332 which will hopefully solve all of these problems by removing all the horrible yaml decoding.

Hey, an plans on adding an install resource? We're installing k8s + flux + specs all in a single tf run. I'm using helm install method with kubectl provider manifest to get it to work but would rather go tf native.

Hi @rafagsiqueira  👋
I hope you are doing well! As part of our ongoing effort to maintain and improve the quality of our project, I've been reviewing open issues and came across the one you've reported. First off, thank you for taking the time to contribute by reporting this issue; your input is crucial to us.
Upon reviewing the details of your issue, I noticed that it involves the use of a resource or feature that has yet to be supported since the 1.0.0 release of our project, which was approximately 9 months ago. This might be a key factor in the challenges you're experiencing. We understand that changes and deprecations can impact your work, and we're here to help navigate these transitions. If there are specific reasons you've continued using this unsupported resource or if there's any way we can assist in migrating to a supported alternative, please let us know.
Additionally, to ensure the efficient management of our issue tracker and to focus on issues that are actively affecting our community, we have implemented a policy for issues that remain inactive. If there is no activity on this issue within the next 3 weeks, we will consider the issue inactive and close it for you. This doesn't mean your issue is not important to us, but rather that we aim to keep our focus on actively pursued concerns. Of course, if the issue continues or if you have further updates in the future, feel free to reopen the issue or create a new one.
Thank you once again for your contribution to our project. Your feedback not only helps us improve but also supports the broader community in overcoming similar challenges. We look forward to hearing from you and hope to resolve any outstanding concerns together.
Best regards,
Steve

Hi @swade1987 , I am not sure how this is impacting other users who made comments here. I can speak about my experience. For the last couple of years we have been using flux cli to install and update flux, instead of the terraform approach. Just because I didn't have enough time to revisit this, and it's not very often that we need to boostrap a cluster.
I am not even sure my original issue is still relevant. I am okay with closing the issue and will revisit it when I can. I also want to thank you for working on this project. Flux is amazing, thank you so much for being a part of it.


fix(deps): update terraform terraform-ibm-modules/event-notifications/ibm to v1.15.7

This PR contains the following updates:

Package
Type
Update
Change

terraform-ibm-modules/event-notifications/ibm (source)
module
patch
1.15.6 -> 1.15.7

Release Notes

terraform-ibm-modules/terraform-ibm-event-notifications (terraform-ibm-modules/event-notifications/ibm)
v1.15.7
Compare Source
Bug Fixes

deps: update terraform-module (#353) (75cb73c)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 1.19.8 :tada:
The release is available on:

GitHub release
v1.19.8

Your semantic-release bot :package::rocket:


Fix description of the compatible-release operator

Thank you for contributing to Pipenv!
The issue
I didn't create a GitHub issue, because the proposed change is trivial.
The checklist

[ ] Associated issue
[x] A news fragment in the news/ directory to describe this fix with the extension .bugfix, .feature, .behavior, .doc. .vendor. or .trivial (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

Thanks for the patch

Thanks for the patch


golang/session: initial implementation and basic test

This is the meat of the implementation. Although I'd like better test coverage and comments, I'm particularly concerned about the use of contexts. I don't believe they are used where given or appropriate where given so I'd like to think about them more.
Also, I don't often centralize errors into predefined variables, but it might be a good idea to here?
Double check min-max limits ... there are packet sizes and data packet sizes and I'm not sure they're correctly used?
After this there is simply a transport subpackage with various transport protocol dialers and such.
If synchronous review works better, schedule a time with me!

Also is auth going to be part of qmux at all, or are we assuming a pre-authed io stream?

Auth is out of scope. Either it's handled in the underlying transport (WebSocket, for example), preceding the session (TLS client certs maybe), or in a protocol above this (maybe you're muxing HTTP, or maybe you have an RPC auth mechanism).

One other scenario that's worth keeping in mind is somekind of graceful reload support when possible, without losing state/connections. That's one thing that's particularly difficult with ssh.

That's hard in general right? What are some examples?

That's hard in general right? What are some examples?

Yea. There's lots of implementations for this in HTTP, but HTTP is fairly stateless so it's less hard. Here's a fairly general TCP implementation: https://github.com/facebookarchive/grace/blob/75cf19382434/gracenet/net.go#L206
This is probably another transport-level thing but it helps when the protocol/API doesn't get in the way.

@mgood @shazow besides removing stored contexts, and adding context to Open, I've also removed the Listener interface (for now) and dropped the LocalAddr/RemoteAddr methods from Session, since if Session is on a net.Conn, you should be able to type assert as net.Conn to get those. They don't make sense if over stdio for example.

That does look better.

I'll take that as a pass for now!


Add dependency to cairo in build files

Hi !
I was trying to build rofi-top on NixOS and saw that the dependency to cairo was missing.
I first added it as a patch for NixOS package but was asked to see if it could be upstreamed.
Can you confirm that it was missing ?

I think the problem is that rofi has a cairo include in a header file that it should not have.
In theory rofi-top should not depend on cairo.
Cleaning up the plugin provided headers that rofi provide has been on my todo list for a long time, I should really get around fixing this.

Thank you for such a fast answer!
I saw the Cairo include in rofi which causes the dependency.
If you plan to remove this I think I shall close this MR and notify this in the one for the NixOS package I made.

Since the rofi.pc already includes cairo as Requires.private it already is there as required
Could you share the build log? (maybe in a separate issue to have a cleaner track than a won’t-be-merged PR)

I think it does not work when the package is installed in another directory.
On nixos each package is installed in its own path Even if the rofi dependency is given it seems the cairo dependency is not propagated to plugins through the pc files.
I am no expert on package-config but do you think that privately required deps are propagated to plugin when one requires the rofi target ?
I'll close the MR and create an issue to request the removal of the cairo dependency if it is ok for you and if there is no issue about that already.

In short, pkg-config will pass CFLAGS from private dependencies, but not libs, which is more or less specifically there for situations such as this one, where a header dependency “leaks” in a public header
If it doesn’t work in the Nix environment, then maybe just add the cairo dependency in the Nix package, but it should not require any fix in rofi-top
Nonetheless, it will be “properly” fixed in rofi at some point by removing the header need altogether


Tests including dune_services fail for dunecore in isolation

Testing dunecore agains larsoft (instead of dunesw as is normally done), I found that tests test_ChannelStatusServiceTool and test_issue19191 fail because they use dune_services.fcl which is in dunecore, but it includes simulationservices_dune.fcl which is not in dunecore or any of its dependencies.

Jake or Tom, can you let me know if the forward dependency problem in services_dune will be fixed?
If not I will change the tests to remove the implicit dependency on simulationservices_dune.fcl.


Show amount when fee increase option is shown

Original report

Bitcoin transactions don't all have the ability to increase fee

Possibly related to #4176)

– and the one that does have it, doesn't show the amount being sent. Pete, might this be a formatting issue?

I'm not sure if we were showing the amount being sent when increasing the fee? I checked the code and it doesn't seem to be a formatting issue.
I checked back to v6.44 and I don't see it there?
I don't know what dictates when we show Increase fee and not but can look into that.
Maybe I can do that along with finishing this PR from an external - https://github.com/leather-io/extension/pull/5533 ?

Presumably we should show the amount being sent and the increase fee option in all cases.

I did some testing here and I am seeing Increase fee for every transaction I try and send:
https://github.com/user-attachments/assets/e0bd1efb-e575-4a0b-b888-a912f83c3dcd
I tried different fee amounts and a custom fee and I kept seeing it. Maybe there was some network issue when you were testing this @markmhendrickson as I know you had other issues with avatars not loading?
When I was testing I noticed that sometimes new activity didn't show up right away and then did after a hard refresh. Also sometimes it did update right away (classically once I was trying to record it being broken)
I can update it so that we always show the amount being sent above the Increase fee button. Right now we only show the amount OR then button

I opened this PR that adds the amount being sent above the Increase fee button:

Maybe there was some network issue when you were testing this @markmhendrickson as I know you had other issues with avatars not loading?

Interesting – is there code that loads data before showing the button to increase fee, and if so, is it necessary for showing the option?

Maybe there was some network issue when you were testing this @markmhendrickson as I know you had other issues with avatars not loading?

Interesting – is there code that loads data before showing the button to increase fee, and if so, is it necessary for showing the option?

I checked deeper into the code for this and there are some scenarios where we don't enable or show this button. We have a setting for isEnabled to disable the button but if it's not enabled we also don't show the button at all.
BTC
In this commit on 23/8/2023 we added hiding this button for transactions with taproot input. There is no issue in the commit so I'm not sure exactly why we did that.
For BTC we enable Increase Fee if these conditions pass:

isOriginator - !isBitcoinTxInbound(bitcoinAddress, transaction)
!transaction.status.confirmed
 !containsTaprootInput(transaction)

Some links:

https://github.com/leather-io/extension/issues/3416
https://github.com/leather-io/extension/commit/a01071fa1f1e97db9d718a4630f680429aeaad00

STX
For STX we show increase fee if:

isOriginator => transaction?.sender_address === currentAccount?.address
isPending => transaction && isPendingTx(transaction);


fix: (removing sentry temporarily): migrating sentry domain

…tch self hosted
Type of Change

[ ] Bugfix
[ ] New feature
[ ] Enhancement
[x] Refactoring
[ ] Dependency updates
[ ] Documentation
[ ] CI/CD

Description
Disabling sentry temporarily
Motivation and Context
Migrating sentry to self hosted domain
Checklist

[x] I ran npm run re:build
[x] I reviewed submitted code
[ ] I added unit tests for my changes where possible

closing this PR


Image serve view does not work on AWS due to missing headers

AWS ELB needs Content-Length header or it fails with 502 bad gateway.  The file serve views know the content-length in advance and should set them https://github.com/torchbox/wagtail/blob/master/wagtail/wagtailimages/views/frontend.py#L23

Granted this can be fixed upstream from the application but since you know the content length here might as well set it. Possibly cache headers too?


1.6.1 release

Bug fixes (JSON properties, etc).
Target date... 7/24?

Wow, here we are with no open issues and nothing to add to 1.6.1. No time like the present!

@sgillies #246 needs a small change - sorry for the miscommunication. See my comment there.


Integrate InheritDoc into the build

Summary
Adds a reference to InheritDoc to replace all /// <inheritdoc/> comments with the actual comment content in the documentation xml produced by the build.
This is a development-only dependency; it will not be deployed with or referenced by the libraries.
Note this produces a lot of warnings for <inheritdoc /> comments that need to be further refined with a cref="somBaseType" attribute. Some examples of this are included in this PR.
Example for Azure.Security.KeyVault.Administration.xml:
Before
<member name="P:Azure.Security.KeyVault.Administration.RestoreOperation.Id">
    <inheritdoc/>
</member>
<member name="P:Azure.Security.KeyVault.Administration.RestoreOperation.Value">
    <inheritdoc/>
</member>
<member name="P:Azure.Security.KeyVault.Administration.RestoreOperation.HasCompleted">
    <inheritdoc/>
</member>
<member name="P:Azure.Security.KeyVault.Administration.RestoreOperation.HasValue">
    <inheritdoc/>
</member>
<member name="M:Azure.Security.KeyVault.Administration.RestoreOperation.GetRawResponse">
    <inheritdoc/>
</member>

After
<member name="P:Azure.Security.KeyVault.Administration.RestoreOperation.Id">
    <summary>
    Gets an ID representing the operation that can be used to poll for
    the status of the long-running operation.
    </summary>
</member>
<member name="P:Azure.Security.KeyVault.Administration.RestoreOperation.Value">
    <summary>
    Final result of the long-running operation.
    </summary><remarks>
    This property can be accessed only after the operation completes successfully (HasValue is true).
    </remarks>
</member>
<member name="P:Azure.Security.KeyVault.Administration.RestoreOperation.HasCompleted">
    <summary>
    Returns true if the long-running operation completed.
    </summary>
</member>
<member name="P:Azure.Security.KeyVault.Administration.RestoreOperation.HasValue">
    <summary>
    Returns true if the long-running operation completed successfully and has produced final result (accessible by Value property).
    </summary>
</member>
<member name="M:Azure.Security.KeyVault.Administration.RestoreOperation.GetRawResponse">
    <summary>
    The last HTTP response received from the server.
    </summary><remarks>
    The last response returned from the server during the lifecycle of this instance.
    An instance of Operation<typeparamref name="T" /> sends requests to a server in UpdateStatusAsync, UpdateStatus, and other methods.
    Responses from these requests can be accessed using GetRawResponse.
    </remarks>
</member>

I presume the manual comment changes are part of testing. Other than that LGTM.

Did you notice a build speed hit from this?

Nothing significant. If, for some reason, it ever becomes an  issue, it can be conditioanlly disabled like this:
dotnet build /p:EnableInheritDoc="false"


[BUG]ValueError: [E090] Extension 'similarity' already exists on Span. To overwrite the existing extension, set force=True on Span.set_extension.

Describe the bug
Hi, I use the example in the demo.
I'm facing this error:
ValueError: [E090] Extension 'similarity' already exists on Span. To overwrite the existing extension, set force=True on Span.set_extension.
To Reproduce
common English pipeline
nlp = spacy.load('en_core_web_sm')
@Language.component('quickumls_component')
def quickumls_component(doc):
return SpacyQuickUMLS(nlp, umls_path1)(doc)
quickumls_component = SpacyQuickUMLS(nlp, umls_path1)
nlp.add_pipe('quickumls_component', last=True)
doc = nlp('Pt c/o shortness of breath, chest pain, nausea, vomiting, diarrrhea')
for ent in doc.ents:
print('Entity text : {}'.format(ent.text))
print('Label (UMLS CUI) : {}'.format(ent.label_))
print('Similarity : {}'.format(ent..similarity))
print('Semtypes : {}'.format(ent..semtypes))
**Environment **

OS: Unbuntu
QuickUMLS version 1.4.0
UMLS version 2023

Thanks :)

I am facing the same issue.

Got the same error code here.

Same issue

You define multiple SpacyQuickUMLS instances in your code:
@Language.component('quickumls_component')
def quickumls_component(doc):
quickumls_component = SpacyQuickUMLS(nlp, umls_path1)

The Span.set_extension method is called repeatedly during each instantiation, and Spacy does not allow repeated definition of the same extension attributes (like similarity and semtypes), so delete quickumls_component.


Alter stdout handling/parsing to be performed mid-job rather than after the fact

Based off Gauri's list json PR-- Alters log scanning to be performed as the job is running, rather than after the fact. This should help support any interactive functions of AzCopy in the E2E framework.

Can we resolve the conflicts on this PR and prioritize getting this merged? This will be very helpful in debugging issues with New E2E tests.


RuntimeError: remove: cannot remove from tree. get: Not in-memory or invalid offset of node.

I doing following steps:

create empty index
add many vectors to index
build index (optionally save)
sequentially remove each index and add again

Then sometime I have got following error:
RuntimeError: /NGT/release/NGT/lib/NGT/Index.h:remove:1519: remove:: cannot remove from tree. id=4923 /NGT/release/NGT/lib/NGT/Common.h:get:1943: get: Not in-memory or invalid offset of node. idx=1826234676 size=126
Error is happened not each run. Sometimes it may not appeared 10-15 times, but sometimes it is happened at each run. Also, it depends from vector type - random vector cause error on multiple last elements, but numpy.full(dims, i) has error only for one last element.
Environment:
Libs:
ngt==2.0.4
numpy==1.23.5
pybind11==2.10.1
Python: Python 3.10.8
Os: Linux 6.0.0-5-amd64 1 SMP PREEMPT_DYNAMIC Debian 6.0.10-1 (2022-11-26) x86_64 GNU/Linux (Tested on two computers with same linux and different hardware)
Example code for reproduce:
#!/usr/bin/python3
import logging
import numpy
import ngtpy

path = "/tmp/test_ngt"
dims=10

logging.root.setLevel(logging.INFO)

ngtpy.create(path, dims)
idx = ngtpy.Index(path)

# this is cause error only for one last element
# make_vector = lambda i : numpy.full(dims, i)

# this is cause error for more last elements
make_vector = lambda i : numpy.random.rand(dims)

vectors = [make_vector(i) for i in range(0, 5000)]

inserted = []

for v in vectors:
    new_id = idx.insert(v)
    inserted.append(new_id)

idx.build_index(1)
idx.save()
logging.info("Index saved, try to rewrite each elements.")

for i, v in enumerate(vectors):
    old_id = inserted[i]
    try:
        idx.remove(old_id)
    except RuntimeError as e:
        logging.exception("Can not remove old_id=%s at i=%s", old_id, i)
    idx.insert(v)

Output:
 % python3 test.py
INFO:root:Index saved, try to rewrite each elements.
ERROR:root:Can not remove old_id=4907 at i=4907
Traceback (most recent call last):
  File "/tmp/test_ngt_dir/test.py", line 35, in <module>
    idx.remove(old_id)
RuntimeError: /NGT/release/NGT/lib/NGT/Index.h:remove:1519: remove:: cannot remove from tree. id=4908 /NGT/release/NGT/lib/NGT/Common.h:get:1943: get: Not in-memory or invalid offset of node. idx=1550698841 size=126
ERROR:root:Can not remove old_id=4910 at i=4910
Traceback (most recent call last):
  File "/tmp/test_ngt_dir/test.py", line 35, in <module>
    idx.remove(old_id)
RuntimeError: /NGT/release/NGT/lib/NGT/Index.h:remove:1519: remove:: cannot remove from tree. id=4911 /NGT/release/NGT/lib/NGT/Common.h:get:1943: get: Not in-memory or invalid offset of node. idx=1549090187 size=126

So far, I have not been able to reproduce that error.
RuntimeError: /NGT/release/NGT/lib/NGT/Index.h:remove:1519: remove:: cannot remove from tree. id=4908 /NGT/release/NGT/lib/NGT/Common.h:get:1943: get: Not in-memory or invalid offset of node. idx=1550698841 size=126
ERROR:root:Can not remove old_id=4910 at i=4910

However. the error message above is quite strange. It shows that the ID of the tree node is too large and invalid. There might be a possibility that ngtpy loads a different version of the ngt library.

There might be a possibility that ngtpy loads a different version of the ngt library.

How I can check this?

So, I have tested this in Docker with "Debian stable" & "Debian testing" (with python 11 and manually build ngt) and can not reproduce this issue.
But it still reproducible at my local PC. Looks like it caused by specific versions of system libraries or something else.
Anyway I using this in Docker, and think that this issue can be closed.


deepFindKey is not working properly

the implementation for checking a key's existence is too naive, the assertion should be based on the key and not the key's value!!!

And I was thinking that deepFindKey was far better than lodash's get.   And to think i doesn't even have array support!!! Even my toaster has that!!!

yeah @kristiantoska you have a fancy toaster out there. Fancier than @andi7's lib though..


pudl-dev setup fails when installing pip dependencies

Describe the bug
When I try and install the pudl-dev environment using miniconda, I get the following error when installing pip dependencies:
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Installing pip dependencies: | Ran pip subprocess with arguments:
['A:\\miniconda3\\envs\\pudl-dev\\python.exe', '-m', 'pip', 'install', '-U', '-r', 'a:\\GitHub\\pudl\\devtools\\condaenv.5feun3q5.requirements.txt']
Pip subprocess output:
Obtaining file:///A:/GitHub/pudl (from -r a:\GitHub\pudl\devtools\condaenv.5feun3q5.requirements.txt (line 1))
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'

Pip subprocess error:
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      Collecting setuptools<63
        Using cached setuptools-62.6.0-py3-none-any.whl (1.2 MB)
      Collecting pip<23,>=21
        Using cached pip-22.1.2-py3-none-any.whl (2.1 MB)
      Collecting wheel
        Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
      Collecting setuptools_scm
        Using cached setuptools_scm-7.0.4-py3-none-any.whl (42 kB)
      Collecting packaging>=20.0
        Using cached packaging-21.3-py3-none-any.whl (40 kB)
      Collecting tomli>=1.0.0
        Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
      Collecting typing-extensions
        Using cached typing_extensions-4.3.0-py3-none-any.whl (25 kB)
      Collecting pyparsing!=3.0.5,>=2.0.2
        Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
      ERROR: To modify pip, please run the following command:
      A:\miniconda3\envs\pudl-dev\python.exe -m pip install --ignore-installed --no-user --prefix C:\Users\Greg\AppData\Local\Temp\pip-build-env-brs_7b8q\overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools<63 pip>=21,<23 wheel setuptools_scm
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

failed

CondaEnvException: Pip failed

Bug Severity
How badly is this bug affecting you?

High: This bug is preventing me from using PUDL.

To Reproduce
I am trying to install pudl-dev using from the current main branch of pudl (technically I am running from my fork of main, but I ensured that it is up to date with upstream main)

I completely removed and reinstalled miniconda using the latest Windows 64bit installer here: https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links
I ran conda update conda to update to conda 4.13.0
I ran conda config --set channel_priority strict
I ran

cd pudl
conda env create --name pudl-dev --file devtools/environment.yml

It successfully downloads and installs all of the packages, but then fails when it gets to the pip dependencies

Expected behavior
I expected the pudl-dev environment to be sucessfully set up.
Software Environment?

OS: Windows 10 Enterprise 21H1 build 19043.1766
Python version: 3.9.12
How did you install PUDL? miniconda

Additional context
I also ran into this issue originally here when updating my pudl-dev environment: https://github.com/catalyst-cooperative/pudl/pull/1608#issuecomment-1178258122

The most similar issue to this that I was able to find online was this: https://github.com/pypa/pip/issues/11120

Thanks for the pointer on the pyproject.toml / pip issue. I definitely wasn't aware that pip need not be there, and I've removed it and wheel and added a build-backend and pip seems to be happy with this arrangement now, if you want to pull the most recent version of the allocate_fuel branch.
A couple of clarifications:

You say you're working off of the main branch of PUDL, but in your PR it looked like you were merging into the allocate_fuel branch (which made sense) and I had merged all the changes in dev into allocate_fuel before this problem started... which branch is your fork based on?
You say you're using Python 3.9.12, but the pudl-dev environment requires Python 3.10.  Is 3.9.12 just the version of Python in your base conda environment, or are your somehow trying to override the requirement for Python 3.10 in the pudl-dev environment? (conda should be able to create an environment with whatever version of Python, regardless of the version in the base environment).

Some things to try:

Pull / merge the most recent version of the allocate_fuel branch with the revised pyproject.toml file.
Would you be willing to try creating the environment using mambaforge instead of miniconda? I don't think it should make a difference since the problem is coming up in the pip install, and you had strict channel priority enabled and pointing at conda-forge but we're using mambaforge to manage our conda environment so it would be good to check. You can download it here
Are you able to create a non-conda virtual environment and run the pip install of just PUDL within it? To do so you would deactivate your current conda environment and run the following commands from within the top level PUDL repo directory. Note you'll need to give it the path to a Python 3.10 interpreter, but that can be the one that's inside one of your conda environments (this is just for temporary testing). These are bash commands so... whatever the Windows equivalent is I'm not sure:

virtualenv --python /path/to/any/python3.10 venv
source venv/bin/activate
pip install -e "./[doc,test,dev,datasette]"

You should see a bunch of output from pip doing the installation, hopefully successfully...
To get out of and remove the virtual environment:
deactivate
rm -rf venv

If you just want to try revising the pyproject.toml, what I changed was the the build system section:
[build-system]
requires = [
    "setuptools<63",
    "setuptools_scm",
]
build-backend = "setuptools.build_meta"

Another note: it's not conda trying to create environments for the pre-commit hooks. Those environments are managed via pre-commit itself, and I think it uses virtualenv and pip -- creating a separate virtual environment for each hook (since they could in theory have different requirements / dependencies). The package being checked (pudl) is also installed into those environments. It's interesting that it gets through creating some of the hook environments and then barfs on doc8 apparently.
Are you able to directly install doc8 in a virtual environment using pip?

Thanks, Zane.
Responses:

You say you're working off of the main branch of PUDL, but in your PR it looked like you were merging into the allocate_fuel branch (which made sense) and I had merged all the changes in dev into allocate_fuel before this problem started... which branch is your fork based on?

When I first attempted to update the environment, I did it based on the allocate_fuel branch in my fork after merging in the upstream changes from allocate_fuel. After that failed, I thought maybe that the cause was that the environment/setup files in this branch weren't working, so I thought it would be safer to try and install the environment based on main - but obviously that also did not work

You say you're using Python 3.9.12, but the pudl-dev environment requires Python 3.10. Is 3.9.12 just the version of Python in your base conda environment, or are your somehow trying to override the requirement for Python 3.10 in the pudl-dev environment?

3.9.2 is the version of my base conda environment. The pudl-dev environment installed 3.10.5
I'll attempt some of the fixes you suggested and let you know how it goes.

So long as the allocate_fuel branch is passing CI, which you can see in the build status on PR #1096 I think we should believe that the upstream environment is working as expected (at least on Linux, which is where the CI runs).
It sounds like @arengel may have run into the same issue recently, and solved it by editing the pip requirement in devtools/environment.yml to:
  - pip>=21,<22

But I'm still curious if the changes to the build system requirements in the pyproject.toml file might fix all this even with pip v22.

Ok so after merging upstream/allocate_fuel into my branch and re-running the environment setup (still using miniconda), the setup now successfully finishes. However, I am running into the same error when I go to do a commit and it sets up the virtual environments for the pre-commit hooks.
I did install a virtual environment and tried installing doc8 by itself, which worked.
Next I'm going to try editing the pip requirement.

After editing the environment file to pip>=21,<22, I was able to successfully create the environment. However, I'm still getting an error when I go to do a commit, at the same point:
[INFO] Installing environment for https://github.com/PyCQA/doc8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('C:\\Users\\Greg\\.cache\\pre-commit\\repovo8ty0b9\\py_env-default\\Scripts\\python.EXE', '-mpip', 'install', '.')
return code: 1
expected return code: 0
stdout:
    Processing c:\users\greg\.cache\pre-commit\repovo8ty0b9
      Installing build dependencies: started
      Installing build dependencies: finished with status 'error'

stderr:
      error: subprocess-exited-with-error

      pip subprocess to install build dependencies did not run successfully.
      exit code: 1

      [20 lines of output]
      Collecting pip>=19.3.1
        Using cached pip-22.1.2-py3-none-any.whl (2.1 MB)
      Collecting setuptools>=42
        Using cached setuptools-63.1.0-py3-none-any.whl (1.2 MB)
      Collecting setuptools_scm[toml]>=3.5.0
        Using cached setuptools_scm-7.0.4-py3-none-any.whl (42 kB)
      Collecting setuptools_scm_git_archive>=1.1
        Using cached setuptools_scm_git_archive-1.4-py2.py3-none-any.whl (3.3 kB)
      Collecting wheel>=0.33.6
        Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
      Collecting packaging>=20.0
        Using cached packaging-21.3-py3-none-any.whl (40 kB)
      Collecting tomli>=1.0.0
        Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
      Collecting typing-extensions
        Using cached typing_extensions-4.3.0-py3-none-any.whl (25 kB)
      Collecting pyparsing!=3.0.5,>=2.0.2
        Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
      ERROR: To modify pip, please run the following command:
      C:\Users\Greg\.cache\pre-commit\repovo8ty0b9\py_env-default\Scripts\python.EXE -m pip install --ignore-installed --no-user --prefix C:\Users\Greg\AppData\Local\Temp\pip-build-env-ptwpbyma\overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- pip >= 19.3.1 setuptools >= 42 setuptools_scm[toml] >= 3.5.0 setuptools_scm_git_archive >= 1.1 wheel >= 0.33.6
      [end of output]

      note: This error originates from a subprocess, and is likely not a problem with pip.
    error: subprocess-exited-with-error

    pip subprocess to install build dependencies did not run successfully.
    exit code: 1

    See above for output.

    note: This error originates from a subprocess, and is likely not a problem with pip.

Looking at doc8's pyproject file, it looks like it still includes pip as well:
[build-system]
requires = [
  "pip >= 19.3.1",
  "setuptools >= 42",
  "setuptools_scm[toml] >= 3.5.0",
  "setuptools_scm_git_archive >= 1.1",
  "wheel >= 0.33.6",
]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]
local_scheme = "no-local-version"

I'm wondering if this is the issue and needs to be submitted as an issue to them?

Ahhh, yeah I bet that's it.
So it sounds like modifying our build requirements fixed the problem on our side -- that the pudl-dev environment creates fine now even with the more recent version of pip and the main issue is probably that doc8 is also exercising the same problem?
I guess for now on your branch just comment out the offending pre-commit hook and we can re-enable it eventually when things get merged in upstream.

Would I just need to comment out doc8 in setup.py, or do you think it will also be necessary to remove from tox.ini and .pre-commit-config.yaml?

I think / hope that you would only need to comment out the doc8 pre-commit hook definition in .pre-commit-config.yaml -- it seems to be getting installed just fine when the conda environment is created (which installs the pudl package along with all of its associated dependencies).
Speaking of which what python environment do you have activated when you're trying to install the pre-commit hooks? Is it the pudl-dev conda environment? Or the base conda environment? Or just your system python?

I installed the pre-commit hooks with my pudl-dev environment activated.
Commenting out doc8 from .pre-commit-config.yaml seems to have did the trick - everything is working now!

Some Hub team members also had issues with pre-commit that produced a similar error message, I was able to fix those issues by adding the setting below to the top of .pre-commit-config.yaml, and then conforming the language version for black as well (or removing it). I didn't try to figure out what was actually going on but specifying the python version in the pre-commit config got it to work.
default_language_version:
    python: python3.10 # or whatever python version you want

I also ran pre-commit clean to get rid of all the old pre-commit virtual environments just in case, though not sure that was necessary. I thought maybe the pre-commit python version had to be the same as the one of the environment you install pre-commit from but I don't think that's right since it seems like pre-commit sets up its own virtual environments.
This issue on pre-commit addresses something apparently similar though supposedly resolved, which is where I got the idea to see if this setting fixed things, which it has. For now.

Okay, it sounds like this issue can be closed now, since the doc8 problem is out of our control (but hopefully they fix it).. Do you agree @grgmiller?

Yes, I agree that this can be closed.
I've created an issue on the doc8 repo https://github.com/PyCQA/doc8/issues/91, and it sounds like hopefully the original issue that is causing this in pip is hopefully going to be fixed as well https://github.com/pypa/pip/issues/11120#issuecomment-1179448615.
For now, running the pre-commit hooks and tox will be non-functional for windows users, but maybe I'm the only windows user of pudl who is doing any development, and I currently have a workaround.

I think you're the only windows developer who isn't using WSL. I think make it work for tox so the CI passes, and if only the expected pre-commit hooks fail when you try to commit, you can re-run the commit with --no-verify and it'll bypass the hooks.

@zaneselvans This has now been fixed in the latest release of doc8 (https://github.com/PyCQA/doc8/releases/tag/v1.0.0) so once pudl updates its doc8 dependency to v1.0 this should hopefully no longer be an issue.

Okay, that should happen automatically as soon as the dependabot picks up the version update!


Flask docs need rebuilding

d3d8a4694a9c1601e213d6b5f27fab34615609b7 hasn't shown up yet here: http://flask.pocoo.org/docs/dev/extensions/
I'm not sure how to trigger a rebuild... on RTD, anyone can trigger a rebuild from the web, but since it's hosted on Pocoo I'm not sure if there's a webhook that watches the repo for commits, or if @mitsuhiko has to manually do it.

Would be nice if the docs were just hosted on read the docs instead.

I tried to manually trigger a build but something is wrong. I will investigate that in detail later.

Would be nice if the docs were just hosted on read the docs instead.

Indeed, this would be fixed by https://github.com/pallets/meta/issues/15


[VS 15] Navigate To / Find all refs / Rename works only on opened projects if "Lightweight solution load" feature is enabled

This feature makes VS to not load any projects when a solution is loaded. Solution loading is super fast, but Navigate to and Find all references (I suspect solution wide "Rename" either) do not work on all projects in solution, them see only those which have been "activating" (i.e. open in Solution Explorer).
I thing it's a huge problem.

We are not going to support VS 2017.


yt_dlp (Python implementation) can't recognize language code for subtitles (i.e. 'em', 'fr', etc.)

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm reporting a bug unrelated to a specific site
[X] I've verified that I'm running yt-dlp version 2023.01.06 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood
You can specify the language (i.e. 'en', 'fr', etc.) and download the correct subtitle file using yt-dlp from the command line, but you can't do the same using yt-dlp in Python. yt_dlp in Python can't recognize language codes.
Here is 2 examples:

Check subtitles:
import yt_dlp
link = ['https://www.arte.tv/fr/videos/108558-001-A/comment-voulons-nous-aimer/']
ydl_opts = {'skip_download': True, 'listsubtitles': True}
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
ydl.download(link)

here is output:
(env_video_dnld) pavel@MISSURI:~/env_video_dnld$ python check_subtitles.py
[ArteTV] Extracting URL: https://www.arte.tv/fr/videos/108558-001-A/comment-voulons-nous-aimer/
[ArteTV] 108558-001-A: Downloading JSON metadata
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[info] Available subtitles for 108558-001-A:
Language Formats
fr       vtt
de       vtt
de_sdh   vtt
en       vtt
es       vtt
pl       vtt
it       vtt

Try to download English subtitles:
import yt_dlp
link = ['https://www.arte.tv/fr/videos/108558-001-A/comment-voulons-nous-aimer/']
ydl_opts = {'skip_download': True, 'subtitleslangs': 'en', 'writesubtitles': True}
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
ydl.download(link)

here is output:
(env_video_dnld) pavel@MISSURI:~/env_video_dnld$ python check_subtitles.py
[ArteTV] Extracting URL: https://www.arte.tv/fr/videos/108558-001-A/comment-voulons-nous-aimer/
[ArteTV] 108558-001-A: Downloading JSON metadata
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[ArteTV] 108558-001-A: Downloading m3u8 information
[info] 108558-001-A: Downloading 1 format(s): VF-STF-2046+VF-STF-program_audio_0-VF
[info] There's no subtitles for the requested languages
(env_video_dnld) pavel@MISSURI:~/env_video_dnld$
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
(env_video_dnld) pavel@MISSURI:~/env_video_dnld$ yt-dlp -vU
[debug] Command-line config: ['-vU']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.01.06 [6becd25] (pip)
[debug] Python 3.10.6 (CPython x86_64 64bit) - Linux-5.15.0-60-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2
[debug] Optional libraries: Cryptodome-3.16.0, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1760 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2023.01.06, Current version: 2023.01.06
yt-dlp is up to date (2023.01.06)
(env_video_dnld) pavel@MISSURI:~/env_video_dnld$

It should be a list of languages ['en'], not a single string
https://github.com/yt-dlp/yt-dlp/blob/9ebac35577e61c3d25fafc959655fa3ab04ca7ef/yt_dlp/YoutubeDL.py#L278-L281

Thanks. Works !
Is there a function, which allows to get the list of all available subtitles.
Or the only option is to use extract_info, then search in the info dictionary the "subtitles' key, then get all keys from the "subtitles' dictionary ?
Thanks in advance.

Or the only option is to use extract_info, then search in the info dictionary the "subtitles' key, then get all keys from the "subtitles' dictionary ?

Yes. Why would a separate function be needed?

In order to facilitate analysis of available subtitles.
Here is example.
Download french subtitles. If 'fr_sdh' is available, take it, and download, otherwise download 'fr' (if it available), otherwise nothing.
I have also question concerning rendering subtitles in video (selecting font, color, etc.). Should I create another issue for that ?

In order to facilitate analysis of available subtitles.
Here is example.
Download french subtitles. If 'fr_sdh' is available, take it, and download, otherwise download 'fr' (if it available), otherwise nothing.

You can override ydl.process_subtitles
class MyYDL(YoutubeDL):
    def process_subtitles(self, video_id, normal_subtitles, automatic_captions):
        if normal_subtitles.get('fr_sdh'):
            return {'fr_sdh': normal_subs['fr_sdh'][-1]}
        elif normal_subtitles.get('fr_sdh'):
            return {'fr': normal_subs['fr'][-1]}

with MyYDL(...) as ydl:
    ...

If you think there should be an easier way, pls open a "feature request" explaining your proposal

I have also question concerning rendering subtitles in video (selecting font, color, etc.). Should I create another issue for that ?

You can ask here, but yt-dlp don't edit the subtitles in any way and so has no control over how it's rendered. If you wish to edit it, you'd have to either write a subtitle parser or use a third party program

Well, first of all I couldn't find a Python option that corresponds to the command-line option --embed-subs

Understood. 'writesubtitles': True corresponds to --embed-subs
Does exist an option just to download subtitles, NOT embed them into video ?

'writesubtitles': True corresponds to --embed-subs

No. It corresponds to --write-sub. --embed-subs is a postprocessor.
https://github.com/yt-dlp/yt-dlp/issues/5859#issuecomment-1363938900 will help you

Here is how subtitles look in original video playing on the website:
https://i.postimg.cc/2SN9QqbK/Screenshot-from-2023-02-16-16-16-31.png
and here is how it is rendered by yt-dlp:
https://i.postimg.cc/Y01nZkT3/Screenshot-from-2023-02-16-16-13-19.png
Is it possible to have the same style
When I specify  'writesubtitles': True in yt-dlp options in Python script, the subtitles are embedded into video ... I've just checked it.
Does exist an option for Python script allowing just to download subtitles as .vtt file, NOT embed them ?

Here is how subtitles look in original video playing on the website: i.postimg.cc/2SN9QqbK/Screenshot-from-2023-02-16-16-16-31.png and here is how it is rendered by yt-dlp: i.postimg.cc/Y01nZkT3/Screenshot-from-2023-02-16-16-13-19.png Is it possible to have the same style

It's the player that's setting the style

When I specify 'writesubtitles': True in yt-dlp options in Python script, the subtitles are embedded into video ... I've just checked it.

No, it does not. Delete the file you have already downloaded, run yt-dlp again, check if it's embedded and provide verbose logs if it is

Yes, it's the case.
Thanks !


Must i wait for supporting windows?

Hi @meff34, I don't have any resources to support Windows sorry. You are welcome to contribute any changes required though.
Closing this for now. Feel free to reopen if there are any more questions.

Thanks for answer! :+1:


Right to Left tabs

Hello there.
Thanks for your great work.
Is there any solution to use tabs in RTL direction?
There are two problems to resolve:
1- Tabs direction
2- Arrow keys functionality
The first problem is resolvable using CSS.
But the second one is the real problem.
If you want to make a quick test, replace <html> with this line:
<html lang="fa" dir="rtl">
This resolves the first problem.

I'm looking into this. I have it partly working right now.

OK, so I think v2.1.0 will support RTL, but this is my first foray into this area so I could be wrong. Let me know if it doesn't work in your real-life application.
Details are in the README here.
And you might also want to enable option reverseScroll(details here) to make it more intuitive for your situation, but I'm not sure.
Thanks for using my little plugin! 😃

Well man, thank you! The RTL support works like a charm.
In my case it also needs to add reverseScroll, like you said.
Thank you 👍


TankLevel does not report temperature

New code plan.  I want to add a DS18B20 to this module.

Checked Into Develop

https://github.com/au-chrismor/datafarm/pull/8


First draft

We're still pretty new to Go, so let's make a first draft that hews as closely to the way the original scripts work as we can.

Listing some libraries I can leverage here.

Best Markdown parser in Goworld: goldmark is actively developed and very popular.
Translation library: GCP rolls their own!

I think that's all I need for a first draft, actually.


niri: init at 0.1.1

https://github.com/YaLTeR/niri
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Currently has some issues, working on fixing it

Works as intended now!

[x] reviewed the diff and commit messages
[x] made sure ofBorg build succeeded for all applicable platforms
[x] run nix-review for a reasonable amount of time without any failures (or marked preexisting failures as broken) - tried running as a windowed application for a bit, and this appears to function similarly as the flake one i had been running

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/1449

Thank you!


iOS not registered operator

SSD Mobilenet v1
TFException: Op type not registered 'NonMaxSuppressionV2'
  at TensorFlow.TFGraph.Import (System.Byte[] buffer, System.String prefix, TensorFlow.TFStatus status) [0x00000] in <00000000000000000000000000000000>:0
  at TensorFlow.TFGraph.Import (System.Byte[] buffer, TensorFlow.TFImportGraphDefOptions options, TensorFlow.TFStatus status) [0x00000] in <00000000000000000000000000000000>:0
  at TensorFlow.TFGraph.Import (System.Byte[] buffer, System.String prefix, TensorFlow.TFStatus status) [0x00000] in <00000000000000000000000000000000>:0
  at PhoneCamera.LoadTFWorker () [0x00000] in <00000000000000000000000000000000>:0
  at PhoneCamera.Start () [0x00000] in <00000000000000000000000000000000>:0

But, Extracting APK(Android) in unity project, it correctly works.
It seems deeply involved, Plugins folder under iOS folder libtensorflow-core.a (not contain NMSV2 ops)
how to make libtensorflow-core.a lately?
or how to resolve this issue?

I dont knw where you got this tensorflow from, I do not ship the native binaries for it.  Perhaps Unity?   In that case, it might be best to report teh issue there.


Bookshelf SignatureDoesNotMatch Errors With Multiple Backends

When more than one chef server is running behind an ELB, Bookshelf complains of SignatureDoesNotMatch when clients are downloading cookbooks.  This happens when in "sql" mode.  The fix is to ensure
bookshelf['access_key_id']
bookshelf['secret_access_key']
are the same on every backend server during bootstrap.  If these settings are not set a random string is generated so each backend is expecting a different keypair.
chef-server-ctl set-secret bookshelf access_key_id
and
chef-server-ctl set-secret bookshelf secret_access_key
need to be run at bootstrap time.  The values could be stored in aws ssm for easy access.

Interesting, I've never seen this issue across probably hundreds of AWS chef server deployments.  Could you please provide more information?  (all package versions involved, did you make any modifications to this template, etc)

@irvingpop Thank you for the quick reply.  You can close this ticket.  I realized that we were not keeping /etc/opscode in sync across all of the chef servers.  Doing this will fix all of our secret mismatch issues.


envoy filter: warn about instability on usage

This is just a small QOL adjustment to ensure user are appropriately
aware of the risks involved with EnvoyFilter usage. This is just a
warning and does not block any usage.

@howardjohn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
359fd3c5d725c4f8b990bc398e1f56d636e96b36
link
true
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


could not find lighthouse hmac token lighthouse-hmac-token in namespace jx: secrets "lighthouse-hmac-token" not found

Summary
what i did is follow the guide "https://github.com/jx3-gitops-repositories/jx3-terraform-gke" to create a jx cluster
Steps to reproduce the behavior
follow this guide to create jx cluster
Expected behavior
"jx boot" and "jx gitops webhook update --warn-on-fail" ends successfully
Actual behavior
I got below error
waiting for the Git Operator to be ready in namespace jx-git-operator...
pod jx-git-operator-5c6bcd66fc-qh2md has status Ready
the Git Operator is running in pod jx-git-operator-5c6bcd66fc-qh2md

waiting for boot Job pod with selector app=jx-boot in namespace jx-git-operator...
waiting for Job jx-boot-6a5d6539-43c4-4446-b07f-706c432f20b9 to complete...
pod jx-boot-6a5d6539-43c4-4446-b07f-706c432f20b9-cz7lv has status Ready

tailing boot Job pod jx-boot-6a5d6539-43c4-4446-b07f-706c432f20b9-cz7lv

jx gitops git setup
about to run: git config --global --add user.name xxx-deploy
about to run: git config --global --add user.email<EMAIL_ADDRESS>about to run: git config --global credential.helper store in dir /home
Generated Git credentials file /workspace/xdg_config/git/credentials
jx gitops apply
found last commit message: chore: regenerated

/pipeline cancel
last commit disabled further processing
# NOTE be very careful about these 2 labels as getting them wrong can remove stuff in you cluster!
kubectl apply --force --prune -l=gitops.jenkins-x.io/pipeline=customresourcedefinitions -R -f config-root/customresourcedefinitions
customresourcedefinition.apiextensions.k8s.io/environments.jenkins.io configured
customresourcedefinition.apiextensions.k8s.io/pipelineactivities.jenkins.io configured
customresourcedefinition.apiextensions.k8s.io/releases.jenkins.io configured
customresourcedefinition.apiextensions.k8s.io/sourcerepositories.jenkins.io configured
customresourcedefinition.apiextensions.k8s.io/previews.preview.jenkins.io unchanged
customresourcedefinition.apiextensions.k8s.io/lighthousejobs.lighthouse.jenkins.io unchanged
customresourcedefinition.apiextensions.k8s.io/externalsecrets.kubernetes-client.io unchanged
customresourcedefinition.apiextensions.k8s.io/vaults.vault.banzaicloud.com unchanged
customresourcedefinition.apiextensions.k8s.io/clustertasks.tekton.dev configured
customresourcedefinition.apiextensions.k8s.io/conditions.tekton.dev unchanged
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/pipelineresources.tekton.dev unchanged
customresourcedefinition.apiextensions.k8s.io/pipelineruns.tekton.dev configured
customresourcedefinition.apiextensions.k8s.io/pipelines.tekton.dev configured
customresourcedefinition.apiextensions.k8s.io/runs.tekton.dev configured
customresourcedefinition.apiextensions.k8s.io/taskruns.tekton.dev configured
customresourcedefinition.apiextensions.k8s.io/tasks.tekton.dev configured
kubectl apply --force --prune -l=gitops.jenkins-x.io/pipeline=cluster                   -R -f config-root/cluster
clusterrole.rbac.authorization.k8s.io/jx-build-controller-jx unchanged
clusterrolebinding.rbac.authorization.k8s.io/jx-build-controller-jx unchanged
clusterrole.rbac.authorization.k8s.io/jx-pipelines-visualizer unchanged
clusterrolebinding.rbac.authorization.k8s.io/jx-pipelines-visualizer unchanged
clusterrole.rbac.authorization.k8s.io/jx-preview-gc-jobs unchanged
clusterrolebinding.rbac.authorization.k8s.io/jx-preview-gc-jobs unchanged
clusterrole.rbac.authorization.k8s.io/gcactivities-jx unchanged
clusterrolebinding.rbac.authorization.k8s.io/gcactivities-jx unchanged
clusterrole.rbac.authorization.k8s.io/tekton-bot unchanged
clusterrolebinding.rbac.authorization.k8s.io/tekton-bot-jx unchanged
namespace/jx-production configured
namespace/jx-staging configured
namespace/jx unchanged
namespace/nginx configured
namespace/secret-infra configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-external-secrets-auth unchanged
clusterrole.rbac.authorization.k8s.io/kubernetes-external-secrets unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-external-secrets unchanged
clusterrole.rbac.authorization.k8s.io/pusher-wave-pusher-wave unchanged
clusterrolebinding.rbac.authorization.k8s.io/pusher-wave-pusher-wave unchanged
clusterrolebinding.rbac.authorization.k8s.io/vault-auth-delegator unchanged
clusterrole.rbac.authorization.k8s.io/vault-operator unchanged
clusterrolebinding.rbac.authorization.k8s.io/vault-operator unchanged
clusterrole.rbac.authorization.k8s.io/tekton-aggregate-edit unchanged
clusterrole.rbac.authorization.k8s.io/tekton-aggregate-view unchanged
clusterrole.rbac.authorization.k8s.io/tekton-pipelines-controller-cluster-access unchanged
clusterrolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller-cluster-access unchanged
clusterrolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller-leaderelection unchanged
clusterrole.rbac.authorization.k8s.io/tekton-pipelines-controller-tenant-access unchanged
clusterrolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller-tenant-access unchanged
clusterrole.rbac.authorization.k8s.io/tekton-pipelines-leader-election unchanged
namespace/tekton-pipelines unchanged
clusterrole.rbac.authorization.k8s.io/tekton-pipelines-webhook-cluster-access unchanged
clusterrolebinding.rbac.authorization.k8s.io/tekton-pipelines-webhook-cluster-access unchanged
clusterrolebinding.rbac.authorization.k8s.io/tekton-pipelines-webhook-leaderelection unchanged
kubectl apply --force --prune -l=gitops.jenkins-x.io/pipeline=namespaces                -R -f config-root/namespaces
deployment.apps/jenkins-x-chartmuseum configured
persistentvolumeclaim/jenkins-x-chartmuseum unchanged
externalsecret.kubernetes-client.io/jenkins-x-chartmuseum unchanged
service/jenkins-x-chartmuseum unchanged
release.jenkins.io/jenkins-x-crds-3.0.5 configured
serviceaccount/jenkins-x-controllerbuild unchanged
deployment.apps/jx-build-controller configured
rolebinding.rbac.authorization.k8s.io/jx-build-controller unchanged
role.rbac.authorization.k8s.io/jx-build-controller unchanged
deployment.apps/jx-pipelines-visualizer configured
ingress.networking.k8s.io/jx-pipelines-visualizer unchanged
serviceaccount/jx-pipelines-visualizer unchanged
service/jx-pipelines-visualizer unchanged
release.jenkins.io/jx-preview-0.0.133 configured
cronjob.batch/jx-preview-gc-jobs unchanged
rolebinding.rbac.authorization.k8s.io/jx-preview-gc-jobs unchanged
role.rbac.authorization.k8s.io/jx-preview-gc-jobs unchanged
serviceaccount/jx-preview-gc-jobs unchanged
ingress.networking.k8s.io/chartmuseum unchanged
release.jenkins.io/gcactivities-2.0.1143 configured
rolebinding.rbac.authorization.k8s.io/gcactivities unchanged
role.rbac.authorization.k8s.io/gcactivities unchanged
cronjob.batch/jxboot-helmfile-resources-gcactivities unchanged
serviceaccount/jxboot-helmfile-resources-gcactivities unchanged
release.jenkins.io/gcpods-2.0.1143 configured
rolebinding.rbac.authorization.k8s.io/gcpods unchanged
role.rbac.authorization.k8s.io/gcpods unchanged
cronjob.batch/jxboot-helmfile-resources-gcpods unchanged
serviceaccount/jxboot-helmfile-resources-gcpods unchanged
role.rbac.authorization.k8s.io/committer unchanged
environment.jenkins.io/dev unchanged
sourcerepository.jenkins.io/dev unchanged
ingress.networking.k8s.io/hook unchanged
configmap/ingress-config unchanged
externalsecret.kubernetes-client.io/jenkins-docker-cfg unchanged
externalsecret.kubernetes-client.io/jenkins-maven-settings unchanged
externalsecret.kubernetes-client.io/jenkins-release-gpg unchanged
configmap/jenkins-x-devpod-config unchanged
configmap/jenkins-x-docker-registry unchanged
configmap/jenkins-x-extensions unchanged
externalsecret.kubernetes-client.io/jx-basic-auth-htpasswd unchanged
externalsecret.kubernetes-client.io/jx-basic-auth-user-password unchanged
role.rbac.authorization.k8s.io/jx-pipeline-activity-updater unchanged
role.rbac.authorization.k8s.io/jx-view unchanged
ingress.networking.k8s.io/nexus unchanged
role.rbac.authorization.k8s.io/owner unchanged
environment.jenkins.io/production unchanged
environment.jenkins.io/staging unchanged
rolebinding.rbac.authorization.k8s.io/tekton-bot unchanged
role.rbac.authorization.k8s.io/tekton-bot unchanged
serviceaccount/tekton-bot configured
externalsecret.kubernetes-client.io/tekton-container-registry-auth unchanged
externalsecret.kubernetes-client.io/tekton-git unchanged
role.rbac.authorization.k8s.io/viewer unchanged
service/hook unchanged
deployment.apps/lighthouse-foghorn unchanged
rolebinding.rbac.authorization.k8s.io/lighthouse-foghorn unchanged
role.rbac.authorization.k8s.io/lighthouse-foghorn unchanged
serviceaccount/lighthouse-foghorn unchanged
cronjob.batch/lighthouse-gc-jobs unchanged
rolebinding.rbac.authorization.k8s.io/lighthouse-gc-jobs unchanged
role.rbac.authorization.k8s.io/lighthouse-gc-jobs unchanged
serviceaccount/lighthouse-gc-jobs unchanged
externalsecret.kubernetes-client.io/lighthouse-hmac-token unchanged
deployment.apps/lighthouse-keeper unchanged
rolebinding.rbac.authorization.k8s.io/lighthouse-keeper unchanged
role.rbac.authorization.k8s.io/lighthouse-keeper unchanged
serviceaccount/lighthouse-keeper unchanged
service/lighthouse-keeper unchanged
externalsecret.kubernetes-client.io/lighthouse-oauth-token unchanged
deployment.apps/lighthouse-tekton-controller configured
rolebinding.rbac.authorization.k8s.io/lighthouse-tekton-controller unchanged
role.rbac.authorization.k8s.io/lighthouse-tekton-controller unchanged
serviceaccount/lighthouse-tekton-controller unchanged
service/lighthouse-tekton-controller unchanged
deployment.apps/lighthouse-webhooks unchanged
rolebinding.rbac.authorization.k8s.io/lighthouse-webhooks unchanged
role.rbac.authorization.k8s.io/lighthouse-webhooks unchanged
serviceaccount/lighthouse-webhooks unchanged
configmap/config configured
configmap/jx-install-config unchanged
pipeline.tekton.dev/jx-meta-pipeline unchanged
configmap/plugins configured
configmap/nexus unchanged
deployment.apps/nexus-nexus configured
persistentvolumeclaim/nexus-nexus unchanged
externalsecret.kubernetes-client.io/nexus unchanged
service/nexus unchanged
externalsecret.kubernetes-client.io/tekton-container-registry-auth unchanged
externalsecret.kubernetes-client.io/tekton-container-registry-auth unchanged
service/ingress-nginx-controller-admission unchanged
configmap/ingress-nginx-controller configured
deployment.apps/ingress-nginx-controller configured
poddisruptionbudget.policy/ingress-nginx-controller unchanged
service/ingress-nginx-controller unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
role.rbac.authorization.k8s.io/ingress-nginx unchanged
serviceaccount/ingress-nginx unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
job.batch/ingress-nginx-admission-create unchanged
job.batch/ingress-nginx-admission-patch unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
role.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
serviceaccount/ingress-nginx-admission unchanged
deployment.apps/kubernetes-external-secrets unchanged
serviceaccount/kubernetes-external-secrets unchanged
service/kubernetes-external-secrets unchanged
release.jenkins.io/pusher-wave-0.4.12 configured
deployment.apps/pusher-wave-pusher-wave configured
serviceaccount/pusher-wave-pusher-wave unchanged
persistentvolumeclaim/vault-file unchanged
serviceaccount/vault unchanged
rolebinding.rbac.authorization.k8s.io/vault-secrets unchanged
role.rbac.authorization.k8s.io/vault-secrets unchanged
vault.vault.banzaicloud.com/vault unchanged
deployment.apps/vault-operator configured
serviceaccount/vault-operator unchanged
service/vault-operator unchanged
configmap/config-artifact-bucket unchanged
configmap/config-artifact-pvc unchanged
configmap/config-defaults unchanged
configmap/config-leader-election unchanged
configmap/config-logging unchanged
configmap/config-observability unchanged
configmap/config-registry-cert unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.pipeline.tekton.dev unchanged
configmap/feature-flags unchanged
serviceaccount/tekton-bot configured
externalsecret.kubernetes-client.io/tekton-container-registry-auth unchanged
externalsecret.kubernetes-client.io/tekton-git unchanged
deployment.apps/tekton-pipelines-controller unchanged
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller unchanged
role.rbac.authorization.k8s.io/tekton-pipelines-controller unchanged
serviceaccount/tekton-pipelines-controller unchanged
service/tekton-pipelines-controller unchanged
podsecuritypolicy.policy/tekton-pipelines configured
deployment.apps/tekton-pipelines-webhook unchanged
horizontalpodautoscaler.autoscaling/tekton-pipelines-webhook unchanged
poddisruptionbudget.policy/tekton-pipelines-webhook unchanged
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-webhook unchanged
role.rbac.authorization.k8s.io/tekton-pipelines-webhook unchanged
serviceaccount/tekton-pipelines-webhook unchanged
service/tekton-pipelines-webhook unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.pipeline.tekton.dev unchanged
secret/webhook-certs unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.pipeline.tekton.dev unchanged
# lets apply any infrastructure specific labels or annotations to enable IAM roles on ServiceAccounts etc
jx gitops postprocess
there is no post processing Secret jx-post-process in namespace default so not performing any additional post processing steps
changing to the jx namespace to verify
jx ns jx --quiet
Now using namespace 'jx' on server ''.
jx verify ingress --ingress-service ingress-nginx-controller
now verifying docker registry ingress setup
jx gitops webhook update --warn-on-fail
Error: failed to validate options: failed to find hmac token from secret: could not find lighthouse hmac token lighthouse-hmac-token in namespace jx: secrets "lighthouse-hmac-token" not found
Usage:
  update [flags]

Examples:
  # update all the webhooks for all SourceRepository and Environment resource:
  jx-gitops update

  # only update the webhooks for a given owner
  jx-gitops update --org=mycorp

  # use a custom hook webhook endpoint (e.g. if you are on premise using node ports or something)
  jx-gitops update --endpoint http://mything.com

Flags:
  -b, --batch-mode                 Runs in batch mode without prompting for user input
      --endpoint string            Don't use the endpoint from the cluster, use the provided endpoint
      --exact-hook-url-match       Whether to exactly match the hook based on the URL (default true)
      --git-kind string            the kind of git server to connect to
      --git-server string          the git server URL to create the scm client
      --git-token string           the git token used to operate on the git repository. If not specified it's loaded from the git credentials file
      --git-username string        the git username used to operate on the git repository. If not specified it's loaded from the git credentials file
  -h, --help                       help for update
      --hmac string                Don't use the HMAC token from the cluster, use the provided token
      --log-level string           Sets the logging level. If not specified defaults to $JX_LOG_LEVEL
  -o, --owner string               The name of the git organisation or user to filter on
      --previous-hook-url string   Whether to match based on an another URL
  -r, --repo string                The name of the repository to filter on
      --verbose                    Enables verbose output. The environment variable JX_LOG_LEVEL has precedence over this flag and allows setting the logging level to any value of: panic, fatal, error, warn, info, debug, trace
      --warn-on-fail               If enabled lets just log a warning that we could not update the webhook

error: failed to validate options: failed to find hmac token from secret: could not find lighthouse hmac token lighthouse-hmac-token in namespace jx: secrets "lighthouse-hmac-token" not found
make: *** [versionStream/src/Makefile.mk:145: verify] Error 1
boot Job pod jx-boot-6a5d6539-43c4-4446-b07f-706c432f20b9-cz7lv has Failed

Jx version
The output of jx version is:
jx version
version: 3.1.15

Diagnostic information
The output of jx diagnose version is:
% jx diagnose version
Error: unknown command "diagnose" for "jx"
Run 'jx --help' for usage.

Kubernetes cluster

gke 1.16.15-gke.4300
and i created it by following the guide  "https://github.com/jx3-gitops-repositories/jx3-terraform-gke", using terraform.
Kubectl version
The output of kubectl version --client is:
kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"darwin/amd64"}

Operating system / Environment

MacOS Big Sur version 11.0 Beta

@jstrachan @rawlingsj @MarckK @borntorock can any available to help me please on this ?

Seeing the same issue.

don't know what went wrong. I repull code from this github repository and also https://github.com/jx3-gitops-repositories/jx3-gke-vault. My issue is fixed successfully.