No in-call audio after first call

When I make a call for the first time after my app launches everything works great. I am able to connect to the other end and I can hear audio on both sides. However, if I end the call and then try to call back, then I do not hear any audio while inside the call. I am able to connect on both ends just as I would expect, but I just hear no audio. Also, all of the provider delegates that we respond to to start the audio are in fact called.
My question here is should I be calling TwilioVoice.sharedInstance().stopAudioDevice() within the provider(_ provider: CXProvider, perform action: CXEndCallAction) delegate and call TwilioVoice.sharedInstance().startAudioDevice() within the provider(_ provider: CXProvider, didActivate audioSession: AVAudioSession) delegate every time we make a call?
Thanks

Hi @miken01
I am sorry that you are experiencing the audio problem with the SDK.
Could you provide us the Voice SDK logs when the audio problem is happening in you app?
And yes, you will have to call the startAudioDevice and stopAudioDevice methods at appropriate timing since the SDK needs to properly deactivate the audio device and re-configure it next time the call is established, as the CallKit framework expects.
Let me know if this helps.
-bobie
Twilio SDK Engineering
9am - 6pm Pacific Time

Hi Bobbie,
Yes I'd be happy to send those logs your way, but I do not see any such log in my documents or library directory of the app. Is there a setting I need to enable in order to generate the log file you are requesting?
Thanks

Hi @miken01
Don't worry. Could you add this line in you application and make sure it gets executed before making/accepting calls?
  TwilioVoice.sharedInstance().logLevel = .verbose

This way the SDK will start print out debug logs into Xcode console window and you can attach it in case you run into trouble using the SDK in the future.
Thanks
-bobie

Hi Bobbie,
I have enabled verbose logging and have replicated the issue. I have attached the log. Many thanks for the help!
Twilio Verbose.txt

Hi @miken01
Thanks for getting the log for us the SDK logs and I've checked it though I could find any immediate errors in it indicating the no-audio problem that you described.
I will need to ask you a couple of questions:

Voice SDK version: is it the latest beta9?
just want to make sure, but could you try the vanilla version of the Swift quickstart and see if this is still happening in your environment (device, network, etc)?
I saw there are four calls in the log you provided. which among them had the no-audio issue?
not sure if I missed this, but could you also tell me who is on the other end of the call - is it another mobile or JavaScript client using Twilio Voice SDK or is it a PSTN number?

Thanks
-bobie

2.0.0-beta9

I haven't had a chance to test this exact issue on the Swift Quickstart. One thing to note is I have copied the code nearly verbatim and verified all areas surrounding the SDK are the exact same. Down to the comments talking about the iOS issue where audio doesn't always start correctly. The only deviations are areas where I grab data unique to my app such as the recipients name and VoIP identity.

The last two had the no audio issue. The first two worked correctly. Usually, this occurs after the first call is completed, but this case the second call worked. The third and forth did not.

When running this test I have two iPhones in my hands, and I put them both to my ear to hear for audio. In these scenarios the caller is an iPhone 7 Plus running iOS 10.3.1 and the recipient is an iPhone 6 running iOS 10.2.1.

Thanks

Hi Bobie,
I just wanted to check to see if you've been able to replicate the issue and/or come up with a solution? My client is getting very agitated as we're waiting on this to be resolved to push to the App Store. Any help you can provide here would be greatly appreciated.
Thanks!

Hi @miken01
We've pulled in the ticket to investigate the no-audio issue into this week.
Sorry for keeping you waiting and please bear with us.
Will keep you updated once we have anything findings.
-bobie

Hi @bchen-twilio
Do you have any updates for this issue and an estimate as to when it'll be resolved? As I stated before we have an app that is ready for production and just waiting for this issue to be resolved before submitting to the App Store.
Also, have you guys herd of anyone else having an issue like this before? I just feel that if this is a legitimate issue with the SDK then other developers would be having the same exact issue. Makes me concerned I'm just missing something in my configuration, but every time I review the code it matches the quick start project exactly.
Thanks

Hi @miken01
Sorry for keeping you waiting, but unfortunately we still have hard trying to reproduce and debug the no-audio issue. Though we are not seeing other SDK users reporting exactly the same symptom and steps, but we definitely are seeing similar no-audio issues being reported by SDK users once in a while and we do treat this with the highest priority.
I've been checking the log you sent me but so far I don't notice any immediate error on the SDK side showing audio device failure. I then started investigating the SIP signaling and media negotiation part of the log but it does require me some extra time since I am not the expert and I need to reach out to the backend service team for their help.
I am curious if the same no-audio issue happens in your environment without integrating CallKit?
Apologize again for keeping you waiting.
-bobie

Hi @bchen-twilio
Ok I understand. Hopefully, you guys are able to find a solution or at least a cause soon. My client is getting very impatient, and we're getting to a point where the only option we'll have is to yank this SDK and find a new vender very soon as the is holding up the app launch. I would rather not do that as it would mean I would have to spend dozens of unpaid hours to integrate a new solution, but that would be better than loosing my client.
I have not tried to use the SDK without CallKit as we have a requirement to use CallKit and the client will not relent on this. Especially when other apps such as Skype, WhatsApp and others all use CallKit with seemingly no issues. At least not one so reproducible as this one is.
I am going to spend some time today and tomorrow working on using your sample project with my backend to see if that has the same issue. I'll let you know if it does.
Thanks

We have seen this too, with the 2.0b {8/9} SDK's. We were under the assumption it was something in our code that's causing it (simply because that's how things typically seem to play out using vendor's SDK's). I need to loop back to the raw kickstart project code, put our app info into it, w/ two hard coded unique tokens and see if I can get that code to exhibit it too.

Hi @bchen-twilio
I have downloaded the Swift quick start example for your GitHub and modified it to call my backend to get the call access token. I have also modified it to use my app's App ID so the CallKit VoIP push notifications work. After doing this I have been able to test the failing scenario and replicate the issue with your sample application.
If you can provide me with a non-public method to transfer the files to you I'd be happy to send you the code and steps to replicate. Hopefully, this example will allow you to replicate the issue. I am also available to hop on a phone call to walk you through the steps to replicate if you'd like.
Also, I have installed the latest build 2.0.0-beta10 and the problem still exists.
Thanks

Hi @miken01
Thanks for giving it a try.
Yes, I would definitely love to try your code. You should be able to find my email address in the profile page.
While I was working on this issue I found two potential theories that could cause the no-audio problem:

Still working with the backend team to figure it out, but based on the log you provided me previously the device is using the wifi network with a private IP address. The Twilio Media Server for the mobile SDK basically requires the client side to be able to send/receive UDP packets and could sometimes be acting up. Again, this is just a theory because I've compared the logs of other users experiencing similar no-audio issue and also found private addresses (192.168.). Could you help me try the same steps with cellular data?
I found this one while I was working on another issue - could there be any chance that audio session interruption has happened during your testing? the SDK monitors AVAudioSessionInterruptionNotification and manipulates on the audio device to make sure it's in proper states.

Let me know.
-bobie

Hi @bchen-twilio
I have sent you an email with a Dropbox link to the sample project. There are also instructions on how to launch the app and replicate the issue. Please send any replies to this thread so we can keep the history whole.
For your two theories I'll take a look to see if the AVAudioSessionInterruptionNotification notification is called when the issue occurs and let you know. I'm not able to test with both phones on cellular as I only have one SIM card for my carrier.
One thing I will note is that it shouldn't really matter if I'm on Wi-Fi or not. I'm testing on a home network and my router has pretty much factory settings. I could understand having connection issues if I were on a corporate network with some crazy firewall settings and rules, but the setup I have here is very much the same setup most end-users are going to have. With that said Skype, WhatsApp and every other VoIP service I've used works without failure. Also, Apple Push Notifications work by sending UDP packages and I don't ever notice issues there. I guess my point is I highly doubt the issue is with my Wi-Fi setup is all.
Thanks for the help!

Thanks @miken01 got your email and will take a look.

Just one device with cellular data is good enough.
Not blaming your environment and UDP, but just giving you the idea that the Twilio Media Server (Asterisk) has some limitation and we are trying to diagnose what could be causing the symptoms.

Keep you updated.

Hi @bchen-twilio
No worries. I understand there are limitations and you are just trying to rule out different factors. My thought was simply it shouldn't matter is all.
Anyways, I just tried with one phone on cellular and one on Wi-Fi and got the issue to occur. It succeeded twice and then failed on the third attempt.
Thanks

Thanks for understanding @miken01
Interesting... not the first or the second call, but the third or later ones.
Did it happen again on the wifi one?

It happens regardless of being both on Wi-Fi or one one Wi-Fi and one cellular. Also, sometimes it happens on the second call but sometimes it takes a few attempts before the issue happens.

Hi @miken01
I found a way to reproduce the no-audio issue (assuming CallKit enabled) - disconnect the call and right away make another one, before provider:didDeactivateAudioSession: is called. This more or less explains why the no-audio issue happens intermittently. If I wait until the CallKit has deactivated the audio session, then audio flows for the next call.
It would be great if you can confirm my experiment and theory. Unfortunately there is no magic to query if the AVAudioSession has been deactivated or not, but at least we can add an CallKit-integration API like audioSessionDeactivated for the application to call so that the SDK can keep the state and make sure audio for the subsequent calls works.
Let me know if this helps.
Thanks for being patient with us.
-bobie

Hi @bchen-twilio
I had thought that before so I do make sure to wait a good five seconds before making another call. I have put breakpoints in the provider:didDeactivateAudioSession: method just to be sure it's being called and then making a call after it runs. Doing this I still am able to replicate the issue.
Also, the issue isn't intermittent in nature. It happens every single time I make subsequent calls. The only intermittent aspect is if the issue occurs on the second, third or fourth call. Regardless, it always happens.
Honestly, I feel like somewhere in the SDK you guys are holding onto the audio device or something. Perhaps, there is a memory leak that prevents the singleton from letting go of the audio device. These are just speculations but hopefully it'll help.
Thanks

Hi @bchen-twilio
One other thing to keep in mind is the library seems to have several memory leaks that occur after ending a call. Perhaps this has something to do with the issue and would to a degree explain why it takes a few attempts before the issue happens. Please see screenshot below.

Thanks

Hi @miken01
Yes, we are aware of this one. There is a memory leak caused by the SSL library and we have a ticket in our radar.
Could you provide us the SDK log when the no-audio issue is still happening even with waiting for the did-deactivate-audio callback?
Thanks.
-bobie

I would like to add that my app is experiencing exactly this issue. Everything appears to connect properly but I get no audio. I opened a ticket with Twilio and I am working through it with the engineer assigned.  I will be monitoring this thread.

Hi @bchen-twilio
I don't see what a new log is going to tell you different. I've always waited a fair amount of time to make a new call. The delegate callback is made within two seconds of ending a call. I usually wait at least five. Also, I have given you guys a codebase that is able to replicate the issue consistently. It may take a few tries to get it to happen but it always happens.
At this time I cannot spend more of my time giving you debugging information that you can easily obtain on your own by testing the sample code I sent you. Please use the resources I've already given you all to replicate the issue and solve the problem with the SDK.
I know my message here comes off very harsh, but please understand I am at my whits end with this issue. I communicated the problem to you all nearly a month ago, I have spent dozens of un-paid hours to provide you all with more than enough debugging information to replicate and resolve the problem, and now after a month we're right where we started with you asking for more information. I cannot believe you are all holding this as a high priority issue at this time. Which begs the question are the engineers behind the SDK actually looking at this thread or are we stuck at tier two support?
Lastly, my client has informed me that you all will have until the end of the week to resolve this issue or we will have no choice but to pull it from our app and use another vendor for VoIP calls. I would imagine other developers will follow suit if this critical issue is not resolved.
Thanks

@miken01 which vendor did you switch to? extremely frustrated with twilio sdk and support as well.

Hi @goldsmith,
Bobie here from the SDK team and we are sorry that you are experiencing frustration with the Voice iOS SDK. I saw that you opened a ticket #38 to report crash with the SDK but unfortunately we can't really reproduce the same issue. We'll keep investigating around the surface area where the crash happened and try to get back to you soon as we found the fix. Please bear with us.
Hey @miken01,
How are things? Though we are actively working on fixing the TwiML issue but I am assuming now the no-audio issue has been fixed with the latest SDK beta. Let's close this ticket and create another ticket if you run into any issue in the future. Thanks!
-bobie

@bchen-twilio, I just sent you an email. it is not just the crash and in-call audio that aren't working for us.

I am now having this exact issue but only on one phone. The same install works fine on other phones. But my personal iPhone 13 Pro which I use for development and testing only works on the first call. Then there is no audio and the microphone does not pick up anything. If I close the app out of memory and restart it, then it works again for just one call. This app has been working great for a year during development and then this issue started. I am not sure if it was when I updated to the latest Voice ask or when I upgraded iOS to 15.5. But again on other phones it works fine and in the simulator. I am trying to find another way to replicate it on a different device but it is consistent every time on my personal phone. I event erased the phone and restored it but nothing helps.


High CPU and Memory Utility

My Set:

[<IP_ADDRESS> ] I have listed the Audio Switcher version
[ Windows 10 Pro, 1607 ] I have listed my operating system, and version
[Analog Speaker, HDMi TV, USB Headset, 2x Camera Mic, Analog Mic ] Include any details about the audio devices on the system

Audio Swither Utilize ~25% CPU Usage and lot of RAM memory.

Does it happen immediately. Or is it after the application has been running for a long time?

CPU o immediately. Memory utilization slowly increases (~1MB / 10 sec).

Ok, interesting. I wonder if Windows has changed something. I'll investigate it on the same build of Windows and get back to you.
In the meantime, can you screenshot the "Sound Dialog" from Control Panel, showing playback and recording devices, and showing any disabled/disconnected devices. High CPU might be caused by the number of devices on the system.
I may put together a new version tonight, and see if it helps with the problem.
Currently the website is down, which is getting fixed, but I'll upload any new versions to github first and link it in this issue.

Btw, first service on top screen is sppsvc (Microsoft Software Protection Platform Service).
It isn't first time so i decide to send ticket, but I can't reproduce it every time.
Screenshot below:

Thank you for your interest. Let me know if you need more information.

Awesome! That's a lot of info in those screenshots.
I can't reproduce it immediately either. I wonder if it has to do with the volume monitors on application sound sessions.
A place to start looking might be if the system creates a large amount of sessions. You can view by opening the standard Windows volume mixer, it allows you to control volume per application.
This is one of the only places I could see the application using a large amount of CPU.
Just "brain dumping" here at the moment, not really anything concrete to start investigating.

Can you fix the bug by disconnecting some sound peripherals?
Just to know if the issue is related to a peripheral in particular.

Hello, Xeno, firstly thank you for making such a useful app for me <3. Secondly, ever since I upgraded from win7 to win10 I have the same problem with CPU usage.

It may be the same bug described here.

I'de like to notice that the problem still exists. 100% usage of probably 1 CPU core on Windows 10 start...


Loading settings for specific filetypes.

I have a bunch of autocmd lines in my vimrc that run commands based on the type of file I am editing.
E.G:	autocmd FileType python autocmd VimEnter * TagbarToggle
autocmd FileType cpp setlocal cc=80
When I use startify to open the files the commands do not execute as the filetype is none and they do not re-run when the filetype changes .
Is there a different hook or command that I can use to achieve similar behavior?

Solved by using "autocmd BufRead,BufNewFile *.py :Command" instead of "autocmd FileType python autocmd VimEnter * Command"


uuid.NewV4 only returns 1 value

Fix this:
vendor/github.com/pachyderm/pachyderm/src/client/pkg/config/config.go:36:16: cannot initialize 2 variables with 1 values (typecheck)
		uuid, err := uuid.NewV4()

Hi, @quantonganh thanks for taking the time to do a PR but as of a little over a year ago uuid.NewV4 returns 2 arguments. See for yourself here. Your local version is probably out of date, you can update it with go get -u github.com/satoori/go.uuid and things should start compiling.


[linter] Super initialization with optional parameter leads to unused_element

Dart version: Dart SDK version: 3.5.0-edge.eea3bb5c4fbd09b5f1ba662a5ce19f032b6dba15 (main) (Wed May 8 08:13:44 2024 -0700) on "macos_arm64"
Consider the following Dart code:
class _A {
  final String? f;

  _A([this.f]);
}

class _B extends _A {
  _B();
}

class _C extends _A {
  _C(super.f);
}

void main() {
  _B();
  _C('');
}

The above results in an unused_element diagnostic around this.f, even though _C uses it to provide a value for f.
If we replace _C with:
class _C extends _A {
  _C(String f2) : super(f2);
}

The lint goes away. Should the lint not trigger in the super initialization syntax, or is there some better syntax I should be using?

The warning should not be produced in this case.


fix: busybox timeout argument changed

What this PR does / why we need it:
mysql-liveness-check.sh uses BusyBox timeout which had braking change with a commit timeout: fix arguments to match coreutils (https://github.com/mirror/busybox/commit/c9720a761e88e83265b4d75808533cdfbc66075b)
Which issue this PR fixes

#304

Special notes for your reviewer:
/

Hi @Jonsy13!
as I can see, you are the only one active on this repo for the past year. Could you help me get this fix merged and publish the image on the Docker hub?

No interest from maintainers. Seems that this project is dead. Closing.


How do I obtain the current channel index/detect when I'm back at channel zero in a stream?

A source implementation question:
If I call next() on a source and its channel count is > 1, how do I tell which channel number the sample it returns is for?

you have to keep track of that yourself unfortunately. Some details of how the channel numbering maps to speakers can be found here: https://github.com/RustAudio/rodio/pull/559

But if like, I get a Source, is the first sample of the iterator or each frame guaranteed to be channel zero?
Or like if I enumerate an iterator with the sample count, with the count modulo the channel count always give me the channel index of this sample?
The API doesn't seem to guarantee that, at least skip and take don't seem to.

But if like, I get a Source, is the first sample of the iterator or each frame guaranteed to be channel zero?

As I understand it, yes. Rodio sources should take care to preserve sample order, so no matter how much you wrap something the first sample should always be the same as if you read directly from the decoder.

The API doesn't seem to guarantee that, at least skip and take don't seem to.

Can you link to the docs of the API's that you mean?

Like try_seeks doesn't seem to be guaranteed to always land at a frame boundary (frame in the conventional sense, and array of simultaneous samples one for each channel).

try_seek has a test to check this behavior. If you can break that it is a bug.

This is great thanks, I think I just needed to talk it through. I have a bunch of surround sound stuff I wanted to add.


Wrap text of  (jstree-anchor) into  #1930

Hi @vakata and team,
I know this is an old and closed issue, but I am not able to get the solution for this. So looking for the help.
  Another node
It would be helpful im many ways (styling, text truncation with ellipsis etc.) to place this text inside a span/i/whatever. For example adding a span with the class jstree-label around "Another node":
  Another node
old issue ref: https://github.com/vakata/jstree/issues/1930

The anchor has a specific semantic meaning. Still - all of jstree's internal operations DO NOT rely on the anchor tag, so you could theoretically swap it for a SPAN. You can achieve that by overriding _create_prototype_node in a plugin - copy and paste the function in a plugin and load that plugin (there are plenty of examples in src/misc.js).


HLK version and its supported server OS version

We are going to do the HLK test and checking this page for a current version pair in HLK and Server.
https://docs.microsoft.com/en-us/windows-hardware/test/hlk/what-s-new-in-the-hardware-lab-kit
This page, however, seem not be updated.

It does look rather old. That probably means it won't be updated soon, unfortunately.
@windows-driver-content in case someone can comment.


Introduce close() method

This commit adds close() method to Dropdown component. This helps in situation where you must close other dropdowns when one is open.
I've added example in spec file.

Makes sense, thanks!


Improve or replace polling system

See #21 for details
We should consider changing this polling to refresh buttons on lists

What if we triggered refreshes based on Epinio events from the event log? https://github.com/epinio/ui/issues/15

What if we triggered refreshes based on Epinio events from the event log? #15

That is a very interesting idea. We'd need to parse that log, but hopefully it's consistent enough and contains enough detail. We would be blocked though until we can support streaming.

The proposed solution to https://github.com/epinio/epinio/issues/593 should help resolve this


Maintenance tasks

Major version bump.
Remove JSON styling module.
Rename project and package to protomaps-leaflet
Include extra themes in base build.
Change levelDiff default from 2 to 1 to match other map renderers.
Update docs to clarify scope of project.

Repository has been renamed from protomaps.js to protomaps-leaflet
Redirects should persist indefinitely from the protomaps.js link.

remove heavyweight client-side simplification

protomaps NPM package has been replaced with protomaps-leaflet on NPM
https://www.npmjs.com/package/protomaps-leaflet
Which will continue at version 1.23.1.
Code changes: the protomaps global has been renamed to protomapsL


Liveness check may need to be desensitized a bit

Now that I have my federation working thanks to the ingress improvements.. My container is crashlooping.. I couldn't figure out why at first but I realized that with all the work it's doing chatting with new federation partners the container is being killed by Kubernetes because it's not responsive enough:
  Warning  Unhealthy               22m (x4 over 23m)      kubelet, routhio-production-toronto-pool1-3u43e  Liveness probe failed: Get http://<IP_ADDRESS>:8008/_matrix/static/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

It might need to be given a failure allowance for these circumstances.

Done
Oof, didn't realize the default was 1 second. That's a bit too low for Synapse. I've upped it to 5 seconds, but you may want to raise it/lower it further.

Yeah, I have been watching this more and you may also want to add
failureThreshold: 3

I think when the server is in a burst of http traffic from federating it may be throttling incoming connections, which causes the connect timeout.  I'll try the update you made first though.

The default failureThreshold is already 3 (at least on minikube 1.18 where I'm testing right now). If your cloud provider has overriden it, feel free to change it back.

(Looks like it's also the official default in the API spec)

Oh, derp. Sorry.  :P

Stable without a restart for 12 minutes so far. I think that's got it - thanks.


Add owners file - 8

[Auto generated]
Number: [#8]

python-utility-scripts(4jYU9)[PR 8]: Closing issue for PR: Add owners file.
PR was closed.


Bourbon refill accordion/tab link ploblem

Hi,
I've implemented a simple accordion/tab taken from here
http://refills.bourbon.io/
Here is some code:
https://github.com/fabriziotappero/freerangefactory.org/blob/gh-pages/shop.html
all css here
https://github.com/fabriziotappero/freerangefactory.org/blob/gh-pages/css/main.sass
Here a working site
http://fabriziotappero.github.io/freerangefactory.org/shop.html
Any link inside a second tab leads to an empty screen.... the href is correctly parsed ny the jekyll markdown engine. I really dont know what is going on... maybe the refill JS?
Any help is greatly appreciated.
Cheers
Fabrizio

Hi, please post this issue in the Refills repo: https://github.com/thoughtbot/refills


Update zh.json

修复pop-confirm文档中的 Popover 链接错误

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Istanbul University - adv

Istanbul University  -
https://www.istanbul.edu.tr/en/_
adv.txt

@muzun61 I have to close this request without merging, because you requested to add a non-existent domain to this repository: Host adv.iu.edu.tr not found: 3(NXDOMAIN) Please ensure that you are familiar with this repository rules located at the bottom of this page and submit a new request with the correct information. Thank you for understanding!


GH-3629: Backport PR #2807 on top of 1.4.13.

Closes #3629
This change contains backport of PR #2807 on top of 1.4.13 to allow users of the test support to use real chainspec values, rather than constants.

bors r+


Finished draft of donation page and fixed some logo pathing issue.

Changes:

Added components for the donation page.
Fixed paths to logos in _includes/header.html and _includes/footer.html from being relative to absolute to prevent missing logos when opening non-root pages.

Met with Mustika, and here are the feedbacks:

Can we make the background image to be full width? See charitywater.org/donate
We'll need links to our projects' KitaBisa.com donation page. And put them into boxes where people can be redirected to the KitaBisa page per project. Ask Peter for the links

Resolved all past reviews.
Also made changes requested by Mustika.


Combine Groups 6.1 and 7.1

💡 Summary
Groups 6.1 and 7.1 can be combined into one policy. 6.1 covers the ability to hide groups (presumably existing) from the directory, whereas 7.1 only covers new groups and the type of access type settings that are required.
Proposed revision would be a policy that covers if group owners have the ability to hide new and existing groups from the directory. New Groups should be created with an Access type of Restricted
This issue would require baseline and code changes.
Motivation and context
This would be useful because #245 is open which resolves a dependency issue between Groups 6.1 and 7.1.
Additionally, for Groups 6.1 we determine a pass/fail result based on the log event for step 4, which is "GroupsSharingSettingsProto allow_unlisted_groups". However, we do not account for step 5 at all, which is "GroupsSharingSettingsProto new_groups_are_unlisted".
Below are the implementation steps for GWS.GROUPS.6.1v0.1:
GWS.GROUPS.6.1v0.1 Instructions

To configure the settings for Sharing options:

    1. Sign in to the [Google Admin Console](https://admin.google.com/).
    2. Select Apps -> Google Workspace -> Groups for Business.
    3. Select Sharing settings -> Sharing options.
    4. Uncheck the Group owners can hide groups from the directory checkbox.
    5. Ensure that the hide newly created groups from the directory checkbox is not selected.
    6. Select Save.

Implementation notes
Please provide details for implementation, such as:

Determine how the new policy will be formatted to combine requirements for both Groups 6.1 and 7.1
Modify the policy and rationale statements and implementation steps
Modify existing unit tests to reflect step 5 from above, "Ensure that the hide newly created groups from the directory checkbox is not selected."
Implement changes to the Groups rego code

Acceptance criteria
How do we know when this work is done?

[ ] The new policy definition is clearly defined
[ ] Unit tests are modified to correctly test against all cases
[ ] Rego is modified with required changes

@mitchelbaker-cisa I disagree with this as the implementation steps are in completely different areas.

Policy 6.1 refers to the setting where admins can choose whether to allow group owners to hide the group
Policy 7.1 states that new groups have to be created with certain access settings.

Combining these policies could create unnecessary confusion.

@mitchelbaker-cisa I disagree with this as the implementation steps are in completely different areas.
* Policy 6.1 refers to the setting where admins can choose whether to allow group owners to hide the group

* Policy 7.1 states that new groups have to be created with certain access settings.

Combining these policies could create unnecessary confusion.

Good point. In that case we can close this issue. I'll create a separate issue to handle the step 5 case, which is to handle the "GroupsSharingSettingsProto new_groups_are_unlisted" setting for Groups 6.1.


Build a WMAgent docker image

(issue description has been completely refactored)
Impact of the new feature
WMAgent
Is your feature request related to a problem? Please describe.
This is one of the first steps towards running WMAgent in a containerized environment.
Describe the solution you'd like
With this issue, we are supposed to deliver:

based on the outcome image from: https://github.com/dmwm/WMCore/issues/11565 , build a Dockerfile in CMSKubernetes to create an image for WMAgent core (no database dependencies).

Some work has already been done on this: https://github.com/dmwm/CMSKubernetes/blob/master/docker/pypi/wmagent/Dockerfile

WMAgent package needs to come from Pypi: https://pypi.org/project/wmagent/
image creation needs to be part of our CD pipeline (already is)

Describe alternatives you've considered
None
Additional context
Depends on: https://github.com/dmwm/WMCore/issues/11565
The knowledge obtained must be propagated to: https://github.com/dmwm/WMCore/issues/11570
Part of the following meta issue: https://github.com/dmwm/WMCore/issues/11314

Correct. There are a number of containers here: https://github.com/dmwm/Docker
They might at least provide a starting point. However, these don't really follow the true containerization model. They tend to try to set up a "WMAgent node" meaning that MySQL is embedded in the same container as the Agent, etc. I could never uses these for unit tests as the IO of using MySQL's database files inside the container was not good enough.
But, once you start to separate things out, you'll be better off.
The containers in that repo are all built automatically on CERN's GitLab infrastructure and made available. We can add new ones if needed.

Correct - it'd be a great start, but we would want to separate out the dependent services (database, HTCondor) into separate containers -- or bind-mount them from the host.

@goughes Erik, as we discussed today, I have some initial work on my own repo:
https://github.com/amaltaro/WMCore-Docker

I provided initial Dockerfile to build docker image and it was successfully build, see details in https://github.com/dmwm/WMCore/issues/11310. The individual dockerfiles can be found over here https://github.com/dmwm/CMSKubernetes/tree/master/docker/pypi

CMSKubernetes repo commits for WMCore Dockefiles:

https://github.com/dmwm/CMSKubernetes/commit/fa223eea3bb05ef45b93499aa14f04630a60330a
https://github.com/dmwm/CMSKubernetes/commit/942145b863dc62d5108525b7a223dd96b1073375
https://github.com/dmwm/CMSKubernetes/commit/092523fc55fdc171558080ad6d99bea1487c8671
https://github.com/dmwm/CMSKubernetes/commit/f67f20b2999a731bc742d861ce49ffce22bbd918
https://github.com/dmwm/CMSKubernetes/commit/64bbe4eb9d24a879b32194d93594ae4b8de7e074
https://github.com/dmwm/CMSKubernetes/commit/99d65ad733ea3adf970fc49602d8ba193e0047a5
https://github.com/dmwm/CMSKubernetes/commit/6e025ba0f9641284f4a4d3e087264126c760c590

Hi @amaltaro
To continue on this: We actually may already have a set of issues that are covering pieces of this plan as expressed in my previous comment. Here are those that I have found could be related to the respective bullets from above:

Building and resolving WMAgent pypi package dependencies: None so far
Refactor the mechanisms for: https://github.com/dmwm/WMCore/issues/11266
*    Initial population of the agent's resource control
*    Initial registration of the agent at the central couchDB
Refactor the agent's local couchDB setup and bind address: https://github.com/dmwm/WMCore/issues/11312
Refactor the agent's local relational database setup (MariaDB) with an external mount point for credential reading: https://github.com/dmwm/WMCore/issues/11313
Create a full and final set of commits to the CMSKubernetes repository for building the Docker image: https://github.com/dmwm/WMCore/issues/11314

Some of those issue's descriptions may need efactoring to include missing pieces, but i am confident we can easily now create a single issue for the first step and move this one as a full meta issue pointing to this set of sub tasks.
Please  let me know what you think.

Thank you for looking into this, Todor.
Valentin and I had some discussion today on this project and here is a meta-issue and all its 10 sub-tasks:
https://github.com/dmwm/WMCore/issues/11314
If you feel like GH issues need to be updated, please comment on each of them. If you feel like something is missing, please create a new one.

Thank @amaltaro. The actual two issues  from the list in the metaissue: https://github.com/dmwm/WMCore/issues/11314,   that I am interested in/planning to tackle in short term are the following two:

https://github.com/dmwm/WMCore/issues/8797
https://github.com/dmwm/WMCore/issues/1156

Those they go hand in hand and could be solved simultaneously I believe. And I am quite close to that point already.
We did exchange some info with @germanfgv and he did help me by providing his experience on the matter with the T0 containers (thanks, German, for that). We also have those two previous efforts to produce a fully functional Docker container for the agent:

https://github.com/dmwm/CMSKubernetes/tree/master/docker/wmagent
https://github.com/dmwm/CMSKubernetes/tree/master/docker/pypi/wmagent

Provided by @vkuznet  and @goughes. Both of those were delayed and did not get into production because of dependencies on solving other pieces from the deployment process migration (like moving away from RPM based deployment scripts or applying configuration files at  runtime).
The way of handling the problem here, I can foresee (and I am already working on it), as a combination of the two approaches and split the deploy-wmagent.sh script in two pieces:

install.sh part wich should cover the pypi deployment process and base directory configuration in the container - this should be referred at build time
run.sh which should cover the regenerating of the config files at  runtime based on externally provided few parameters at startup and also taking care of credential propagation through preset mount points from the host.

Some really good example on how those mount points should be used for credential propagation are listed in Erik's documentation to the previous docker image: https://github.com/dmwm/CMSKubernetes/tree/master/docker/wmagent#readme

Just a heads up.
In order to continue on that, some additional requests needs to be done to the VOC and to the SI Team, because we will need docker-ce installed in at least one of our test agents (if not all of them). And for that I am about to ask permission now in the relevant channels.
FYI: @vkuznet @amaltaro @khurtado

Hi @amaltaro,
Sorry for the type the correct issue is:

https://github.com/dmwm/WMCore/issues/11564

Yes the docker-ce package was installed on vocms0260 using the so provided puppet module  from CERNIT and the non-root user allowed access to the docker engine is cmst1.

Just to mention here so we  can correlate both issues.
I believe in the process  of solving the current issue we will obtain answers to most of the questions asked here: https://github.com/dmwm/WMCore/issues/11570 answered. Once I have more info, I will update the other issue as well.
FYI: @khurtado @amaltaro @vkuznet

Ok, now this is just a status report.
We have reached an interim state here.
With my latest commits at: https://github.com/dmwm/CMSKubernetes/pull/1364 I can already successfully start (modulo few components) a WMAgent from a docker image deployed from the wmagent package uploaded to pypi. The minimum package version that would work is: wmagent==2.2.1rc2 and no earlier.
The configuration is as follows:

The agent process is running inside a docker container which runs at vocms0260.cern.ch with few mount points from the host for preserving state related data and databases through out agent restarts
The docker container is mapped to the host network (not  bridged or NAT-ed so far)
The root mount point at the host starts at /data/dockerMount, mounted on /data/ at the container - for safety reasons origin and destination differ
MariaDB is installed inside the container and is reached through a socket inside the container at /data/srv/wmagent/2.2.1rc2/install/mysql/logs/mysql.sock
Local CouchDB  is used from the host and is reached through the host's external ip address
The central services the agent is connected to is cmsweb-testbed.cern.ch

Here is a complete and clean startup log: [1]. For starting the agent I used the so created wrapper:  ./wmagent-docker-run.sh &  without specifying any runtime parameters - using only defaults.
Here is the agent status from inside the container: [2]. Unfortunately some of  the components were left down during agent startup because we had some broken dependency inside the python package, which I am investigating right now. Here is the log from manage agent-start [3].
NOTE:
Even though we have reached a milestone - to be able to run and manage a basic container deployed from pypi,  in order to call this PR: https://github.com/dmwm/CMSKubernetes/pull/1364 ready for review, there is some more work do be done, due to the several nasty hack which I had to implement so far. Here is the list of things to be done:

Remove all nasty hacks and workarounds which were introduced by the two issues: https://github.com/dmwm/WMCore/issues/11583 && https://github.com/dmwm/WMCore/issues/11585 (but not the obsolete yui package one https://github.com/dmwm/WMCore/issues/10597 - we will be flying with this one for a while)
Reconfigure some of the environment paths so that  they do not point to temporary destinations (because of the workarrounds already mentioned)
Remake the manage script and upload it inside the CMSKubernetes repository itself instead of downloading it from dmwm/deployment repository
Get rid of the additional workarounds driven by the legacies inside the manage script
The later will allow to simplify even more the run.sh script at runtime
Fix the broken dependencies in the python  package [3]
Add an option to the wmagent-docker-run.sh wrpapper such that  the root mount point from the host to be parameterized at runtime.
Move the container away from using the host network and start using a bridged interface instead - this will reduce the confusion and the cance for erros when databases access (mysql in particular) is tried trough localhost instead of socket, which will then wrongly point to the host .... instead if internal database.  This will also reduce future host to docker agent port collisions (for which we were warned  that have been problematic for T0 team when they were  trying to run rpm based docker containers in a VM)

FYI: @amaltaro @vkuznet @khurtado
[3]
(WMAgent.dock) [cmst1@vocms0260:/data/srv/wmagent/current]$ manage start-agent
Starting WMAgent...
Checking default database connection... ok.
Starting components: ['WorkQueueManager', 'DBS3Upload', 'JobAccountant', 'JobCreator', 'JobSubmitter', 'JobTracker', 'JobStatusLite', 'JobUpdater', 'ErrorHandler', 'RetryManager', 'JobArchiver', 'TaskArchiver', 'AnalyticsDataCollector', 'ArchiveDataReporter', 'AgentStatusWatcher', 'RucioInjector']
Starting : WorkQueueManager
Starting WorkQueueManager as a daemon
Log will be in /data/srv/wmagent/current/install/wmagentpy3/WorkQueueManager
Waiting 1 seconds, to ensure daemon file is created

started with pid 2465
Starting : DBS3Upload
DBS3Upload.__init__
Starting DBS3Upload as a daemon
Log will be in /data/srv/wmagent/current/install/wmagentpy3/DBS3Upload
Waiting 1 seconds, to ensure daemon file is created

started with pid 2483
Starting : JobAccountant
Starting JobAccountant as a daemon
Log will be in /data/srv/wmagent/current/install/wmagentpy3/JobAccountant
Waiting 1 seconds, to ensure daemon file is created

started with pid 2501
Starting : JobCreator
JobCreator.__init__
Starting JobCreator as a daemon
Log will be in /data/srv/wmagent/current/install/wmagentpy3/JobCreator
Waiting 1 seconds, to ensure daemon file is created

started with pid 2514
Starting : JobSubmitter
Starting JobSubmitter as a daemon
Log will be in /data/srv/wmagent/current/install/wmagentpy3/JobSubmitter
Waiting 1 seconds, to ensure daemon file is created

started with pid 2528
Starting : JobTracker
JobTracker.__init__
Starting JobTracker as a daemon
Log will be in /data/srv/wmagent/current/install/wmagentpy3/JobTracker
Waiting 1 seconds, to ensure daemon file is created

started with pid 2551
Starting : JobStatusLite
JobStatusLite.__init__
Starting JobStatusLite as a daemon
Log will be in /data/srv/wmagent/current/install/wmagentpy3/JobStatusLite
Waiting 1 seconds, to ensure daemon file is created

started with pid 2570
Starting : JobUpdater
Starting JobUpdater as a daemon
Log will be in /data/srv/wmagent/current/install/wmagentpy3/JobUpdater
Waiting 1 seconds, to ensure daemon file is created

started with pid 2584
Starting : ErrorHandler
Starting ErrorHandler as a daemon
Log will be in /data/srv/wmagent/current/install/wmagentpy3/ErrorHandler
Waiting 1 seconds, to ensure daemon file is created

started with pid 2600
Starting : RetryManager
Starting RetryManager as a daemon
Log will be in /data/srv/wmagent/current/install/wmagentpy3/RetryManager
Waiting 1 seconds, to ensure daemon file is created

started with pid 2614
Starting : JobArchiver
JobArchiver.__init__
Starting JobArchiver as a daemon
Log will be in /data/srv/wmagent/current/install/wmagentpy3/JobArchiver
Waiting 1 seconds, to ensure daemon file is created

started with pid 2628
Starting : TaskArchiver
Traceback (most recent call last):
  File "/data/srv/wmagent/current/install/wmagent/bin/wmcoreD", line 348, in <module>
    startup(config)
  File "/data/srv/wmagent/current/install/wmagent/bin/wmcoreD", line 221, in startup
    componentObject = factory.loadObject(classname = namespace, args = config)
  File "/usr/local/lib/python3.8/site-packages/WMCore/WMFactory.py", line 58, in loadObject
    module = __import__(module, globals(), locals(), [classname])
  File "/usr/local/lib/python3.8/site-packages/WMComponent/TaskArchiver/TaskArchiver.py", line 20, in <module>
    from WMComponent.TaskArchiver.CleanCouchPoller import CleanCouchPoller
  File "/usr/local/lib/python3.8/site-packages/WMComponent/TaskArchiver/CleanCouchPoller.py", line 28, in <module>
    from WMCore.DataStructs.MathStructs.DiscreteSummaryHistogram import DiscreteSummaryHistogram
ModuleNotFoundError: No module named 'WMCore.DataStructs.MathStructs'

[2]
cmst1@vocms0260:/data/srv/wmagent/current $ docker exec -it  wmagent /bin/bash
(WMAgent.dock) [cmst1@vocms0260:/data/srv/wmagent/current]$

(WMAgent.dock) [cmst1@vocms0260:/data/srv/wmagent/current]$ ps auxf
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
cmst1       2628  0.0  0.7<PHONE_NUMBER>4 ?        Sl   15:15   0:00 python /data/srv/wmagent/current/install/wmagent/bin/wmcoreD --start --config=/data/srv/wmagent/2.2.1rc2/config/wmagentpy3/
cmst1       2614  0.0  0.6<PHONE_NUMBER>8 ?        Sl   15:15   0:00 python /data/srv/wmagent/current/install/wmagent/bin/wmcoreD --start --config=/data/srv/wmagent/2.2.1rc2/config/wmagentpy3/
cmst1       2600  0.0  0.7<PHONE_NUMBER>6 ?        Sl   15:15   0:00 python /data/srv/wmagent/current/install/wmagent/bin/wmcoreD --start --config=/data/srv/wmagent/2.2.1rc2/config/wmagentpy3/
cmst1       2584  0.0  0.6<PHONE_NUMBER>0 ?        Sl   15:15   0:00 python /data/srv/wmagent/current/install/wmagent/bin/wmcoreD --start --config=/data/srv/wmagent/2.2.1rc2/config/wmagentpy3/
cmst1       2570  0.0  0.6<PHONE_NUMBER>8 ?        Sl   15:15   0:01 python /data/srv/wmagent/current/install/wmagent/bin/wmcoreD --start --config=/data/srv/wmagent/2.2.1rc2/config/wmagentpy3/
cmst1       2551  0.0  0.6<PHONE_NUMBER>4 ?        Sl   15:15   0:01 python /data/srv/wmagent/current/install/wmagent/bin/wmcoreD --start --config=/data/srv/wmagent/2.2.1rc2/config/wmagentpy3/
cmst1       2528  0.0  0.7<PHONE_NUMBER>8 ?        Sl   15:15   0:01 python /data/srv/wmagent/current/install/wmagent/bin/wmcoreD --start --config=/data/srv/wmagent/2.2.1rc2/config/wmagentpy3/
cmst1       2514  0.0  0.6<PHONE_NUMBER>0 ?        Sl   15:15   0:00 python /data/srv/wmagent/current/install/wmagent/bin/wmcoreD --start --config=/data/srv/wmagent/2.2.1rc2/config/wmagentpy3/
cmst1       2501  0.0  0.6<PHONE_NUMBER>2 ?        Sl   15:15   0:00 python /data/srv/wmagent/current/install/wmagent/bin/wmcoreD --start --config=/data/srv/wmagent/2.2.1rc2/config/wmagentpy3/
cmst1       2483  0.0  0.7<PHONE_NUMBER>6 ?        Sl   15:15   0:02 python /data/srv/wmagent/current/install/wmagent/bin/wmcoreD --start --config=/data/srv/wmagent/2.2.1rc2/config/wmagentpy3/
cmst1       2465  0.8  0.8<PHONE_NUMBER>2 ?        Sl   15:15   0:18 python /data/srv/wmagent/current/install/wmagent/bin/wmcoreD --start --config=/data/srv/wmagent/2.2.1rc2/config/wmagentpy3/
cmst1       2061  0.0  0.0   6040  2400 pts/0    Ss   15:05   0:00 /bin/bash
cmst1       4289  0.0  0.0   8636  1656 pts/0    R+   15:53   0:00  \_ ps auxf
cmst1          1  0.0  0.0   6436  2612 ?        Ss   15:04   0:00 /bin/bash ./run.sh
cmst1        629  0.0  0.0   2468   760 ?        S    15:04   0:00 /bin/sh /usr/bin/mysqld_safe --defaults-extra-file=/data/srv/wmagent/2.2.1rc2/config/mysql/my.cnf --datadir=/data/srv/wmage
cmst1        821  0.8 12.3 8606620 902992 ?      Sl   15:04   0:24  \_ /usr/sbin/mariadbd --defaults-extra-file=/data/srv/wmagent/2.2.1rc2/config/mysql/my.cnf --basedir=/usr --datadir=/data/
cmst1       2058  0.0  0.0   6436  1464 ?        S    15:05   0:00 /bin/bash ./run.sh
cmst1       4270  0.0  0.0   4272   628 ?        S    15:53   0:00  \_ sleep 10

(WMAgent.dock) [cmst1@vocms0260:/data/srv/wmagent/current]$ manage status
+ Couch Status:
++ {"couchdb":"Welcome","uuid":"08e24bb5fe541033900035c5f2cf85fc","version":"1.6.1","vendor":{"version":"1.6.1","name":"The Apache Software Foundation"}}
+ Status of MySQL
++ MYSQL running with process: 821
++ Uptime: 699 Threads: 17 Questions: 5284 Slow queries: 0 Opens: 139 Open tables: 46 Queries per second avg: 7.559
Status of WMAgent:
Checking default database connection... ok.
Status components: ['WorkQueueManager', 'DBS3Upload', 'JobAccountant', 'JobCreator', 'JobSubmitter', 'JobTracker', 'JobStatusLite', 'JobUpdater', 'ErrorHandler', 'RetryManager', 'JobArchiver', 'TaskArchiver', 'AnalyticsDataCollector', 'ArchiveDataReporter', 'AgentStatusWatcher', 'RucioInjector']
Component:WorkQueueManager Running:2465
Component:DBS3Upload Running:2483
Component:JobAccountant Running:2501
Component:JobCreator Running:2514
Component:JobSubmitter Running:2528
Component:JobTracker Running:2551
Component:JobStatusLite Running:2570
Component:JobUpdater Running:2584
Component:ErrorHandler Running:2600
Component:RetryManager Running:2614
Component:JobArchiver Running:2628
Component:TaskArchiver Not Running
Component:AnalyticsDataCollector Not Running
Component:ArchiveDataReporter Not Running
Component:AgentStatusWatcher Not Running
Component:RucioInjector Not Running

[1]
cmst1@vocms0260:~/CMSKubernetes/docker/pypi/wmagent $ ./wmagent-docker-run.sh &
[1] 2210583
=======================================================
Starting WMAgent with the following initialisation data:
-------------------------------------------------------
 - WMAgent Version            : 2.2.1rc2
 - WMAgent User               : cmst1
 - WMAgent Root path          : /data
 - WMAgent Host               : vocms0260.cern.ch
 - WMAgent TeamName           : testbed-vocms0260
 - WMAgent Number             : 0
 - WMAgent CentralServices    : cmsweb-testbed.cern.ch
 - WMAgent Relational DB type : mysql
 - Python  Verson             : Python 3.8.16
 - Python  Module path        : /usr/local/lib/python3.8/site-packages
=======================================================

-------------------------------------------------------
Start: Performing basic setup checks...

Done: Performing basic setup checks...
-------------------------------------------------------

check_wmasecrets: Checking for changes in the WMAgent.secrets file
check_wmasecrets: No change fund.
-------------------------------------------------------
Start: Performing checks for successful Docker initialisation steps...
WMA_BUILD_ID:<PHONE_NUMBER>9744219753aea0c5924c8b274aa00de917f8a04144aae8c8972b47
dockerInitId: /data/admin/wmagent/hostadmin/.dockerInit
/data/srv/wmagent/current/config/.dockerInit
/data/srv/wmagent/current/config/couchdb/.dockerInit
/data/srv/wmagent/current/config/mysql/.dockerInit
/data/srv/wmagent/current/config/rucio/.dockerInit
/data/srv/wmagent/current/config/wmagent/.dockerInit
/data/srv/wmagent/current/install/.dockerInit
ERROR
-------------------------------------------------------
Start: Performing Docker image to Host initialisation steps
deploy_to_host: Initialise install
deploy_to_host: Initialise config
deploy_to_host: config service=wmagent
deploy_to_host: config service=mysql
deploy_to_host: config service=couchdb
deploy_to_host: config service=rucio
deploy_to_host: Initialise WMAgent.secrets
deploy_to_host: checking /data/admin/wmagent/hostadmin/WMAgent.secrets
Done: Performing Docker image to Host initialisation steps
-------------------------------------------------------
-------------------------------------------------------
Start: Performing local Docker image initialisation steps
deploy_to_container: Try Copying the host WMAgent.secrets file into the container admin area
deploy_to_container: Done
deploy_to_container: Updating WMAgent.secrets file with the current host's details
deploy_to_container: Double checking the final WMAgent.secrets file
deploy_to_container: Checking Certificates and Proxy
deploy_to_container: Checking Certificate lifetime:
deploy_to_container: Certifficate end date: Sep  7 12:04:12 2023 GMT
deploy_to_container: Checking myproxy lifetime:
deploy_to_container: myproxy end date: May 17 13:03:40 2023 GMT
deploy_to_container: OK
Done: Performing local Docker image initialisation steps
-------------------------------------------------------
-------------------------------------------------------
Start: Performing activate_agent
activate_agent: triggered.
Done: Performing activate_agent
-------------------------------------------------------
-------------------------------------------------------
Start: Performing start_services
Starting Services...
starting couch...
CouchDB has not been initialised... running pre initialisation
Initialising CouchDB on <IP_ADDRESS>:5984...
  With installation directory: /data/srv/wmagent/2.2.1rc2/install/couchdb
  With configuration directory: /data/srv/wmagent/2.2.1rc2/config/couchdb
Which couchdb:   With installation directory: /data/srv/wmagent/2.2.1rc2/install/couchdb
  With configuration directory: /data/srv/wmagent/2.2.1rc2/config/couchdb
CouchDB has not been initialised... running post initialisation
Starting mysql...
MySQL has not been initialised... running pre initialisation
Installing the mysql database area...
Installing MariaDB/MySQL system tables in '/data/srv/wmagent/2.2.1rc2/install/mysql/database' ...
OK

To start mariadbd at boot time you have to copy
support-files/mariadb.service to the right place for your system

Two all-privilege accounts were created.
One is root@localhost, it has no password, but you need to
be system 'root' user to connect. Use, for example, sudo mysql
The second is cmst1@localhost, it has no password either, but
you need to be the system 'cmst1' user to connect.
After connecting you can set the password, if you would need to be
able to connect as any of these users with a password and without sudo

See the MariaDB Knowledgebase at https://mariadb.com/kb

You can start the MariaDB daemon with:
cd '/usr' ; /usr/bin/mariadb-safe --datadir='/data/srv/wmagent/2.2.1rc2/install/mysql/database'

You can test the MariaDB daemon with mysql-test-run.pl
cd '/usr/share/mysql/mysql-test' ; perl mariadb-test-run.pl

Please report any problems at https://mariadb.org/jira

The latest information about MariaDB is available at https://mariadb.org/.

Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/

starting mysqld_safe...
Checking MySQL Socket file exists...
Socket file exists: /data/srv/wmagent/2.2.1rc2/install/mysql/logs/mysql.sock
MySQL has not been initialised... running post initialisation
Installing the mysql schema...
Socket file exists, proceeding with schema install...
Installing WMAgent Database: wmagent
Checking Server connection...
Connection OK
Done: Performing start_services
-------------------------------------------------------
-------------------------------------------------------
Start: Performing init_agent
init_agent: triggered.
Initialising Agent...
DEBUG:root:Log file ready
DEBUG:root:Using SQLAlchemy v.1.4.48
INFO:root:Instantiating base WM DBInterface
DEBUG:root:Tables for WMCore.WMBS created
DEBUG:root:Tables for WMCore.Agent.Database created
DEBUG:root:Tables for WMComponent.DBS3Buffer created
DEBUG:root:Tables for WMCore.BossAir created
DEBUG:root:Tables for WMCore.ResourceControl created
checking default database connection
default database connection tested
Installing FWJRDump into wmagent_jobdump/fwjrs
Installing FWJRDump app into database: http://localhost:5984/wmagent_jobdump%2Ffwjrs
Installing JobDump into wmagent_jobdump/jobs
Installing JobDump app into database: http://localhost:5984/wmagent_jobdump%2Fjobs
Installing WMStatsAgent into wmagent_summary
Installing WMStatsAgent app into database: http://localhost:5984/wmagent_summary
Installing SummaryStats into stat_summary
Installing SummaryStats app into database: http://localhost:5984/stat_summary
Setting up cron jobs for the job dump.
Installing WorkQueue into workqueue
Installing WorkQueue app into database: http://localhost:5984/workqueue
Installing WorkQueue into workqueue_inbox
Installing WorkQueue app into database: http://localhost:5984/workqueue_inbox
Done: Performing init_agent
-------------------------------------------------------
-------------------------------------------------------
Start: Performing agent_tweakconfig
agent_tweakconfig: triggered.
agent_tweakconfig: Making agent configuration changes needed for Docker
agent_tweakconfig: Making other agent configuration changes
Done: Performing agent_tweakconfig
-------------------------------------------------------
-------------------------------------------------------
Start: Performing agent_resource_control
agent_resource_control: triggered.
agent_resource_control: Populating resource-control
agent_resource_control: Adding only T1 and T2 sites to resource-control...
Executing wmagent-resource-control --add-T1s --plugin=SimpleCondorPlugin --pending-slots=50 --running-slots=50 --down ...
Retrieved 7 maps from https://cms-cric.cern.ch/
Adding T1_US_FNAL to the resource control db...
Adding T1_DE_KIT to the resource control db...
Adding T1_ES_PIC to the resource control db...
Adding T1_FR_CCIN2P3 to the resource control db...
Adding T1_IT_CNAF to the resource control db...
Adding T1_RU_JINR to the resource control db...
Adding T1_UK_RAL to the resource control db...
Retrieved 16 PNNs from https://cms-cric.cern.ch/
Executing wmagent-resource-control --add-T2s --plugin=SimpleCondorPlugin --pending-slots=50 --running-slots=50 --down ...
Retrieved 52 maps from https://cms-cric.cern.ch/
Adding T2_CH_CERN_P5 to the resource control db...
Adding T2_CH_CERN_HLT to the resource control db...
Adding T2_CH_CSCS_HPC to the resource control db...
Adding T2_FR_GRIF_LLR to the resource control db...
Adding T2_FR_GRIF_IRFU to the resource control db...
Adding T2_US_Vanderbilt to the resource control db...
Adding T2_PL_Cyfronet to the resource control db...
Adding T2_AT_Vienna to the resource control db...
Adding T2_BE_IIHE to the resource control db...
Adding T2_BE_UCL to the resource control db...
Adding T2_BR_SPRACE to the resource control db...
Adding T2_BR_UERJ to the resource control db...
Adding T2_CH_CERN to the resource control db...
Adding T2_CH_CSCS to the resource control db...
Adding T2_CN_Beijing to the resource control db...
Adding T2_DE_DESY to the resource control db...
Adding T2_DE_RWTH to the resource control db...
Adding T2_EE_Estonia to the resource control db...
Adding T2_ES_CIEMAT to the resource control db...
Adding T2_ES_IFCA to the resource control db...
Adding T2_FI_HIP to the resource control db...
Adding T2_FR_GRIF to the resource control db...
Adding T2_FR_IPHC to the resource control db...
Adding T2_GR_Ioannina to the resource control db...
Adding T2_HU_Budapest to the resource control db...
Adding T2_IN_TIFR to the resource control db...
Adding T2_IT_Bari to the resource control db...
Adding T2_IT_Legnaro to the resource control db...
Adding T2_IT_Pisa to the resource control db...
Adding T2_IT_Rome to the resource control db...
Adding T2_KR_KISTI to the resource control db...
Adding T2_PK_NCP to the resource control db...
Adding T2_PL_Swierk to the resource control db...
Adding T2_PT_NCG_Lisbon to the resource control db...
Adding T2_RU_IHEP to the resource control db...
Adding T2_RU_INR to the resource control db...
Adding T2_RU_ITEP to the resource control db...
Adding T2_RU_JINR to the resource control db...
Adding T2_TR_METU to the resource control db...
Adding T2_TW_NCHC to the resource control db...
Adding T2_UA_KIPT to the resource control db...
Adding T2_UK_London_Brunel to the resource control db...
Adding T2_UK_London_IC to the resource control db...
Adding T2_UK_SGrid_Bristol to the resource control db...
Adding T2_UK_SGrid_RALPP to the resource control db...
Adding T2_US_Caltech to the resource control db...
Adding T2_US_Florida to the resource control db...
Adding T2_US_MIT to the resource control db...
Adding T2_US_Nebraska to the resource control db...
Adding T2_US_Purdue to the resource control db...
Adding T2_US_UCSD to the resource control db...
Adding T2_US_Wisconsin to the resource control db...
Retrieved 51 PNNs from https://cms-cric.cern.ch/
Done: Performing agent_resource_control
-------------------------------------------------------
-------------------------------------------------------
Start: Performing agent_upload_config
agent_upload_config: triggered.
agent_upload_config: Tweaking central agent configuration befre uploading
agent_upload_config: Testbed agent, setting MaxRetries to 0...
*** Upload WMAgentConfig to AuxDB ***
Executing wmagent-upload-config {"MaxRetries":0} ...
Pushing the following agent configuration:
{'AgentDrainMode': False,
 'CondorJobsFraction': 0.75,
 'CondorOverflowFraction': 0.2,
 'DiskUseThreshold': 85,
 'IgnoreDisks': ['/mnt/ramdisk'],
 'MaxRetries': 0,
 'NoRetryExitCodes': [70,
                      73,
                      8001,
                      8006,
                      8009,
                      8023,
                      8026,
                      8501,
                      50660,
                      50661,
                      50664,
                      71102,
                      71104,
                      71105],
 'SpeedDrainConfig': {'CondorPriority': {'Enabled': False, 'Threshold': 500},
                      'EnableAllSites': {'Enabled': False, 'Threshold': 200},
                      'NoJobRetries': {'Enabled': False, 'Threshold': 200}},
 'SpeedDrainMode': False,
 'UserDrainMode': False}
Done: Performing agent_upload_config
-------------------------------------------------------
-------------------------------------------------------
Start: Performing checks for successful Docker initialisation steps...
WMA_BUILD_ID:<PHONE_NUMBER>9744219753aea0c5924c8b274aa00de917f8a04144aae8c8972b47
dockerInitId:<PHONE_NUMBER>9744219753aea0c5924c8b274aa00de917f8a04144aae8c8972b47
OK
-------------------------------------------------------
Start: Performing local Docker image initialisation steps
deploy_to_container: Try Copying the host WMAgent.secrets file into the container admin area
deploy_to_container: Done
deploy_to_container: Updating WMAgent.secrets file with the current host's details
deploy_to_container: Double checking the final WMAgent.secrets file
deploy_to_container: Checking Certificates and Proxy
deploy_to_container: Checking Certificate lifetime:
deploy_to_container: Certifficate end date: Sep  7 12:04:12 2023 GMT
deploy_to_container: Checking myproxy lifetime:
deploy_to_container: myproxy end date: May 17 13:03:40 2023 GMT
deploy_to_container: OK
Done: Performing local Docker image initialisation steps
-------------------------------------------------------
-------------------------------------------------------
Start: Performing start_services
Starting Services...
starting couch...
Which couchdb:   With installation directory: /data/srv/wmagent/2.2.1rc2/install/couchdb
  With configuration directory: /data/srv/wmagent/2.2.1rc2/config/couchdb
Starting mysql...
starting mysqld_safe...
Checking MySQL Socket file exists...
Socket file exists: /data/srv/wmagent/2.2.1rc2/install/mysql/logs/mysql.sock
Checking Server connection...
Connection OK
Done: Performing start_services
-------------------------------------------------------
-------------------------------------------------------
Start: Performing start_agent
-------------------------------------------------------
Start sleeping now ...zzz...

@todor-ivanov , thanks for update and all details. There are extremely useful. Meanwhile, I want to make few observations:

I think packaging MariaDB within WMAgent container is a mistake, and only leads to complexity at different levels
I still think that each individual WMAgent component should be packaged as individual image and we should only need to start them separately
the YUI (or in this sense any other JavaScript library) should not be packages together with WMA components and rather put into separate FileServer which later can be used as CDN. This will allow to separate WMA from JS libraries
I still advocate for proper design/architecture discussion as Python maintenance exceed its benefits as service language

Hi @vkuznet thnks for the feedback.

I think packaging MariaDB within WMAgent container is a mistake, and only leads to complexity at different levels

For the time being I did it that way, just to assemble everything in one place and make it run and to deliver an agent ready for testing.  At a first glance this design may seem a complication, but it actually simplifies things a lot and gives some pretty good benefits and flexibility. One of which is being able to run multiple docker agents at the same machine. It would also help later if we decide to separate the wmagent from the schedd.
The initial plan was to use the database from the host, and not dealing with MariDB in this PR, but the possibility to connect to it were basically two - trough a socket or through the network interface. The socket is our default for the agent, but artificially sharing a sockets between the host and a container is a security breach. At the end the database installation was really easy and I also deliberately kept the commit related to the MariDB installation separate so that it all is in one place and we later can move that in a different Docker file, which could either be inherited within the agent or run as a separate container at the same host. There are few possible architectures the 3 main of which I can list (all with their benefits and drawbacks)

MariaDb inside the container - solid, easy to install, easy for maintenance, provides complete integration and wmagent isolation from the rest of the system, gives the flexibility to run multiple agents on the same host, the only drawback is  that for any sort of debugging one needs to be connected to the container in order to make queries to the database (we still can expose it through the network interface and try to overcome this complication, though). Honestly this is the best option for us if we do not decide to disassemble the whole agent component by component and running them in a separate container each (which to me is a mistake and unneeded complication providing no benefits and only future obstacles for real horizontal scaling)
Using the database from the host - at a first glance easy to install - turned out exactly the opposite. If we want to have more than one agent at the same host we cannot do that unless we redesign the database schema which would allow us to create different databases per agent. Not impossible, but needs code change in the agent and moves the complication centrally. Leads towards a design with incorporated single point of failure. At the end it is actually hurting a design towards horizontal scaling rather than helping in that regards. Also involves security issues if sharing a socket between the host and the container.
Using a different container providing a database inside the the same host, through possible socket sharing or network interface - does not overcome any of the issues from above, all concerns are valid here as well. On top of that it introduces additional complication in the installation process and maintaining yet another container is an operational overhead. I honestly cannot see benefits from this approach at all.
Having a centralized database server, serving multiple agents from multiple hosts, similarly to what we have with Oracle - This not only centralizes the system even more rather than distributing it, but would also suffer from an extreme network overhead. Also this design I think would be a  no go for the FNAL agents.

Given that the relational database is a monolithic piece of the agent which is actually under heavy stress and needs the least overhead, I do not see a good argument why should we separate it from the agent. When it comes to oracle there are other benefits and optimizations that Oracle provides and should worth paying the network price, but not sure how big that price is and if that would worth with MariaDB as well. I have not been around when such scale tests have been performed and eventual overhead being estimated, so this must be done before concluding with certainty here.
For the YUI part - I think we should completely get rid of it.

Oh, and I forgot to stress it enough. But if we want to go towards real horizontal scaling the place where we should slice is between the agent and the schedd, rather than thinking of breaking the whole agent in components. Our internal agent design we can remake later in many many ways. But the hard link between the agent and the schedd is the actual blocker for SI to be able get separate from the wmagent and implement their own schema of distributing the load among schedds, independently of the load in the agent. And since those loads have different nature (even though coming from the same place, the system hits different limitations with different origins in these two), it is natural to think of implementing different scalability schema in those levels. Something that in the CRAB system is marvelously done, and we should try it here as well.
FYI: @vkuznet @amaltaro @khurtado @klannon

Todor, it's great that you managed to start up most of the services. Thank you for this summary.
I have a few remarks to make:

From your logs, you seem to have installed an ancient WMAgent package to get CouchDB available in the host (you likely deployed the wmagent package instead of wmagentpy3). I would highly recommend you to remake this setup with the latest stable releases for CouchDB/wmagentpy3)

++ {"couchdb":"Welcome","uuid":"08e24bb5fe541033900035c5f2cf85fc","version":"1.6.1","vendor":{"version":"1.6.1","name":"The Apache Software Foundation"}}

I have the impression that your current setup initializes services twice. During the init process, it already starts up mariadb and couchdb, but this line seems to repeat it again:

Performing start_services
...

About MariaDB, I agree with Valentin that we should try to separate it from the WMAgent image (and this is what we have been discussing so far, having 3 containers in total: MariaDB, CouchDB, WMAgent). In addition, when we run WMAgent at CERN we do not actually need MariaDB. If you are concerned that T0 will not manage to run multiple replays in the same node, I think we will have to come back to that in the future. Having the ability to stop components without affecting services is extremely important IMO. In any case, it is fine to have MariaDB in right now as you run tests as real as possible.
It might be beneficial to isolate components in their own docker containers. But it's definitely not how we planned it to go at this stage. I also think it's important that we start it this way to: reduce the boostrap overhead in this new model and to get experience with this system.

This has been addressed with: https://github.com/dmwm/CMSKubernetes/pull/1393
Thanks Todor! Closing this one out.

Even though we considered the GH actions workflow, there was still a detailed that was missed and the CMSKubernetes PR actually broke our workflow:
https://github.com/dmwm/WMCore/actions/runs/5427386861/jobs/9870613631#step:5:858
The important lines when building the docker image are:
  ---> 4389876fc8ff
Step 34/43 : ADD bin $WMA_DEPLOY_DIR/bin
ADD failed: file not found in build context or excluded by .dockerignore: stat bin: file does not exist
Error: Process completed with exit code 1.

I think the best option here would be to actually break the wmagent dockerfile into two (wmagent-base and wmagent), this way we can actually benefit from a more reproducible build as well. I might give it a try during the weekend.

@amaltaro I am trying to understand the actual origin of this error.
I do not happen to find any  .dockerignore file in my fork of the CMSKuberenetes repository. And I am not quite sure, if splitting the image in two pieces may help in resolving this one here. Such an approach may of course be beneficial in terms of image size etc. , but I cannot see how would the situation change when it comes to this error.
Can you please add a pointer or two on where and how exactly we set up all the CI/CD processes? Do we have documentation on those?

@todor-ivanov as mentioned in our weekly meeting, the issue comes from the fact that in the GH actions we simply download the Dockerfile and try to build the image. As expected, it fails because we do not have the auxiliary scripts in the same directory (different than when we run it manually from a clone of CMSKubernetes).
The docker build workflow can be found in: https://github.com/dmwm/WMCore/blob/master/.github/workflows/docker_images_template.yaml#L28-L33

Hi @amaltaro, Thanks for providing this  really crucial line in your previous comment. Indeed downloading just a Dockerfile from the upstream CMSKubernetes repository could be the reason for the missing substructure of directories (meaning bin and etc), which in this case are needed. Do you happen to know the quick answer to:

Firstly, why are we using curl or web based tools in those github actions in order to link projects from the same realm, even though different  repositories, but still... under github and inside the dmwm area?
Secondly, why are we downloading only the Dockerfile. This seems quite restrictive to me. It is basically preventing us from doing  any ADD command from inside any of the Dockerfiles managing the image creation for WMCore/WMAgent services/components.

Saying that, I am now even more confident that splitting the image creation in two steps with this PR: https://github.com/dmwm/CMSKubernetes/pull/1394 is not the solution of this exact problem. Splitting the image in two is beneficial in many other ways and I do support this. I really like this approach. But for solving the current error in question I suggest we fix the CI/CD actions.
If we insist on using external tools inside our CI/CD pipeline and github actions instead of git or any other internally provided mechanism from github itself(if any), I may suggest a quick improvement by swapping web based tools like curl with subversion, and download a single directory. This [1] will give us the expected result.  NOTICE the swap of tree/master with trunk.

However, the comment you made here https://github.com/dmwm/CMSKubernetes/pull/1393/files#r1251146936 about docker build argument is very helpful and I totally support it. That's how we should have got started with this!
Having said that, here is a WMCore PR updating the GH action workflow:
https://github.com/dmwm/WMCore/pull/11638
and
https://github.com/dmwm/CMSKubernetes/pull/1394
Lastly, a new image for wmagent-base:pypi-20230703 has been uploaded to CERN registry.

Hi @amaltaro, Thanks for adopting this approach for passing the $TAG as an external parameter to the   Docker build command. This is indeed the proper approach to my opinion as well.
But I do not understand this argument:

the curl tool doesn't bother me too much, we are using Ubuntu image for the GH actions and it should be fine.

I did not object the usage of the curl command, because I did not like it or it was somehow related to the type of image we are using inside the CI/CD pipeline. I objected it because it lacks the way of recursively fetching contents from a github project, given a sub directory from the project. In this case these sub directories are docker/pypi/*. This will prevent all those Docker files from using any ADD command. And indeed if one goes there and checks them, one would immediately notice none of them contains ADD.
My point is - we should not violently mutilate the Docker containerization system functionality, just because we do not use the proper tool capable of doing recursive operations in our CI/CD pipeline.


Ammend NHS pathways data to show last 180 days

Closing as pathways data intermittently disappearing from source causing issues with pptx build. Replacing with other healthcare indicators.


No panel rendered

Thanks for the plugin!
I've just installed it but can't get it to work.
v1.5
I'm in resizable mode. No panel shows up. I've rebooted, played with the offsets.
Ended up also having an issue where it just blocked the whole game when the plugin was enabled, and I couldn't click anywhere while the fragments view was open. Could reliably fix this by disabling the plugin and reproduce it again by reenabling the plugin.
Sorry I can't give a clearer ticket / better repro. I'll keep trying it out and report back if I find anything to help narrow it down.

thanks for reporting! this is a bug introduced in v1.5 when there are no presets configured.
I'm working on pushing out a fix now. In the meantime, you can manually add a preset to the configuration file (settings.properties somewhere in your RL install):
io.tja.osrs.shattered_relics_fragment_presets.v1.all_presets=[{"fragments"\:["Unholy Warrior"],"name"\:"Sample preset"}]

just make sure not to delete all your presets until you've upgraded to v1.6.
apologies for the issue!

#23 is now in plugin-hub master, closing this issue. Thanks again for reporting @gricey432!


how can I get the Document Reference?

What the Document Reference means? If it means API documents, then please try to generate javadoc. Some comments can be found there. Basically the methods don't have documentation, but they are kind of a subset of the original matplotlib's functions. Please refer to the documents for the function effects.
I am preparing more examples in README to show more usages since that also helps to find missing methods to be added. Stay tuned!


Document why there is no JRE or a jre_headless for java derivations after 8

The variant jre11_headless is missing.

cc @edwtjo

Basically there is no jre variant of openjdk11 at all.

relates to https://bugs.openjdk.java.net/browse/JDK-8200132

@asbachb does not adoptopenjdk-jre-bin suffice?

I also encountered this issue today when I noticed an end user java based program I use references all of openjdk11 which is ~700Mb in size, whereas jre8 is ~100Mb in size.
@asbachb:

Basically there is no jre variant of openjdk11 at all.

Could you please elaborate on this? I mean, in all-packages.nix, jre8 is just a reference to opendjk8.jre which is just an output:
https://github.com/NixOS/nixpkgs/blob/90c0191735bc729acd36e4ba72ef3ffd88f679c6/pkgs/development/compilers/openjdk/8.nix#L83-L84
And in installPhase, it is used via $jre (vs $out). Perhaps this approach could be used for all versions of opendjk?

Sorry for the delay.
@edwtjo It might be sufficient. But the problem is what @doronbehar describes it seems that end user software is currently relying on a development kit not on a runtime environment.

There an make target legacy-jre-image which is also used to generate AdoptOpenJDKs jre variants.

@doronbehar Please do not add non-conversion issues to the conversion project.

I went over some java related tickets. I wonder if we should close this one.

For reference: https://github.com/NixOS/nixpkgs/blob/2129356a6405cc342e860d87623d3823e8ee3b8c/doc/languages-frameworks/java.section.md#L51-L58

Actually I thought about that issue again. I wonder if it really makes sense to have an individual jre for each application. As these jre are unlikely to match they'd need to be downloaded for every application where as a generic jre could be shared by most of the applications.


ddeboer/data-import null values for CSV

What is the proper format for a null value when importing CSV file using "ddeboer/data-import" bundle in Symfony2?
The "CSVReader" class uses "SplFileObject" class and it escapes anything. If I use "\N" for instance it changes it to "\N".
I've tried empty value formats, like ..,,.. or ...,"",... and it always says that the value is not correct for an integer field.
The mysql field is nullable or allow null, I must be able to use null values and not 0 for example.
Thanks!

I've found a solution for this problem and it's using a CallbackItemConverter.
So each item is checked if it's a string and its value is "NULL" change it to a PHP null value.
$nullConverter = new CallbackItemConverter(function ($item) {
foreach ($item as $key => $value) {
if (is_string($value)) {
$item[$key] = $value == 'NULL'? null: $value;
}
}
return $item;
});
$workflow->addItemConverter($nullConverter)
Actually you can change the string "NULL" to any value you use in your CSV export, and use the same converter to convert any other fields while the workflow is processing.

@vsalvans NULL values comes are from RDBMS systems, CSV/Excel or similar formats doesn't need such cases. You have to find you own way to it.


GCF file split: concepts

cc @saturnism
N.B: I ran mvn dependency:analyze -DignoreNonCompile to remove unused dependencies, but this does not detect unused test dependencies.
(If someone could point me to a command that identifies those, that would be appreciated. 🙂)

@ace-n -DignoreNonCompile ignores test dependencies. Cant you just not use that flag?

Kurtis - I'm looking for something that doesn't ignore test dependencies. 🙂

Thanks Kurtis - but I'm looking for something that doesn't ignore test dependencies. 🙂
(In any case, this is more a "nice-to-have" than anything.)

Right. I'm suggesting you don't use the flag that tells it to ignore test dependencies.
From the ignoreNonCompile doc:

Ignore Runtime/Provided/Test/System scopes for unused dependency analysis.

So presumably the default is to include "runtime/provided/test/system scopes".

Ah - my bad, I assumed some sort of flag was required! 🙃


[Bug] Some RPC methods don't expose BlockId

Component
provider, pubsub
What version of Alloy are you on?
v0.1.1
Operating System
macOS (Apple Silicon)
Describe the bug
Some rpc methods are documented as also accepting a block height parameter, but they currently don't expose it.
Example: https://github.com/alloy-rs/alloy/blob/041c7c47b33907523cbe4023db03ae7bb57bf7bd/crates/provider/src/provider/trait.rs#L250-L254
This is the case for: get_account, get_balance, get_proof (as far as I can tell, I didn't check all other methods)
If you have a solution in mind I'm happy to open a PR for it

this can be set via RpcWithBlock::block_id: https://docs.rs/alloy-provider/latest/alloy_provider/struct.RpcWithBlock.html#method.block_id


ndim for DataContainer

Our DataContainer class is (almost) a numpy wrapper.
Atm, it is only a numpy wrapper due to
https://github.com/TomographicImaging/CIL/blob/74a4231eb6f458fcbf60a9c81ec0295e63c0be1c/Wrappers/Python/cil/framework/framework.py#L2051-L2058
However, we miss some important properties based on the Array API. One of them is the ndim. This is important for future enhancements of CIL and will allow CIL to be used within other libraries, e.g., scipy, dask, jax.
See also #976, #978

Instead of issues, could this and #1244 be captured in a discussion, please?
The transformations you are suggesting are very important and we are going to discuss these next week. I think they are just not issues yet.


lex - When an error happens, how can I display all tokens matched so far?

When an error happens (lark.exceptions.UnexpectedCharacters), there is usually some "Previous tokens" information such as this:
Previous tokens: Token('__ANON_0', 'CL79')`

That only seems to contain the token immediately preceding the error, but not the ones before.
Am I doing something wrong, or is there a way to display all tokens matched so far?

It's possible to collect all the tokens by writing a postlexer.
Another way is to parse using the interactive parser.
Mind if I ask what you need it for?

@erezsh I was trying to see what tokens were being matched so I could debug the lexer rules

Can I still use the postlexer in my case, where an exception is thrown (so the lexing process isn't yet complete)?

Yes, the postlexer gets the tokens one by one, so if you save them somewhere (like in a global list, or inside the postlexer instance), you will have the latest list.
Lark doesn't save those tokens, because we want to support memory-efficient streaming. But perhaps we could do it when debug=True.

Lark doesn't save those tokens, because we want to support memory-efficient streaming. But perhaps we could do it when debug=True.

That would be wonderful for ease of development


Bleve search doc updates

Documentation ticket: https://mattermost.atlassian.net/browse/MM-37211
Updated:

Install, Deploy, Upgrade, and Scale > Upgrade Mattermost > Upgrading Mattermost Server

Updated the Bleve notes to clarify that Bleve indexes can be migrated without reindexing between MM versions
Updated page formatting to improve readability and clarity of process steps

Please take a look at the rich diff of the rendered result:
https://github.com/mattermost/docs/pull/4924/files?short_path=fcc1287#diff-fcc12871eb71ce483f8dbcabf9b5a0a79ae12f56e7b91a91b6fd3fba83b1a3bd
A lot of useful content was deleted, and I believe it's by accident.

Thank you for letting us know about this issue! This content was removed by mistake and is being added back via https://github.com/mattermost/docs/pull/4936.


Run one single linter for file?

Is there a way to run a single linter on a file instead of all of them? For python i have isort and yapf set, but once in a while I'd like to only run isort (to not muck up the git blame a huge file from an old project too much). Is there a way to define a binding for just one?
I've tried stuff like :call ale#engine#RunLinters(bufnr(''), ['isort'], 1), but this gives me an error:
Error detected while processing function ale#engine#RunLinters[4]..<SNR>164_RemoveProblemsForDisabledLinters:
line    6:
E121: Undefined variable: name

I'm guessing the linter arg is not just an array of strings.
thanks!

So "linters" are for checking for problems, and "fixers" are for fixing things. For selecting the linters to run, you can configure the g:ale_linters variable, and for selecting fixers to run, you can configure the g:ale_fixers variable. You can look up the documentation for either with :help.
It sounds like you want to only run a single fixer in some cases. If you want to use a different setting for a specific project, you can use the b: buffer-local variable versions instead. The easiest way to configure the buffer-local variables is with the option for using different settings based on regular expressions for absolute filesystem paths. See :help g:ale_pattern_options for the documentation on that.
If you wanted to call some function to run a specific fixer, you could temporary change the b:ale_fixers variable, run :ALEFix, and then change the variable back again.

I'm trying to write a function that does this, but I can't seem to get the variable reset correctly:
function! RunISort()
    let b:ale_fixers = ['isort',]
    call ALEFix()
    let b:ale_fixers = ['isort', 'autopep8']  " or ideally  = g:ale_fixers['python']
endfunction

This gives me the behavior that :ALEFix runs isort and autopep8 initially, :call RunISort() only runs isort, but afterwards :ALEFix doesn't seemed to have picked up on the last assignment because it keeps only running isort and :echo b:ale_fixers returns ['isort', 'autopep8'] as one would expect. What am I doing wrong?
I'm quite new to vimscript, so my apologies if this is just due to my misunderstanding of how scope works in vim...

After some help in #ale-vim, the solution:
function! RunISort() abort
    try
        let b:ale_fixers={'python': ['isort',]}
        ALEFix
    finally
        unlet b:ale_fixers
    endtry
endfunction

Hello, I would like to propose enhancement to ALEFix. It would be nice if it accepted arguments with fixer names, and if not passed, behaved as it does now.
Naive implementation would be something like this.
function! s:Fixers(...)
    try
        if len(a:000)
            let b:ale_fixers={&filetype: a:000}
        endif
        ALEFix
    finally
        unlet! b:ale_fixers
    endtry
endfunction

command! -nargs=* ALEFixParams call s:Fixers(<f-args>)


adding map_filepath to floorplan schema

Adds map_filepath to floorplan schema.
Use case:
While working with APIs it's more convenient to work with urls/endpoints to download the image, we would like to support exporting/importing a project with a flat file. Working with urls can be difficult in that case, since they usually require authentication and also the traffic can be restricted to unknown locations due to security reasons.
For this reason, we would like to import and export to project in a way, so that it would contain the image files in addition to the OpenIntent definition. This requires that each floor plan would have a reference to the file that has the map image.
This wouldn't replace the map_uri, but just provide an alternative way to reference the images.
Let me know if you want to discuss this further on a call.

My colleague noted that since the property called map_uri, we could just use the same property to handle image files and remote images. The description is talking about "url endpoint", so it would require just changing that to describe both use cases.


Add argsort to BatchedArray

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (e07db53) 100.00% compared to head (7fe1789) 100.00%.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #632   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           31        31
  Lines         2635      2644    +9
  Branches       339       340    +1
=========================================
+ Hits          2635      2644    +9

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


docs(installations):  changed a package download command in overview …

…page to match the imported package in the usage page
Description
the overview page was install a package (firebase_app_installations) and in the usage page importing another one (flutterfire_installations), it's confusing even though both package offer same functionality so I did changed the download command to get the flutterfire_installations package
Related Issues
no issue
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process. Updating the pubspec.yaml and changelogs is not required.

[ x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[ x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[ x] All existing and new tests are passing.
[ x] I updated/added relevant documentation (doc comments with ///).
[ x] The analyzer (melos run analyze) does not report any problems on my PR.
[ x] I read and followed the Flutter Style Guide.
[ x] I signed the CLA.
[ x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change.
[x ] No, this is not a breaking change.

Hey @ABDERRAHMANE-OUALI, thank you for bringing to our attention. Could you revert this change and instead update the import in the "installations/usage.mdx" file?
import should look like this:
import 'package:firebase_app_installations/firebase_app_installations.dart';

@russellwheatley ok, but how can I revert this pull request ?
this is my first contribution :)


fix the sort of images version on GetUpdateAvailableForDevice function

related to THEEDGE-2326
Signed-off-by: mgold1234<EMAIL_ADDRESS>Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
What is it?

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[ ] Tests update
[ ] Refactor

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] I run make pre-commit to check fmt/vet/lint/test-no-fdo

Test results are available in Ibutsu:
https://url.corp.redhat.com/ibutsu-runs-01922a3b-c802-41e7-8d1c-b1cff5a598ab


confidence parameter not working with opencv-python v <IP_ADDRESS>

The 'confidence' parameter in the 'locate_on_screen' method doesn't work with opencv-python v <IP_ADDRESS>
The stacktrace output looks like the library is not installed, but it is not.
The error appeared when upgrading from version <IP_ADDRESS>. Rolling back opencv to an older version solved the problem.

obj = pyautogui.locateOnScreen(self.locator, region=region, confidence=confidence, minSearchTime=search_time)
File "C:\Python\Python38\lib\site-packages\pyautogui_init_.py",
line 175, in wrapper return wrappedFunction(*args, **kwargs)
File "C:\Python\Python38\lib\site-packages\pyautogui_init_.py",
line 213, in locateOnScreen return pyscreeze.locateOnScreen(*args, **kwargs)
File "C:\Python\Python38\lib\site-packages\pyscreeze_init_.py",
line 373, in locateOnScreen retVal = locate(image, screenshotIm, **kwargs)
File "C:\Python\Python38\lib\site-packages\pyscreeze_init_.py",
line 353, in locate points = tuple(locateAll(needleImage, haystackImage, **kwargs))
File "C:\Python\Python38\lib\site-packages\pyscreeze_init_.py",
line 253, in _locateAll_python raise NotImplementedError('The confidence keyword argument is only available if OpenCV is installed.')
NotImplementedError: The confidence keyword argument is only available if OpenCV is installed.

Could you provide the code you were using that resulted in this issue? I was able to replicate that error but it was fixed by installing python-opencv.

Could you provide the code you were using that resulted in this issue? I was able to replicate that error but it was fixed by installing python-opencv.

- @jtranquilli
That can not be true because there is no "python-opencv" package. You likely meant "opencv-python". Your comment was absolutely no help and not regarding the topic.
I'm having the same issue.
python code:
py import pyautogui ImageBox = pyautogui.locateOnScreen(image, grayscale=grayscale, confidence=confidence)
pip-freeze & py -v:
opencv-python==<IP_ADDRESS> PyAutoGUI==0.9.54
Python 3.11.9  @ conda 24.7.1 @ Win11
Error:
File "%User%\anaconda3\Lib\site-packages\pyscreeze\__init__.py", line 273, in _locateAll_pillow     raise NotImplementedError('The confidence keyword argument is only available if OpenCV is installed.') NotImplementedError: The confidence keyword argument is only available if OpenCV is installed.

I have a working version opencv-python <IP_ADDRESS> installed


Multiple issues with MH_FILESET loading on an iOS 17 kernelcache

Version and Platform (required):

Binary Ninja Version: 3.5 Stable
OS: macOS

Bug Description:
Loading an iOS 17 MH_FILESET cache appears to have multiple issues with which segments are properly loaded and where.
Steps To Reproduce:

Open the following kernelcache located in the root of this iOS update package

As shown in the below image, the segment which contains the actual file header and should be loaded into memory at 0xfffffff027004000 is not loaded. Additionally, whichever header was loaded at the top of the file (possibly the first image? however this address does not correspond with anything in other tools) is not the appropriate header, and was not typed automatically whatsoever, indicating deeper issues with the header processing.

Seems there's some progress with this in the -dev 3.6 path, as its loading MH_FILESET pretty well, but the big issue is that it doesn't seem to 1) actually group using the component API, and that the parsing of the symbol names doesn't occur leaving you living in sub_* land...
My presumption is that whatever is to pick up the symbols isn't aligned yet, but most of the other support is there.


Loaded label above table only counts results of the page

The 'Loaded: 10 results' label above the table in the viewer only counts the results of the page, not the total result. It is confusing as the actual table has more results.

True! And the count can be seen at the bottom right of the displayed table anyway:
"Rows per page: 10       1-10 of 66".
Better remove the display of "Loaded ...".