Add new section for Server Integration topics

Signed-off-by: Drew Moseley<EMAIL_ADDRESS>

This is the 1.5.x version but we can wait for the content in master before merging.


We should be able to track connection status of Device objects other than by callback

The suggestion is that it should be possible to detect whether Device is connected from the object itself. This would allow a client to iterate the vector of devices and only handle those that are connected - without having to separately manage tracking connection/and disconnection.

OK, though for completeness ... this was not quite what was delivered. We now have a top level DroneCore mechanism that can check if a particular UUID is connected. What this originally asked for is a mechanism for a device to check itself for a connected state.
The semantics are slightly different but I am not sure if what has been done is better or not. Really depends on whether you'd write code where you have a device and check its connection status before "doing stuff" or you would instead manage all connection/disconnection at a single top level location. My object orientation head makes me think that we might well want device-centric behaviour, but not sure how realistic that is.


Does this script still work?

As of now there are script executors that still work, and can execute scripts like Infinite Yield, when I tried this script in my executor it didn't do anything.

YousufSSyed

as of right now, seems like it doesn't work
might work on krampus but idk

This UI Library is made with a Drawing library, so these are all the dependencies in order for the script to work:
ESP:

Aimbot:

And yes this script works just fine there is no bug or any logical error, your issue is in the nature of an executor / unsupported functions issue which means I can't help you here in any way unless you tell your script developer to add support for these required functions.


Support systemd.

Description
Add systemd support for ubuntu
Issues Resolved
#43

I've tested the source based systemd support again to make sure it was in fact working. Here's what I get on an Ubuntu 16.04 system:
root@source-ubuntu-1604:/usr# systemctl status nginx
* nginx.service - The nginx HTTP and reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; static; vendor preset: enabled)
   Active: active (running) since Mon 2016-12-05 17:45:46 UTC; 3min 9s ago
  Process: 8674 ExecStop=/bin/kill -s QUIT $MAINPID (code=exited, status=0/SUCCESS)
  Process: 8657 ExecReload=/bin/kill -s HUP $MAINPID (code=exited, status=0/SUCCESS)
  Process: 8679 ExecStartPre=/opt/nginx-1.10.1/sbin/nginx -t (code=exited, status=0/SUCCESS)
 Main PID: 8685 (nginx)

On CentOS 6 we get init properly
On Ubuntu 14.04 we get upstart properly
On Ubuntu 16.04 we get systemd properly
I'm going to close this out at this point since it seems like everything is working as designed.


Random and enchanted_book

The output of /version on my server is: paste here
The output of /version denizen on my server is: paste here
(If relevant) A link to a paste of a script demonstrating the issue: paste here
The issue is:
Describe your issue in full here
About random I wonder if there is any possibility of randomization according to probability.
- random :
              - probability 50:
              - execute as_server "say hello"
              - probability 10:
              - execute as_server "say hi"
              - probability 40:
              - execute as_server "say ok"

I want to know how enchanted_book item is written to keep its original function [enchanted]

For the random, like this:
- define value <util.random.int[1].to[100]>
- if <def[value]> < 50:
  - announce "hello"
- else if <def[value]> < 60:
  - announce "hi"
- else:
  - announce "ok"

I don't know what you're asking regarding enchanted books

For the random, like this:
- define value <util.random.int[1].to[100]>
- if <def[value]> < 50:
  - announce "hello"
- else if <def[value]> < 60:
  - announce "hi"
- else:
  - announce "ok"

I don't know what you're asking regarding enchanted books

I hope you can give me an example of minecraft: protection level 1


Centrifugo v3 compatibility

Hello! Currently readme states that this package:
Compatible with latest Centrifugo 2.8.5

Recently we released Centrifugo v3 with many improvements and new features. Unfortunately there are also several backwards incompatible changes. Everything described in Migrating to v3 guide.
Regarding this package I noticed the following things to be updated:

Response format of channels API method changed, and it now accepts pattern argument
Behavior of history API changed - it does not return publications by default, only current stream position, to return publications limit argument must be set to a value greater than 0.
Some additional fields added  to publish, broadcast, history methods (though new params fields do not break requests, just extend functionality)
Also, some deprecated option names like secret now removed from Centrifugo, it's now token_hmac_secret_key - so maybe this package should match updated option names.

Would be awesome to have laravel-centrifugo inherit all of this. I am ready to help answering any questions during migration.
And thanks for an awesome package @denis660!

Thanks! I will definitely update to v3

Сделал обновление до Centrifugo  v 3
https://github.com/denis660/laravel-centrifugo/releases/tag/2.0.0

Thanks!


AttributeError: module 'gmplot' has no attribute 'GoogleMapPlotter'

How can I fix this problem? I just copied the test code to run

I face this issue too.

The Readme should now be updated to show the correct syntax. If it doesn't
solve your problem, can you share the exact piece of code you're trying to
execute?

This is definitely not fixed. I installed and ran the unit test...got the error. Copy and pasted the readme code...got the error. Copied the contents of the gmplot directory into the script directory, got a new error:
from gmplot import gmplot
ImportError: cannot import name gmplot

Ok, let's get this fixed. Can you share your exact commands, including
cloning and any cd'ing, to reproduce the error?

An uninstall/reinstall of the package fixed the issue. :)


Avoid sticky errors in subsequent *Init.Do calls

When *Init.Do executes the *Init's f function, it stores the error
in the *Init's err field and returns ini.err. If we call an *Init's
Do method again, and there is no error in the subsequent call,
ini.err remains unchanged, and *Init.Do returns it again.
This becomes an issue for the "hugo server" command, which calls
h.init.data.Do within *HugoSites.Data every time the site is
rebuilt. If "hugo server" is started with a valid data file, and
that data file is edited to make it invalid (e.g., with poorly
formatted JSON), then any attempt to fix the data file only causes
Hugo to return the same error.
The fix is to ensure that *Init.Do resets ini.err to nil with each
call. The err field of lazy.Init is not used outside of *Init.Do,
so I removed it in order to simplify the code slightly and ensure
that errors would not be part of an *Init's state. Instead, this
change declares an error variable at the top of *Init.Do and
assigns it in place of ini.err.
After making this change, the go race detector began reporting
a race in TestImageOps between one goroutine created in
*Site.renderPages:
go pageRenderer(ctx, s, pages, results, wg)

...and another goroutine created in *Init.withTimeout:
	go func(cv chan verr) {
		v, err := f(ctx)
		c <- verr{v: v, err: err}
	}(c)

The race affected *pageContentOutput.content. To
prevent this race, I added getter and setter methods for
pageContentOutput.content that make use of a sync.RWMutex.
Otherwise, this field is assigned/read directly by multiple
goroutines.
I made other changes in order to refactor the code and prevent
unintentional memory sharing:

Removed lazy.AddWithTimeout and its tests. This function is not
called anywhere, and removing it makes it slightly easier to
make sense of the lazy package.

Edited the *Init.withTimeout to remove a redundant select
statement.

Edited *onceMore.Do to return an error, making it a bit
more straightforward to share errors between its function
argument and its caller without risking unintentional memory
sharing.

Fixes #7043

Seems like my changes trip the go race detector (https://github.com/gohugoio/hugo/runs/4280641814?check_suite_focus=true#step:16:203), probably because of something having to do with goroutine safety in *Init.Do. I'll take another look later this week.

Can you post the full stack trace of the data race on content? I'm not sure I follow this.

I think I have a simpler fix for this, will create a PR in a minute.

@bep The stack trace of the data race is in this gist, beginning at line 106. The gist records the output of removing the locking logic from my PR and running go test ./... -race.
Any ideas as to why this race took place after removing Init.err and using a local variable instead? This seemed strange to me because, as far as I know, no other part of the codebase referred to the err field outside of *Init.Do.
On another note, after PR #9232, I think there needs to be a way to ensure that Reset is called after Do, otherwise this bug could come back in a way that's even more difficult to troubleshoot, since assigning *Init.err to nil is a side-effect of Reset. Maybe at least indicate this as a comment above Do?


SDK attempts to connect to firebasedynamiclinks-ipv6.googleapis.com from IPv4 network.

[REQUIRED] Step 1: Describe your environment

Xcode version: 11.3.1 (11C504)
Firebase SDK version: Firebase/Core 6.13.0
Firebase Component: DynamicLinks
Component version: 4.0.7
Installation method: CocoaPods

[REQUIRED] Step 2: Describe the problem
Firebase Dynamic Links attempts to connect to an IPv6 domain that it's not capable of connecting to (firebasedynamiclinks-ipv6.googleapis.com).
Steps to reproduce:
Configure Dynamic Links in Firebase.
Open up a Dynamic Link within the iOS 12.2.2 simulator.
Check the logs, no Firebase attribution is made, the error is logged instead.
Task <6B76A91C-2DD5-4B8B-8B92-8FBAA08AEBAA>.<3> finished with error [-1003] Error Domain=NSURLErrorDomain Code=-1003 "A server with the specified hostname could not be found." UserInfo={NSUnderlyingError=0x600000c95ad0 {Error Domain=kCFErrorDomainCFNetwork Code=-1003 "(null)" UserInfo={_kCFStreamErrorCodeKey=8, _kCFStreamErrorDomainKey=12}}, NSErrorFailingURLStringKey=https://firebasedynamiclinks-ipv6.googleapis.com/v1/installAttribution?key=AIzaSyA0dh9J7ITwtRjMAWiaY_ZvxM1AiHXBAKo, NSErrorFailingURLKey=https://firebasedynamiclinks-ipv6.googleapis.com/v1/installAttribution?key=AIzaSyA0dh9J7ITwtRjMAWiaY_ZvxM1AiHXBAKo, _kCFStreamErrorDomainKey=12, _kCFStreamErrorCodeKey=8, NSLocalizedDescription=A server with the specified hostname could not be found.}

Relevant Code:
The invalid domain (firebasedynamiclinks-ipv6.googleapis.com) is present in this file:
https://github.com/firebase/firebase-ios-sdk/blob/c8625ec52eb076069efb87ca2e4819627e449efb/FirebaseDynamicLinks/Sources/FIRDynamicLinkNetworking.m
Executing the nslookup command shows that the domain doesn't exist:
$ nslookup firebasedynamiclinks-ipv6.googleapis.com

Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53

Non-authoritative answer:
*** Can't find firebasedynamiclinks-ipv6.googleapis.com: No answer

You can compare this to the other domains in the same file:
nslookup firebasedynamiclinks-ipv4.googleapis.com

Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53

Non-authoritative answer:
Name:   firebasedynamiclinks-ipv4.googleapis.com
Address: <IP_ADDRESS>

nslookup firebasedynamiclinks.googleapis.com

Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53

Non-authoritative answer:
Name:   firebasedynamiclinks.googleapis.com
Address: <IP_ADDRESS>

It is possible to view the IP v6 address:
dig firebasedynamiclinks-ipv6.googleapis.com AAAA

; <<>> DiG 9.10.6 <<>> firebasedynamiclinks-ipv6.googleapis.com AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49237
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;firebasedynamiclinks-ipv6.googleapis.com. IN AAAA

;; ANSWER SECTION:
firebasedynamiclinks-ipv6.googleapis.com. 194 IN AAAA 2a00:1450:4009:806::200a

;; Query time: 45 msec
;; SERVER: <IP_ADDRESS>#53(<IP_ADDRESS>)
;; WHEN: Fri Mar 06 16:02:41 GMT 2020
;; MSG SIZE  rcvd: 97

I've also reproduced the issue in the iOS 13.3 simulator.
I've also reproduced the issue in SDK version 6.15.0.

A similar issue was filed in the .NET SDK in 2018 and was resolved by updating the domain to firebasedynamiclinks.googleapis.com, see https://github.com/googleapis/google-api-dotnet-client/issues/1186 for more details.

Thanks for reporting. We will investigate this and report back.

I also have this error when launching the app.

Hi, in case its helpful I also see this issue.
2020-03-20 21:53:30.115019-0400 [22075:10757038] Task <5562A0D9-05C7-4593-960F-C09CBE08188C>.<2> finished with error [-1003] Error Domain=NSURLErrorDomain Code=-1003 "A server with the specified hostname could not be found." UserInfo={NSUnderlyingError=0x600001670810 {Error Domain=kCFErrorDomainCFNetwork Code=-1003 "(null)" UserInfo={_kCFStreamErrorCodeKey=8, _kCFStreamErrorDomainKey=12}}, NSErrorFailingURLStringKey=https://firebasedynamiclinks-ipv6.googleapis.com/v1/installAttribution?key=AIzaSyAS6sGtB1V55M-g8c4Z3rzZaM5DTCnaDZM, NSErrorFailingURLKey=https://firebasedynamiclinks-ipv6.googleapis.com/v1/installAttribution?key=AIzaSyAS6sGtB1V55M-g8c4Z3rzZaM5DTCnaDZM, _kCFStreamErrorDomainKey=12, _kCFStreamErrorCodeKey=8, NSLocalizedDescription=A server with the specified hostname could not be found.}
I'm running:
Xcode Version 11.2.1 (11B53)
iPad running iOS 13.3.1

Here are the Firebase pod versions that I'm using. I've run 'pod update' to get the latest:
  - Firebase/Analytics (6.20.0):
    - Firebase/Core
  - Firebase/Core (6.20.0):
    - Firebase/CoreOnly
    - FirebaseAnalytics (= 6.3.1)
  - Firebase/CoreOnly (6.20.0):
    - FirebaseCore (= 6.6.4)
  - Firebase/Database (6.20.0):
    - Firebase/CoreOnly
    - FirebaseDatabase (~> 6.1.4)
  - Firebase/DynamicLinks (6.20.0):
    - Firebase/CoreOnly
    - FirebaseDynamicLinks (~> 4.0.7)
  - FirebaseAnalytics (6.3.1):
    - FirebaseCore (~> 6.6)
    - FirebaseInstallations (~> 1.1)
  - FirebaseAnalyticsInterop (1.5.0)
  - FirebaseAuthInterop (1.1.0)
  - FirebaseCore (6.6.4):
    - FirebaseCoreDiagnostics (~> 1.2)
    - FirebaseCoreDiagnosticsInterop (~> 1.2)
  - FirebaseCoreDiagnostics (1.2.2):
    - FirebaseCoreDiagnosticsInterop (~> 1.2)
  - FirebaseCoreDiagnosticsInterop (1.2.0)
  - FirebaseDatabase (6.1.4):
    - FirebaseAuthInterop (~> 1.0)
    - FirebaseCore (~> 6.0)
  - FirebaseDynamicLinks (4.0.7):
    - FirebaseAnalyticsInterop (~> 1.3)
    - FirebaseCore (~> 6.2)
  - FirebaseInstallations (1.1.0):
    - FirebaseCore (~> 6.6)

Hello, I've tested my IPv6 using https://test-ipv6.com and it says I (my ISP) don't have any IPv6 compatibility.
And this causes my app freeze on the first launch. Relaunching helps. I'm also getting the same error messages

I can also definitely reproduce. And if this occurs in your app's startup sequence, it will hang the app, with iOS killing it after 20-odd seconds...

Is there a fix for this? Dynamic links don't open the app and I am sure this might be the issue. If the attribution does not install, then the app will not open. Please help

Is there a fix for this? Dynamic links don't open the app and I am sure this might be the issue. If the attribution does not install, then the app will not open. Please help

I was able to work around this by moving Firebase initialization out of the app's startup sequence, i.e. delaying it much as possible, and not having it in AppDelegate.init or AppDelegate.appDidFinishLaunching. This fix hasn't gone into production for our app yet, so we'll still need to see if it actually works. It's a hard one to reproduce, but when you do, you can't get out of it, unless you reinstall the app.

Is there a fix for this? Dynamic links don't open the app and I am sure this might be the issue. If the attribution does not install, then the app will not open. Please help

I was able to work around this by moving Firebase initialization out of the app's startup sequence, i.e. delaying it much as possible, and not having it in AppDelegate.init or AppDelegate.appDidFinishLaunching. This fix hasn't gone into production for our app yet, so we'll still need to see if it actually works. It's a hard one to reproduce, but when you do, you can't get out of it, unless you reinstall the app.

This is kind of an issue for me as I’m working with a Flutter app and barely have to touch the actual iOS code, but I will try and take a look at that to see what’s going on. I did realize that the links work on the simulator but not on a real device, which is weird

I ran into this myself today. Firebase 6.23.0, using Flutter.  Happy to provide any data but basically I'm getting the same errors that others have listed.

Yeah , some problem.
2020-05-03 20:05:48.492994+0300 spade[839:408719] Task <4D1203B1-9569-4E97-84BB-5929DF9C4FD5>.<2> HTTP load failed, 0/0 bytes (error code: -1003 [12:8]) 2020-05-03 20:05:48.494219+0300 spade[839:408719] Task <4D1203B1-9569-4E97-84BB-5929DF9C4FD5>.<2> finished with error [-1003] Error Domain=NSURLErrorDomain Code=-1003 "A server with the specified hostname could not be found." UserInfo={NSUnderlyingError=0x283d02c70 {Error Domain=kCFErrorDomainCFNetwork Code=-1003 "(null)" UserInfo={_kCFStreamErrorCodeKey=8, _kCFStreamErrorDomainKey=12}}, NSErrorFailingURLStringKey=https://firebasedynamiclinks-ipv6.googleapis.com/v1/installAttribution?key=AIzaSyCOlWqHh0GQndcTenatwf1VuZwhJ6HQKOM, NSErrorFailingURLKey=https://firebasedynamiclinks-ipv6.googleapis.com/v1/installAttribution?key=AIzaSyCOlWqHh0GQndcTenatwf1VuZwhJ6HQKOM, _kCFStreamErrorDomainKey=12, _kCFStreamErrorCodeKey=8, NSLocalizedDescription=A server with the specified hostname could not be found.}

Yeah , same problem.
`2020-05-03 20:05:48.492994+0300 [839:408719] Task <4D1203B1-9569-4E97-84BB-5929DF9C4FD5>.<2> HTTP load failed, 0/0 bytes (error code: -1003 [12:8])
2020-05-03 n20:05:48.494219+0300 [839:408719] Task <4D1203B1-9569-4E97-84BB-5929DF9C4FD5>.<2> finished with error [-1003] Error Domain=NSURLErrorDomain Code=-1003 "A server with the specified hostname could not be found." UserInfo={NSUnderlyingError=0x283d02c70 {Error Domain=kCFErrorDomainCFNetwork Code=-1003 "(null)" UserInfo={_kCFStreamErrorCodeKey=8, _kCFStreamErrorDomainKey=12}}, NSErrorFailingURLStringKey=https://firebasedynamiclinks-ipv6.googleapis.com/v1/installAttribution?key=, NSErrorFailingURLKey=https://firebasedynamiclinks-ipv6.googleapis.com/v1/installAttribution?key=, _kCFStreamErrorDomainKey=12, _kCFStreamErrorCodeKey=8, NSLocalizedDescription=A server with the specified hostname could not be found.}

I got the same issue, Is there any update?

Same issue here

same issue here

Same issue here, too.

Any solution for this?

Same here...

It is working by uploading to hardware device.

Same issue.

same issue

Hello, Folks! I have same issue here.
Is there some update?

Hello, Folks! I have same issue here. Seems that this merge request didn't help.

You can use an hardware iphone or android device instead of a simulator. I dont have seen any issues in live environment. Only by testing it in simulator.

I am testing it on my iPhone 7 (
2020-06-11 10:03:23.734395+0300 Zakaz.ua[788:164795] Task <7825F82C-02F0-48EA-956A-08D2B422CD39>.<3> HTTP load failed, 0/0 bytes (error code: -1003 [12:8])
2020-06-11 10:03:23.737720+0300 Zakaz.ua[788:164792] Task <7825F82C-02F0-48EA-956A-08D2B422CD39>.<3> finished with error [-1003] Error Domain=NSURLErrorDomain Code=-1003 "A server with the specified hostname could not be found." UserInfo={NSUnderlyingError=0x281c732a0 {Error Domain=kCFErrorDomainCFNetwork Code=-1003 "(null)" UserInfo={_kCFStreamErrorCodeKey=8, _kCFStreamErrorDomainKey=12}}, NSErrorFailingURLStringKey=https://firebasedynamiclinks-ipv6.googleapis.com/v1/installAttribution?key=AIzaSyDoHkOhXlPDs5OpyL9N0NIHn0QzUhdE4eY, NSErrorFailingURLKey=https://firebasedynamiclinks-ipv6.googleapis.com/v1/installAttribution?key=AIzaSyDoHkOhXlPDs5OpyL9N0NIHn0QzUhdE4eY, _kCFStreamErrorDomainKey=12, _kCFStreamErrorCodeKey=8, NSLocalizedDescription=A server with the specified hostname could not be found.}

that's fresh

Did you double checked you have activated dynamiclinks api with that api key?

@rikivesa yes - triple time only for today.
Only thing bothering me is that this api key is restricted to my app bundle. It should not, but can it be related to this issue?
also my Firebase/DynamicLinks version is 6.20.0 if this is related

Than i dont know what the issue can be. With our app it wasnt working on simulator but it was working in hardware device and also in live environment.

@harrygt @fbuland pls verify - Is this issue reproduced for your projects on real equipment and in a real environment or not?

I reproduced this issue on a physical iPhone 5s with iOS 12.4.6 and a physical iPhone 6s with iOS 13.4.1. Both are using Wifi connection.
I don't have the opportunity yet to test on mobile network.

In my Podfile.lock : FirebaseDynamicLinks (~> 4.0.8). How to upgrade to v.4.1.0 ?

same here
facing this issue in RN
  - Firebase/DynamicLinks (6.25.0):
    - Firebase/CoreOnly
    - FirebaseDynamicLinks (~> 4.0.8)

Seriously!! no one from the people who made this SHIT is not gonna answer this issue?????

Seriously!! no one from the people who made this SHIT is not gonna answer this issue?????

I think they don't...

@alif1993 that is abusive. 1) these are humans on the other side of your keyboard 2) this SDK is marvellous but also marvellously complicated, how's your migration to IPv6 going for instance? To imply low quality based on this issue, and to infer some kind of lack of care is so insulting.

Hello, Folks! I have same issue here. Seems that this merge request didn't help.

The pull request (in review comments) actually mentioned it was a separate issue, but the developer who proposed it was aware it wouldn't be fixed
I quote @eldhosembabu from https://github.com/firebase/firebase-ios-sdk/pull/5665#issuecomment-636132977

Its not actually, for fixing #5032 we need to optimize the calls i guess for that. Currently we are doing ipv4 and ipv6 calls parallel and consolidate the results which needs to be serialized. In this current fix, I'm just making sure we don't repeat checkforpending calls when there is an error in the previous one.

and one could assume from "this current fix" that's this one is probably on an issue board somewhere pending attention

@mikehardy I agree that I went too far, but when our team has a deadline and the product was supposed to be delivered two weeks ago and coz of this bug which is so weird since everything is okay on Android and it happens on IOS. You are right, it's my fault, I should have checked all our product feature in pre-production. I APOLOGIZE to the team, I left that comment after four consecutive days of searching:) which was completely a wrong move:(

@a-h Thanks for the terrific bug report.
And everyone: I'm sorry that we're having trouble making progress on this issue. We'll continue to try to work with the team to get some focus here. In the meantime, we'd be happy to evaluate any PRs.

Hehey! Any progress on this? For now I just replaced all the ipv6 urls with ipv4 ones to go forward, but yeah it's quite hacky.

Hey! Same problem at my end. I'm inside IPV4 network at my local ISP and firebase tries to connect to ipv6 domain which cannot be resolved from ipv4 network.

Same issue here...
- Firebase/DynamicLinks (6.27.1):     - Firebase/CoreOnly     - FirebaseDynamicLinks (~> 4.1.0)

It's been months there's still nothing on this?

@IsaiahJTurner you have to dig a bit but there is activity, it is just a mix of different issues and internal conversations. Specifically examine discussion here: https://github.com/firebase/firebase-ios-sdk/pull/5966 - a rejected PR that might have fixed this but had some unacceptable backend consequence - from 9 days ago, so I'd infer it's an active topic, just without messaging here for status

Does this issue still active? Maybe someone has something like workaround/hack or we still need to find a solution?

Any solution to this? It's been at least 6 months without a fix.

Any solution to this? would gladly get one

Hi everyone,
Sorry for the delays and lack of communication here, but this issue is now resolved. We were originally worried that by switching to the base API that doesn't specify ipv4 vs ipv6 would lose accuracy based on attribution, but have determined that it's a small corner case and an acceptable tradeoff from a performance and confusion standpoint (printing error logs when it may be working as intended).
Thanks for your patience as we worked through this, this change will be available in the next release expected sometime next week.


Pass the netgo tag when building static binaries

When building static binaries with CGO, cgo unix network segfaults
When building dynamically linked binaries with CGO, everything is fine
When building static binaries with CGO but with netgo enabled, dns lookups seem fine

So add the netgo tag when building static binaries.
https://github.com/prometheus/alertmanager/issues/304
https://github.com/golang/go/issues/7857

also for noting: this closes https://github.com/docker/notary/issues/1037 and https://github.com/docker/notary/issues/1022.
I'm working on a v0.4.3 with this fix as well.

Have filed https://github.com/golang/go/issues/17935 to see if it's an issue in go

Hm... we might want to just revisit packaging debs and rpms again...

Superseded by #1072


Added ConnectionAdapterProcessExecutor as intermediate between connection adapters and Chrono

Also made required changes to connection adapter implementation:

Added changeWorkingDirectory and getWorkingDirectory in ConnectionAdapterInterface.
Implemented new methods in LocalConnectionAdapter and SSHConnectionAdapter.

Closes #37.

Also related to #43.


Marketplace Account UI Improvement

Summary
Improve the Marketplace account UI to load each view as complete page.
Details
The the pages in account UI are organized to load as single pages with all content including aside menu, header, footer etc.
Espescially with slow network connections and with highly loaded clients this results in annoying slow screen build up.
When clicking an menu entry in the aside menu the respective view is loaded including all templates. This request is to refactor the UI in a way that the views are decoupled from the surrounding common page framework.

The views shall be implementented as composites that are embedded into one single page.
By clicking a menu entry in the aside menu, the content shall be loaded with AJAX request and rerendered into the embedding page.
Concept hint: In order to reduce this impact of this refactoring, the different existing (sub-)pages shall be preserved. This can be achived by loding the pages with a4j:include and making the menu items to trigger AJAX Requests that only rerender the embedding container of the a4j:include.
A state of the art replacement with bootsfaces JSF instead of a4j:include is very wellcome but not a must.

Acceptance criteria

[ ] Concept or technical description in this GitLab issue is done an approved.
[ ] Sub page contents is loading fast with AJAX.
[ ] Only sub page content is rerenders. Header, footer and aside menues stay fix.
[ ] Contained tab views are not jumping on load.

Additional context
Add any other context or screenshots about the feature request here.

@GoebelL Unfortunately, the attempt to create a page in which only some of its elements would change failed. A lot of errors came out with submiting forms and refreshing info messages.
In task was done:

Minify css and javascripts
Unnecessary files transferred from head to body section
Tomcat is configured to compress files
Tomcat is configured to allow saving styles, js, images in the cache

What else can be done to speed up page loading:

[ ] Implement faster DNS resolve
[ ] Changing the framework [Richfaces weighs a lot compared to other technologies]
[ ] Removal of bootstrap and put styles inside xhtml files

Previous:

At present

Hi @KabacinskiPiotr
I will catch up on the requested Account UI refacoring when I am back from holiday. For the time beeing, performance tuning is ok as long as it's risk less and meaningful. For evidence please add some figures, e.g. page xxx before your change x ms, improved to y ms.


Feature request - Collection templates

I have some different types of collections: Actors, Directors, Genres and the normal movie collections. My issue is that there is only one folder where all the collections assets go. Instead of having these different types of collections mixed together in the assets  folder, I would like them to have their own sub folders, like assets/Actors and assets/Directors . That would make organizing the files a lot easier.
For each category I have a certain prefix for the sort title: ++D_ for directors, ++A_ for actors and so on. These common settings could be put into a collection template, which would make it easier to handle these different types of collections. Example:
templates:
  Directors:
    sort_title_prefix: ++D_
    asset_directory: assets/Directors

  Actors:
    sort_title_prefix: ++A_
    asset_directory: assets/Actors

collections:
  David Cronenberg:
    template: Directors
    director: tmdb
    tmdb_person: 224

  Arnold Schwarzenegger:
    template: Actors
    actor: tmdb
    tmdb_person: 1100

So with this, the sort titles for the collections would be ++D_David Cronenberg and ++A_Arnold Schwarzenegger and they would have the asset folders assets/Directors/David Cronenberg and assets/Actors/Arnold Schwarzenegger
I'm sure there are a lot more uses with templates, this is just what I have thought of with my own collections.

ok done! can you test in develop look at the new template file to figure out how they work. I'll be writing up something soon

Trying to run it but I get
| Connecting to Movies Library...                                                                    |
| local variable 'Radarr' referenced before assignment                                     |

ok now it should work

Template Details Wiki Page is up


Is there any method or workaround to use system tray in WinUI3 desktop App?

I am developing a desktop app using winui3, and I am going to put the app into system tray after user closing it by X button. Only user right click the icon in system tray and choose exit then the app could be finally closed.
I am not sure if currently there is any API related it. If no, is there any workaround to get it? Thank you for your kindly help.

similar to https://github.com/microsoft/microsoft-ui-xaml/issues/4792  @MikeHillberg and @Austin-Lamb and @marb2000

It's not a built-in feature of WinUI3 and I don't have a sample that demonstrates how to do it. @dotMorten do you have any?
These are good docs to reuse from Win32 APIs and encapsulate it into a helper like the @dotMorten sample or mine.

@marb2000 @gscheartA Yes my repo does show how to minimize to tray here: https://github.com/dotMorten/WinUIEx/blob/3aaa34b58c488dba054e80311bfc3aa1cc772950/src/WinUIExSample/MainWindow.xaml.cs#L70-L80

Closing the issue given @dotMorten provides a workaround.


Issues with installlation

Hello,
Nice work so far. Whilst installing the lib, I encountered few issues.
First the command used to install on your readme.md and readme.rst was
pip install crypto_compare

Well, that doesn't work. So looking at the lib on https://pypi.python.org/pypi/crypto-compare/0.0.1 , it's clearly different.
The right command then should be:
pip install crypto-compare

Running that command, I received this error

Collecting crypto-compare
  Downloading crypto_compare-0.0.1-py2-none-any.whl
Collecting urllib2 (from crypto-compare)
  Could not find a version that satisfies the requirement urllib2 (from crypto-compare) (from versions: )
No matching distribution found for urllib2 (from crypto-compare)

I'm running it on a MacOS Sierra
Any help?

@mohataher I have actually tried it once but it's not working. I would suggest instead you can download the source and use it, I am not able to work on this one right now :(

@mohataher @ParthBarot-BoTreeConsulting Download the source code and remove urllib2 in install_requires solved my problem.
But be aware that this may not be the best solution because I didn't review all the source code.


Performance degradation when switching from v3.6.3 to v4.25.1

Description
Using the same changelogs and database/schema liquibase after upgrade to version 4.25.1 is a few times slower, e.g. 33 minutes instead of 8 minutes.

First I waited for checksums upgrade, then I executed liquibase once again without any new migrations in version 4.25.1 to compare performance between liquibase versions.
In my setup multiple Liquibase objects are executed in parallel migrating different DB schemas and DB hosts.
ThreadLocalScopeManager is registered before starting migration.

I managed to profile liquibase 4.25.1 and I have two observations:

There are a lot of calls to AbstractMapConfigurationValueProvider#getProvidedValue. Threads are waiting on a lock. In this critical section, among other things, EnvironmentValueProvider is called and it looks like computing hash of its map may be time consuming, which may be relevant in the context of multiple other threads waiting.
There are child scopes spawned in high numbers. Scope initialization takes time, mainly scope id generation.

Could you take a look at this problem?
Expected/Desired Behavior
No performance degradation.
Liquibase Version
3.6.3 => 4.25.1
Database Vendor & Version
MySQL
Liquibase Integration
Called directly, programmatically.
Liquibase Extensions

OS and/or Infrastructure Type/Provider
Linux

@tati-qalified, could you send them information on running the profiler built into Liquibase?

@tati-qalified, could you send them information on running the profiler built into Liquibase?

--monitor-performance

@Dasiu as Sebastian said, the --monitor-performance parameter creates a file containing all information about the execution of Liquibase code.
If you could provide us with that file, as well as some information about the approximate number of Liquibase instances and changesets being executed, we'll look into this.
Thank you,
Tatiana

Unfortunatley JFR file can contain potentially sensitive information, I would like to be judicious and abstain from providing it.
More context:

~1300 changesets already persisted in DATABASECHANGELOG table.
Hundreds of similar DB schemas are migrated on the same DB host on multiple DB hosts.
No new changesets executed -- just "dry run".
4 Liquibase instances migrating simultaneously. Total number of Liquibase instances created ~1100.

More observations:

Liquibase 4.25.1 executes more SQL queries than 3.6.3. Screenshots compare queries executed for just one DB schema (GRANT and CREATE SCHEMA queries are not issued by Liquibase directly).
Own execution time reveals that apart from DB communication, significant time is devoted to scope id generation: "Arrays.copyOfRange", "SecureRandom", "RandomIO" are all back tracing to scope id generation.

@Dasiu I understand that uploading the JFR publically is not ideal, but without it we won't be able to find the source of the issue.
To ensure more privacy, we can provide a channel between you and our team, and limit the amount of people that have access to the file.
Let me know if this works for you.

https://github.com/Dasiu/liquibase-performance-degradation/tree/main Here is code which reproduces the issue. Its profile reveals the issues/observations I mentioned.
To summarize what I found earlier.

Looks like more queries are executed per schema in newer version of Liquibase. The question is why more queries are executed and whether something could be done to alleviate performance drop.
Scope id generation in its current form is CPU consuming.

@Dasiu  Seems you are right! In my tests changing how we generate the scope id improved the performance in ~30%!
Hi - could you give my fix a try? You can download the zip file here or obtain it using GPM (gradle
implementation("org.liquibase:liquibase-core:5a353215241fed56653538be52673262abb3bca1-SNAPSHOT")

Great, thank you! I'll try to verify new version in coming days.

@Dasiu If there is something else that we can improve please let us know!

Ok, after a couple of weeks I can safely say that I managed to upgrade to 4.28 and it has been working fine. No concurrency issues observed. Performance is now almost on par with 3.6.3. Thank you very much.
Side remarks:

I additionally switched off "show summary" UpdateSummaryEnum.OFF. I had an impression based on logs that it led to excessive amount of new Scopes being created which still created some overhead.
The biggest optimization potential I see now (for my parallel setup) is to cache safely parsed XML changelogs. Currently parsed changelogs are not thread-safe judging by my code inspection.

Thanks for the feedback @Dasiu , that's great to hear!
Do you mind opening an issue (or even better 2 issues) for the other 2 optimization points?


test(wallet): WalletInfoCard and EditableName

PR Type

[x] Bug fix
[ ] Feature
[ ] Code style update (whitespace, formatting, missing semicolons, etc.)
[ ] Refactor (i.e., no functional changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation changes
[x] Other (please describe below):

Description
Add tests for WalletInfoCard and EditableName.
Also fix a couple of minor bugs found by tests:

Autofocus editable name text field
Remove unused children prop from WalletInfoCard

Depends on https://github.com/unchained-capital/caravan/pull/80

Does this PR introduce a breaking change?

[ ] Yes
[x] No

Does this PR fixes open issues?

[ ] Yes
[x] No

This will supersede #80 now right?

This will supersede #80 now right?

No I planned on rebasing it after #80 was merged.

@bucko13 rebased and ready for review/merge


additionalHeaders not working as expected

The doc of rememberWebViewState states:

param additionalHttpHeaders Optional, additional HTTP headers that are passed to [AccompanistWebView.loadUrl]. Note that these headers are used for all subsequent requests of the WebView.

It seems the bolded part isn't actually true. The headers are passed along via the initial WebContent.Url object, but discarded when navigating inside the webview or using navigator.loadUrl().
I tried it by attaching an Authorization header to a request; the initial request worked, subsequent requests were rejected.
I can see a few options here:

Change the docs and use the headers only for the initial request. But that would leave us with no option to actually provide universal headers.
Keep the headers and apply them to every subsequent navigator.loadUrl() call. That would apply the headers to all requests created via the navigator, but not to requests created by the webview itself.
Apply these headers to all requests. That's probably a bit more work, on Android it would involve overrideUrlLoading, on Cef I don't know.

My personal, related issue: I want to attach an Authorization header to all requests (if possible, only requests to a certain domain). I think at the moment this is impossible.

Tested using 1.6.0 and Android, but I think the current master works the same (except that additionalHeaders don't work for desktop at all in 1.6.0).
Thanks for stepping up to build and maintain an KMP webview! :muscle:

@JuBan1 Thank you for your suggestions! After checking, I also confirmed that additionalHttpHeaders only works for the initial request. I apologize for any confusion caused by the comment, which was copied from the accompanist Webview. I will make sure to address this issue in the future.
However, I am currently focused on resolving the JsBridge and other issues, so it may take some time before I can start working on this particular problem. If you are interested, you are welcome to submit a PR for the Android and iOS implementation. For the Desktop side, I think @DatL4g  would be happy to provide assistance.

I think a good first step would be to figure out what the goal should be.
(1) and (2) are easy to implement, but are lacking.
For my purposes I would need at least (3), but after thinking about it a little it might be a better idea to basically implement shouldOverrideUrlLoading. That allows users to intelligently add headers to all requests but also do things like intercept certain URLs or prevent certain domains.
I'm thinking users can add an RequestInterceptor to WebViewNavigator; the interceptor is a callback which is triggered for every (GET) request and enables you do (a) allow it, (b) modify it or (c) block it. I think this is possible on Android via shouldOverrideUrlLoading, on Cef via onBeforeBrowse, and I guess it kinda works on iOS too.

I agree with you. I think RequestInterceptor is a great idea and I would go with this plan.

@JuBan1 Hi, I have implemented the basic feature for RequestInterceptor on this branch. It should works well on Android and iOS, but have some problem on Desktop. Can you review it and give some suggestions? Thank you so much!

@DatL4g Could you also have a look at the Desktop implementation on this  branch? I am trying to intercept the request in CefRequestHandler.onBeforeBrowse. It did intercept the request but will also block future requests if it return false and reload another Url like I did in the example.

Adding a CefRequestHandler means overriding the default CefRequestHandler.
We had that problem before (user-agent) and we (or I) have to reimplement the default behavior in and make setting a CefRequestHandler extend the default reimplementation instead of overriding.
Additionally this doesn't only break the ``onBeforeBrowse` implementation, it breaks all request handling methods.
I don't have that much time currently as I'm in a release phase right now, best would be to add it directly to KCEF so we don't have to deal with it in this library, maybe you could add a PR if you have time or we have to wait until I have time again (could be anytime in December).

Hey, thanks for tackling this @KevinnZou! I have a half-finished implementation as well, and I got stuck on the desktop implementation too. So that strengthens @DatL4g's point of moving it into KCEF if possible. Unfortunately I think I'm a little out of my depth there :)
I pushed my implementation into a PR for inspiration.
I have had more luck with overriding onBeforeResourceLoad and discovered that modifying CefRequests works fine. But cancelling a request breaks the implementation. So it's probably a dead end too.
Besides, I've discovered a few more problems:

My initial intent was to be able to set the Authorization-header for particular websites. This works on Android, but CEF actually filters out the Authorization header (and probably a few others) if you try to set it via setHeaderMap(). So this isn't a multiplatform solution, and authorization management would have to be a separate feature...
Different platforms have very different scenarios in which they trigger the request interception. On Android, shouldOverrideUrlLoading is only called for a few URLs but on CEF onBeforeResourceLoad is called for literally everything: images, css-files, etc. On iOS decidePolicyForNavigationAction probably has its own rules. So it's tough to find common grounds so it works well on every platform without huge differences in behavior. (For example, in my implementation I tried to exclude non-mainframe requests for CEF)

The only thing I am happy about is the RequestInterceptor and how it is used.
On a side note, (K)CEF provides a CEFRequestHandlerAdapter that has default-implementations for all methods, meaning you don't have to do that yourself

@JuBan1 Thank you for your great work! I really appreciate your idea for RequestResult.Modify, it eliminates the need for users to manually reload the URL. I will incorporate your ideas into my implementation, as they are extremely helpful to me.
Regarding the issues you mentioned, I will collaborate with @DatL4g to see whether the Authorization header problem can be solved. Additionally, I also observed that shouldOverrideUrlLoading does not intercept all URLs like onBeforeBrowse does. I will also work on finding a solution for this issue.
Generally, I think we should be able to support Android and iOS first. As for the Desktop side, I will attempt to address the CefRequestHandler issue following @DatL4g's instructions. If I am unable to do that, we will have to wait for @DatL4g to resolve it next month.
Thanks for your contribution again! I will continue to update the progress here, and you are welcome to review it.

@DatL4g I briefly looked at the implementation of addRequestHandler in KCEF, but I have some confusion.

Adding a CefRequestHandler means overriding the default CefRequestHandler.
We had that problem before (user-agent) and we (or I) have to reimplement the default behavior in and make setting a CefRequestHandler extend the default reimplementation instead of overriding.

Can you provide more detailed instructions for this paragraph? I'm not sure where your default CefRequestHandler is located. Because from the implementation of addRequestHandler, it seems that requestHandler_ is initially null, and this method only saves the handler we provide into requestHandler_ and uses it in onBeforeBrowse. I haven't seen the implementation of the default CefRequestHandler, so I am confused about how can we extend the default reimplementation instead of overriding?

@JuBan1 Regarding shouldOverrideUrlLoading not called issue, I searched online but couldn't find a solution. One solution I came up with is to replace shouldOverrideUrlLoading with intercepting shouldInterceptRequest. I tested shouldInterceptRequest and found that it gets called for every request, including internal resource loads on the page. One possible solution is to check request?.isForMainFrame to decide whether to intercept, which should cover most routing interception scenarios. The main commit is here. Do you have any suggestions?

@KevinnZou That's right, you have to dig a bit deeper, the CefBrowser_N.java is doing the magic.
If we provide a CefRequestContext which is required for CefRequestHandler the request_context_ variable is set.
The problem here is that this overrides the default behavior/methods since the request_context_ is set to CefRequestContext.getGlobalContext if the request_context_ is null, otherwise this default context is not applied.
So we basically need a extendable RequestContext class which defaults to CefRequestContext.getGlobalContext

@DatL4g Thanks for your explanation! I think I understood why we need an extendable RequestContext. However, I'm a bit concerned that I might make mistakes since I'm not very familiar with it. I want to ensure that the code modifications are done accurately. Thus, I would prefer to leave it to you to handle the changes when you have some free time. In the meantime, I will continue working on the JSBridge implementation. Please let me know if there's anything I can do to assist you. Thank you for your contribution again!

The main commit is here. Do you have any suggestions?

I'm not sure if this mechanism makes sense for shouldInterceptRequest. As I understand it, shouldInterceptRequest is called not only for the "main" url but also for content like images, css, etc.
The request interceptor gives the impression that I can Allow/Reject/Redirect this request, but Reject/Redirect changes the webview's main url, not the request that triggered the interceptor.
It seems to me that this is ultimately confusing and not very useful.
I would say it's better to use shouldOverrideUrlLoading and either

accept (and document) that the intercepted requests depend on the platform, or
limit everything to the lowest common denominator by only intercepting main urls and maybe iframe urls (or whatever actually triggers  shouldOverrideUrlLoading)

@JuBan1 Thanks for your suggestions! I understand your concern, shouldOverrideUrlLoading should be the first choice if it can intercept all the URLs. However, it is still unclear why it can only intercept partial requests. Thus, we have to make a choice:

use shouldOverrideUrlLoading but cannot intercept all requests.
use shouldInterceptRequest but only intercept the main frame and use stopLoading to perform the request rejection.

In my opinion, I would prefer the second option.  For the first choice, developers are unsure which requests can be intercepted and which cannot, making it unsuitable for production use. The second option is not ideal, but it can at least intercept all requests. What do you think?

Adding a CefRequestHandler means overriding the default CefRequestHandler.
We had that problem before (user-agent) and we (or I) have to reimplement the default behavior in and make setting a CefRequestHandler extend the default reimplementation instead of overriding.
Additionally this doesn't only break the ``onBeforeBrowse` implementation, it breaks all request handling methods.
I don't have that much time currently as I'm in a release phase right now, best would be to add it directly to KCEF so we don't have to deal with it in this library, maybe you could add a PR if you have time or we have to wait until I have time again (could be anytime in December).

@DatL4g  Are there any updates?

@JuBan1 I reported a bug regarding the shouldOverrideUrlLoading not intercepting all requests, and received a reply from the bug tracker. They explained that it is intentional for shouldOverrideUrlLoading to not intercept the initial load request.
Thus, regarding the previous discussion, we may have to make a choice:

use shouldOverrideUrlLoading and make iOS and desktop also not intercept the first load request.
use shouldInterceptRequest to request all the requests.

Do you have any suggestions?

Hi all, I have implemented a basic interception in the latest version 1.9.8. Please follow these instructions to conduct a test and let me know if you encounter any issues.


Support semver 'query' in spec.forProvider.chart.version

What problem are you facing?
Given the following release:
apiVersion: helm.crossplane.io/v1beta1
kind: Release
metadata:
  name: waas-helm-crossplane-release-example-stage-east
spec:
  deletionPolicy: Delete
  forProvider:
    chart:
      name: test
      repository: https://my.artifactory.foo/helmcharts
      version: ~0.1.1

It should respect the ~0.1.1 query and resolve the proper version to use. Right now I get an error like 'create failed: failed to install release: failed to load chart: stat /tmp/charts/test-~0.1.1.tgz: no such file or directory'
How could Crossplane help solve your problem?
It should conform with how the helm cli client resolves this issue.
I'll dig more into how the helm client code and see how they resolve this and perhaps prepare a pullrequest.

Talking to a few people and spelunking the code found me this:
https://github.com/helm/helm/blob/main/pkg/downloader/chart_downloader.go#L247 which calls:
https://github.com/helm/helm/blob/db2485b20c8f53d802f4fc98c2f95e1fc0460d15/pkg/repo/index.go#L179
which resolves the semver constraint to an available version.


suitcase not available on CHX analysis environment in jupyterhub

Thanks @ordirules for bringing this to my attention.  @danielballan @tacaswell I would imagine we should include suitcase with the analysis metapackage?

:+1:

Fixed.


HTML

#HTML
User Story
As bootcamp student
I want the prework notes to be structured on a webpage
so that i can easily find and read information
Acceptance Criteria
GIVEN a Prework Study Guide website
WHEN I visit the website in my browser
THEN I see four boxes titles HTML, CSS, Git, and JavaScript with associated notes listed

I updated the HTML code for your Prework Study Guide and pushed up the changes from your local repository to the remote repository on GitHub.


Meeting 2021-02-25

Where & When

#rust-gpu on the Embark Discord.
Time: 17:00 CET
Previous Meeting

Agenda

Project updates
Review & discuss any open MCPs

Feel free to propose additional items in this thread.

Project Updates
@XAMPPRocky: Mostly working on investigating bugs in spirv-cross's MSL compiler that we found when trying to integrate rust-gpu with Ark shaders.
@eddyb: Having attribute, instead of wrapper types.
@eddyb: Wants to tackle attributes next.
@eddyb: Essentially model rustc's attribute parsing pass.
@XAMPPRocky: there's also #443 which changes it to be value.
@eddyb: Yeah, that is a starting point, but we need to figure out what we want first.
@Jasper-Bekkers: More progress with bindless
@Jasper-Bekkers: One limitation is that you can't have template parameters to external functions.
@eddyb: I could make a PR to upstream some of that work in rustc.
@Jasper-Bekkers: I worked around it for now, with a custom attribute.
@Jasper-Bekkers: In the builder-methods struct, is it okay to add global ops?
@eddyb: Depends, probably yeah.
@Jasper-Bekkers: Right now I'm adding global types and I'm running into some issues.

@Jasper-Bekkers: Planning on following DXE is doing.
@VZOut: MoltenVK 1.1.2, but not much other.


View MP Systems Data

Definition of Ready

[x] Does this meet the INVEST criteria?
[x] Does this have a clearly defined user?
[x] Does this have an estimate?
[x] Does this have acceptance criteria?
[x] Does this have sufficient context (wireframes, business rules, etc.)?
[x] Is this traceable to a feature, epic, or PI?

User Story
As a user I want to view production monitoring plan data so that I can look at real time data.
Acceptance Criteria
Given a user has selected a facility
When the Monitoring Plan Systems data is selected
Then it is displayed in accordance with the business glossary (Monitoring Systems) and the wireframes
Dependency: Create MP data contract for system data
Assumption: no keyword search and no pagination
Testing
Test Case: TC116
Tester: @mosesdeeCVP
Definition of Done

[x] Have all Acceptance Criteria been agreed to and validated by the scrum team?
[x] Has the code been unit tested and peer reviewed?
[x] Have the functional tests been executed?
[x] Have all defects been reviewed and dispositioned, resolved, or deferred?
[x] Has the user story been accepted by the Product Owner?
[x] Is the user story ready to be deployed to the staging/integration environment?

Prerequisite:
Need data contracts for MP data completed, in progress in sprint 2.1
Wireframes, expected completion in 2.1

There is a bug that is preventing the accordion table from appearing open by default. This is not specifically mentioned in the AC and the test are passing. The Product owner has accepted the story and thus it will be closed. A separate bug ticket will be created for the default open.

Bug ticket 994 was created for this story


Creating Workspace after setup fail

Bug description
Dears,
any one face this error after install latest gitpod image
Request startWorkspace failed with message: 13 INTERNAL: cannot resolve workspace image: failed to do request: Head "https://registry.gitpod.yourcompany.com/v2/workspace-images/manifests/cf074bfb1bc0c95cbc61a47f9cba11e08c59422d224069be68f80160a3cf9e8e": dial tcp: lookup registry.gitpod.yourcompany.com on <IP_ADDRESS>:53: no such host
Steps to reproduce
create a new workspace
Expected behavior
No response
Example repository
No response
Anything else?
No response

thanks, i figure out the problem
I was missing 1 record at the ACM. the final result should be as the following screenshot 5 records

and this is the final result at DNS Records (Cloudflare-GoDaddy-etc...)


Can't add the "GuiTransition" node, no clear error message

Good day! I recently found the addon on the AssetLib and it's doing basically everything I've been doing manually, which is awesome.
I added the addon to my project (and enabled in the "Plugins" tab), and I'm able to run the examples as expected. However, when trying to add the GuiTransition node to a scene, I'm getting a weird error that's not giving me much:

For context, I'm adding the GuiTransition node as a child to a Control parent.
As a hack, I can copy the GuiTransition node from the example scene and paste it in my scene, and that seems to work, but I don't think that's the way it should be haha.
Am I missing something somewhere?

Hello, @mathieubruguier! Thanks for the report. I tried to install it from the AssetLib on a brand new project, and I could reproduce your issue. However, after closing and reopening the project, it works as expected.
Probably I should add to the project's README "reload your project after enabling the plugin", because it must be some issue with the class_name of the node.
Could you confirm if it works as expected after the step noted above?

Thank you for the quick response! I figured that could have been the case so I had already restarted Godot once or twice, no dice.
However, it looks like it actually works this morning after restarting my computer! Maybe I didn't restart Godot hard enough 😝
Sorry for wasting your time with it! Everything's well that ends well. 😄


Encrypted uuid's for android sdk?

Wondering if this is coming to android sdk as well:
https://github.com/Estimote/iOS-SDK/issues/9#issuecomment-28573302
Also, any rough time-frame or setbacks? One month/year/release out? Thanks!

+1 for this

+1

+1 for this

+1
Any updates on this?

+1

Secure UUID has been released in SDK version 0.9.
More info: https://github.com/Estimote/Android-SDK/releases/tag/0.9


Add command line option for program entry point

To run a function other than main:
$ gr taste.gr.md --entry-point firstChar
Checking taste.gr.md...
OK, evaluating...
'A'

This pull request has the bad commit we wanted to revert in it (re ordering).

Good spot. Should be fixed now.


Thunderbird buggy behavior

Hey there!
I've noticed a buggy behavior while using Thunderbird. When I switch theme with yin-yang, the only thing that changes is the upper toolbar's font color (see screenshot 1). Only when I alt-tab to other app, and then go back, the upper toolbar is rendered correctly (see screenshot 2). The rest of the app is sadly not switched at all, unless I close & open the window. Is there something that can be done about that?
I know that there is a Firefox extension to be used with yin-yang, but it seems to be working about half a times. If I'm not mistaken, Firefox & Thunderbird uses both GTK, on which the switching works just fine. Maybe it is connected to this warning, which I get when I run yin-yang -t
QDBusConnection: error: could not send signal to service "" path "//home/jukka/.config/kdedefaults/kdeglobals" interface "org.kde.kconfig.notify" member "ConfigChanged": Invalid object path: //home/jukka/.config/kdedefaults/kdeglobals
QDBusConnection: error: could not send signal to service "" path "//home/jukka/.config/kdedefaults/kdeglobals" interface "org.kde.kconfig.notify" member "ConfigChanged": Invalid object path: //home/jukka/.config/kdedefaults/kdeglobals

Using:

Arch Linux x86_64
KDE Plasma 5.25.5
master branch of yin-yang
Thunderbird 102.3.1

Thanks for helping me out!

For me, it works. I was using the flatpak version of thunderbird, which is 102.3.2. Can you try that?
Maybe it's related to this bug. Since then, Yin & Yang just runs the command twice. Can you please check if you get the same behavior when running the following manually:
lookandfeeltool -a org.kde.breeze.desktop
lookandfeeltool -a org.kde.breezedark.desktop
lookandfeeltool -a org.kde.breezedark.desktop

Sadly, no. I've tried multiple times, but apps refuse to change theme.
But, I've tried the flatpak version of the app and it switches instantly, after single command.
What is weird though, is that when I was installing flatpak version, I uninstalled the native one. After reinstallation, the issue seems gone - Firefox & Thunderbird works flawlessly...


Standalone version

I just discover this Github Actions and it's interesting.
I'm interesting having a standalone version that I can run outside of a Github Actions.
Is it something you think to do in future ?

Hi @jdecool yes. I'm working on separating the core of this action to be used elsewhere, and a platform to publish these stats in Slack. Sadly, I'm not moving as fast as I'd like but I hope it will be ready in the next weeks.

Sadly, I'm not moving as fast as I'd like but I hope it will be ready in the next weeks.

No problem. It's cool if you work on it.
I'm happy to see that when it will be ready ;)

@manuelmhtr  no pressure but is der any update on when we can see this feature?

Hello -joining late the party here, wondering whether anyone came up with the option to run this already in the stand-alone way?


Add LoRA training supports for diffusers version from diffuser==0.10.x to latest 0.16.1

majorly 2 things have been changed since v0.10
1, xformers op injection has been handled by diffusers nicely, after v0.10, and if using the latest 0.16.x the memory consumption is no longer any big difference in regardless turn on/off the xformers option
2, the LoRA create_network function make the network layer duplicated in the latest lora version.
Please read the details in the code change.
Thank you for a great job

Thanks for the feedback, i have tested it with below environment
xformers==0.0.19
transformers==4.28.1
diffusers==0.16.1
torch==2.0.0
torchaudio==2.0.0
torchvision==0.15.0
(the diffuser will work since 0.12 - 0.16) i have tested them all
the important thing is this patch works with torch 2.0,  in torch 2.0 they implemented its own improvement of the attention module, so natively without xformers and any other plugin, it runs 4 times faster than torch 1.x.
i would also recommend you try to switch to the latest torch v2, after this patch, the code works totally fine with torch 2.0
my LoRA training parameters are below:
accelerate launch --num_cpu_threads_per_process=2 "train_network.py" --enable_bucket --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" --train_data_dir="/proj/suchka/image-generation/lora/jixian/img" --resolution=512,512 --output_dir="/proj/suchka/image-generation/lora/jixian/model" --logging_dir="/proj/suchka/image-generation/lora/jixian/log" --network_alpha="128" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=5e-7 --unet_lr=5e-6 --network_dim=128  --output_name="jixianwang_v3" --lr_scheduler_num_cycles="10" --learning_rate="5e-6" --lr_scheduler="constant" --train_batch_size="24" --max_train_steps="2000" --save_every_n_epochs="1" --mixed_precision="bf16" --save_precision="bf16" --caption_extension=".txt" --cache_latents --optimizer_type="AdamW8bit" --max_data_loader_n_workers="0" --bucket_reso_steps=64 --xformers --bucket_no_upscale --noise_offset=0.1 --seed=1234

Still getting the same error on your branch, btw

This patch breaks the --mem_eff_attn option because of the CrossAttention module refactor. There are also reports that, at least on ROCM, torch 2.0's SDP attention actually increases vram usage rather than reducing it, so it's not a drop-in replacement.


feat(axe-core-4.4.1): update to use axe-core 4.4.1

Details

Updates axe-sarif-converter to use new axe-core version 4.4.1
Motivation

Part of feature work. Updating all products to axe-core 4.4.1
Context

Pull request checklist

[x] PR title respects Conventional Commits (starts with fix:, feat:, etc, and is suitable for user-facing release notes)
[x] PR contains no breaking changes, OR description of both PR and final merge commit starts with BREAKING CHANGE:
[x] (if applicable) Addresses WI: 1940675
[x] Added relevant unit tests for your changes
[x] Ran yarn precheckin
[x] Verified code coverage for the changes made

:tada: This PR is included in version 2.9.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Consolidate blog data

There are two configs where blog data is stored: the cardData.ts which is the summary card shown on the home page and the Markdown file which has headers for metadata.
Adding a new blog requires updating two separate places.

Having centralized configuration will make it easier for us to split articles by Work Experience and Blogs.  Having one config would also enable easier tracking of types of articles as opposed to writing metadata in different places.  Writing metadata in the markdown files as we do now will cause the inefficiency in creating configs since the generation of static files will cause the reading of the markdown twice.
Escalating priority given consequences in further features.


connection protocol not supported

I copied and pasted the example on first page and the app throw this exception
if (err) throw err;
^
Error: CONNECTION_PROTOCOL_NOT_SUPPORTED
at Object.defineProperty.value (/home/icarvalho/node_modules/orm/lib/ErrorCodes.js:12:13)
at Object.exports.connect (/home/icarvalho/node_modules/orm/lib/ORM.js:129:32)
at Object. (/home/icarvalho/Área de Trabalho/nodeorm/app.js:3:5)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Function.Module.runMain (module.js:497:10)
at startup (node.js:119:16)
at node.js:901:3

npm update
npm install mysql

Obvious but small tip that might folks who land here scratching their heads about what this is suddenly happening, if you get this after upgrading between node versions (e.g. 6/7/8) and run into this error when it previously working for you, deleting your node_modules folder and running npm install may resolve it for you.

@iaincollins this issue is closed, the date is from 2013.

@IgorDePaula I know. It's normal on GitHub to update popular problems with solutions, including closed issues that have new triggers.


error in installation

I was installing home-assistant by command pip3 install homeassistant, everything was successfully installed except one :
coala-bears 0.11.1 has requirement pyyaml~=3.12, but you'll have pyyaml 5.1 which is incompatible.
then I run the command hass --open-ui, the result was this:

This is the bug/issue tracker for Home Assistant. Please use the forum for questions about installation, setup, configuration and usage.
If it's a bug, please re-open the issue with the issue template filled.


Pretrained models now missing

Hi @chenlin9
Great work, but seems that all pretrained checkpoints are down, since all rely on AWS, i.e.:
https://image-editing-test-12345.s3-us-west-2.amazonaws.com/checkpoints/bedroom.ckpt
Several works depend on these, would you mind uploading them again?
Personally, I'm looking for LSUN Bedrooms, but I see a lot of people requesting CelebA-HQ too.
Thanks.

Hello, we use the same checkpoint as DDIM https://github.com/ermongroup/ddim

Right, I was wondering that. Thanks for the quick answer!
For future reference, here they are
URL_MAP = {
    "cifar10": "https://heibox.uni-heidelberg.de/f/869980b53bf5416c8a28/?dl=1",
    "ema_cifar10": "https://heibox.uni-heidelberg.de/f/2e4f01e2d9ee49bab1d5/?dl=1",
    "lsun_bedroom": "https://heibox.uni-heidelberg.de/f/f179d4f21ebc4d43bbfe/?dl=1",
    "ema_lsun_bedroom": "https://heibox.uni-heidelberg.de/f/b95206528f384185889b/?dl=1",
    "lsun_cat": "https://heibox.uni-heidelberg.de/f/fac870bd988348eab88e/?dl=1",
    "ema_lsun_cat": "https://heibox.uni-heidelberg.de/f/0701aac3aa69457bbe34/?dl=1",
    "lsun_church": "https://heibox.uni-heidelberg.de/f/2711a6f712e34b06b9d8/?dl=1",
    "ema_lsun_church": "https://heibox.uni-heidelberg.de/f/44ccb50ef3c6436db52e/?dl=1",
}

Also available in Diffusers https://huggingface.co/models?library=diffusers&sort=downloads&search=ddpm
Cheers

Thanks for the pre-trained model link, besides, did you know of alternative links for these data https://github.com/ermongroup/SDEdit/blob/66a5e44db6c36d5979c323781675d67117c0fb04/functions/process_data.py#L8-L12 (they also failed)

Thanks for the pre-trained model link, besides, did you know of alternative links for these data
https://github.com/ermongroup/SDEdit/blob/66a5e44db6c36d5979c323781675d67117c0fb04/functions/process_data.py#L8-L12
(they also failed)

Got the same problem. Did you solve it?


UTRAnnotator VEP plugin

Description
A new plugin was added to VEP_Plugins, UTRAnnotator for Release 109.
Should be merged after https://github.com/Ensembl/ensembl-rest_private/pull/126 is merged.
This PR is sister to #559
Benefits
Provide docs for new VEP plugin
Possible Drawbacks
None
Testing
Test are successful
Changelog
This is DRAFT PR - waiting for Changelog changes to be declared.
eg. [/xenobiology/orthologs] Added the ability to look up orthologs and paralogs from klingons and andorians

Coverage decreased (-0.01%) to 44.427% when pulling f665ea88c1ad049e4a940d91a5213e4bf7457be4 on feature/vep_pu_utranno into dd80c7a0b1ce2457a9c0cd7d1319fa94a9516ca8 on release/109.


Alternative remove docs from publish

Alternative to #10839 that does work
Resolves: https://github.com/chartjs/Chart.js/issues/10835

@LeeLenaleee Hi.
    "files": [
        "auto/**",
        "dist/**",
        "helpers/**",
        "!dist/docs/**"
    ],

Tested locally

@dangreen nice catch, works indeed and lot cleaner as my approach 👍🏻
Never would have thought that was possible since it is an include by default :)


Review and improve CMSSW I/O metric reporting

Impact of the bug
CMSSW I/O metric reporting is inaccurate.
Describe the bug
(followup from https://github.com/dmwm/WMCore/pull/11533#issuecomment-1500437122)
The I/O metrics extract from the framework job report in
https://github.com/dmwm/WMCore/blob/431f08f70f14733830e546d0ac4b1adc04fc912c/src/python/WMCore/FwkJobReport/XMLParser.py#L446-L462
need to be reviewed, taking into account e.g.

exact meaning of each Timing-* number
what other numbers (that are reported further) are derived from those
why some quantities are read for some protocol (readMethod), and some from tstoragefile

In addition, the readMethod used in some of the quantities seems to not be well defined as the loop
https://github.com/dmwm/WMCore/blob/431f08f70f14733830e546d0ac4b1adc04fc912c/src/python/WMCore/FwkJobReport/XMLParser.py#L427-L432
picks the first protocol it sees that had non-zero number of operations. For a job reading a local file and a remote file, like pileup mixing in step chain the local file protocol comes before the remote root protocol in the XML (and more generally these numbers report statistics from any files opened by cmsRun, not just the event data).
How to reproduce it
N/A
Expected behavior
The quantities reported onwards by WMCore have a well-defined and documented meaning.
Additional context and error message
How much do these performance metrics matter for WMCore itself, or does it just treat them as "black box" defined by someone else?

@makortel thank you for creating this issue, Matti.
I think this code - or part of it - is likely more than 10 years old.
I would say that WMAgent is mostly propagating these FJR metrics upstream, so we would need to hear back from you/Core CMSSW team the following:

what are the relevant I/O metrics that we need to collect from the FJR (these will get published to at least WMArchive)
we might want to review the CPU performance metrics as well (we might want to decide to have a specific issue for that)
are those documented anywhere?
can we keep renaming some of those (like converting bytes to MB, and so on)

this way, we can agree and ensure that the relevant FJR metrics are consumed and published upstream, including a documentation of Core Framework + WMAgent metrics.

@makortel , I'll start working on this issue and to start I need an example of FWJR coming from CMSSW along with questions Alan posted in his comment. Please provide this info at your convenience.

@vkuznet @amaltaro
At this time I'm not able to give full answer to the questions of Alan in https://github.com/dmwm/WMCore/issues/11538#issuecomment-1502569727
I attached an example job report (from CMSSW_13_1_X_2023-04-24-1100 IB, workflow 11834.0 step 3) here
JobReport3.xml.txt

what are the relevant I/O metrics that we need to collect from the FJR (these will get published to at least WMArchive)

At this time I don't have a good answer. We'd need to think about these out in core software as well (may well happen only after CHEP).

we might want to review the CPU performance metrics as well (we might want to decide to have a specific issue for that)
are those documented anywhere?

With "those" do you refer to the I/O metrics or CPU performance metrics? The I/O metrics are, in principle, documented in https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideStorageStats, but given its age I think we (core) should refresh our memory (even if the storage system has not evolved much since that time)

can we keep renaming some of those (like converting bytes to MB, and so on)

I don't object as long as the unit is clear.

@amaltaro , @makortel , I used Matti's xml file and it provides the following Timing- metrics:

Timing-file metrics related to IO on the input LFN
Timing-root metrics related to IO of root operations over LFN
Timing-tstoragefile metrics related to IO storage file operations

The WMCore XMLParser will take first non-zero metric and use it in construction of FWJR report JSON. In this specific case it picked Timing-file-read-numOperations metric. In my view all of these metrics should be reported to upstream, e.g. WMArchive, since none of us, and neither Core software or WMCore groups, have a-prior knowledge how these metrics should be used. We may guess and pick one or few combinations but it will simply cut off entire picture.
Therefore, my suggestion would be to keep current code intact for backward compatibility, but add new sections to FWJR like:
{"file-io": {...}, "root-io": {...}, "store-io": {....}

Doing this way we'll

preserve existing upstream clients/dashboards which relies on current FWJR structure, and
we'll enhance FWJR with specific IO metrics which we may used in a future studies, e.g. dashboard. The new categories will clearly define IO operations and remove ambiguity of their interpretation.

If you agree with such proposal I can make necessary changes to the code to introduce these groups. We'll also need to adjust WMArchive schema (part of WMCore) and add them on a MONIT side to preserve their appearance in ES/HDFS.

With this PR https://github.com/dmwm/WMCore/pull/11575 I provide additional IO and Site metrics to FWJR. Please check full report here.
      "performance": {
        "storage": {...},
        "memory": {...},
        "cpu": {...},
        "multicore": {},
        "io": {...},
        "site": {...},
      }

So far, I put in flat structure the IO and site since usually in MONIT we prefer flat structure. The IO part contains all Timing-XXX metrics, while site part contains XrdSiteStatistics metrics.
Let me know your feedback about it.

Do I understand correctly that https://github.com/dmwm/WMCore/pull/11575 just forwards all the performance numbers from the job report XML file to the JSON file?
I suppose this approach would mean that any derived quantities (that are currently calculated in https://github.com/dmwm/WMCore/blob/431f08f70f14733830e546d0ac4b1adc04fc912c/src/python/WMCore/FwkJobReport/XMLParser.py#L446-L462) would have to be defined in the monitoring layer, did I get that right?
I mean, in general we could provide such "summary metrics"

by the framework
by the WM (as done currently)
by the monitoring

The question of exact meaning of each I/O metric  comes back to the Core software, as, at least currently, does also the question of what derived quantities are useful to identify and investigate CPU efficiency problems. That would favor the putting the logic to the framework level, so everyone else would see well-defined relatively easy-to-understand numbers. On the other hand, this would also be the least flexible option towards changes, as we can't change existing releases.
Calculating the derived quantities in the monitoring layer would probably be the most flexible. On the other hand, it is also farthest away from where the "raw quantities" are defined, and whenever we'd change certain parts of the framework, we'd also need to update the monitoring code accordingly, and in a way that the monitoring code can digest information from different CMSSW versions.
The current WM option is then somewhere in between.
We'd still need to do the "review" part of this issue, and I'm a bit hesitant for making changes before we have gained better understanding of the big picture of the I/O metrics.

Matti, your understanding is correct, but as I wrote to preserve backward compatibility I think we should preserve current WM logic. Said that, in my view, WM is a wrong layer to do the aggregation for many reason:

it consumes metrics from CMSSW
it puts its own logic to the aggregation which may or may not be desired at monitoring layer (dashboards)
it add another layer of complexity to metrics which are supplied from upstream code, e.g. CMSSW

When I was in charge of CMS Monitoring team I always advocate for transparent storage of metrics since it is only end-user can tell what they want. Therefore, in my view you should clearly define logic/metrics at CMSSW layer, the WM should serve as a proxy, or if it does perform some other actions within WM should add its own metrics to report, and metrics should be shipped to MONIT "as is". Moreover, it is desired to keep metrics with specific prefix to have clear definition of them, e.g. if specific metrics are defined within CMSSW, let it be CMSSW_XXX, while if metric is calculated in WM let it be WM_XXX. Doing this way, we will have clear definition of metrics and let CMS Monitoring team provide appropriate dashboard for end-users based on their specific use-case(s). Said that, if CMSSW wants to change some metrics it should also follow backward compatibility path, i.e. and instead of changing existing metrics it should add whatever derivatives we may have. The metrics storage is cheap and we can easily extend existing schema.
The above approach is very successful in other layers like various exporters within Prometheus where we may have different providers of similar metrics, e.g. system metrics can be provided by different exporters based on /proc file system or based on application based ones. Having both provide much more reach representation of the monitoring layer, and aggregation can be possible across multiple dimension without affecting underlying code-base.

@makortel Hi Matti, apologies for not following this ticket very closely, but did we discuss what is the preferred way to deal with these CMSSW statistic metrics? Is it:
a) collect everything that is provided by CMSSW FJR and push it upstream to CMS Monitoring; or
b) do we want to identify what are the metrics that we consider really relevant and only collect and propagate those
?
With those IO and Site new metrics implemented in the PR linked by Valentin, should we make similar conversions as it is already done for other metrics (e.g., converting from bytes to mega-bytes, and so on)?

Does the MONIT infrastructure allow us to define (and maintain) the higher-level metrics such that they would be easy to use in the various dashboards without everyone having to derive the high-level metrics themselves? (and e.g. if we identify a bug in some of the high-level metric definition, we can fix it only once and the fix gets automatically applied for everyone who is using it?)

@mrceyhun , @nikodemas , @leggerf , @brij01 could you please answer Matti's question about MONIT. I do know that Prometheus setup allows custom aggregated metrics, and we have few of those for DBS, but this is a new use case which we discuss. In my view we should stored unaggregated metrics to MONIT(OpenSearch/ES) while later build dashboard for those we need. Matti questions is slightly different, does OpenSearch/ES allows to define aggregated metrics from specific index and if so can we do that on our own (CMS side). If it does I'll assume the following data flow:
WMAgent -> CMSSW (job) -> WMAgent -> WMarchive -> MONIT -> ES/HDFS -> aggregation -> ES/HDFS

Hi @vkuznet, by default, OpenSearch does not do aggregation of an index and store the results in different index. We need another service to do it.
The aggregation and then storing aggregated data in different index(agg) in MONIT -> ES/HDFS -> aggregation -> ES/HDFS is possible in 2 ways:

by consuming Messaging team's topic that we use while sending data to MONIT: consume the raw data from this topic, aggregate and send results to MONIT aggregated index.
by consuming MONIT Kafka queue, which has 3 days of retention policy, with Flume service, like we do incCondor Job Monitoring: raw data is aggregated hourly on MONIT side and stored in long-term OpenSearch index. MONIT recently informed us that they can provide access to their Kafka, however, they have lack of resources to deploy aggregation service(Flume) in their infrastructure, so it should be deployed in CMS infra. In our experience, if schema does not change, it is a very stable service.

In any case, we need an additional service to aggregate data.
My general comment for WMArchive data is that we should get rid of nested JSON schema. It makes difficult to create queries in Grafana and OpenSearch. Just because of this difficulty in WMArchive documents, we developed a new service https://cms-monitoring.cern.ch/web/cpueff which does aggregations in Spark. IMHO, if we have a chance to improve the schema, it will solve many of the problems and will limit the need of additional services to provide required monitoring.
Secondly, in monitoring point of view, it is hard to understand  the data. A single WMArchive document in MONIT includes all the details of a job; including very long error log as a text, LFNArrays/LFNArrayRef as string lists, and also many numeric values in nested step arrays. If you can separate data to two index such that first one contains only flat numeric details of a job used in monitoring and second one has general job information such as error logs, LFNArrays, etc. used in debugging, it would be a great improvement for monitoring and for future aggregations. In this regard, backward-compatibility will be an issue but it will pay off in near future. In the mean time, we can continue to use and support current monitoring. In short, new flat schema will increase the data volume, so we should get rid of the parts that are not needed for monitoring.
About Prometheus usage, I think it is not a good fit. Because you need to report all job results and none of them should be missed. Therefore, you have to expose all job metrics to Prometheus at most in each 5 minutes; which I think not possible. Also Prometheus cannot store values other than numeric type like logs, string arrays, etc.. If you can expose limited set of high-level metrics, then it can be a good use-case IMO.

@mrceyhun , thanks for your feedback, now it is clear to me how to move forward. For the record, originally WMArchive was designed to store FWJR docs and it was not indented to be used in monitoring. Later we realized that this data is useful and now data-ops relies on it for Monitoring their activities.
Said that, here is my proposal how to move forward:

flat out WMArchive data for monitoring purposes and adjust the service to have three different streams to MONIT 3 distinct MONIT topics:

original WMArchive data should be intact and go to MONIT and ES/HDFS
we create flat out WMArchive data for monitoring purposes and send to to separate topic
we will add new configuration file to WMArchive to declare aggregation, e.g. take metrics A and B and create another one via specific math operation/function.  This aggregated data will to to another MONIT stream.

If we'll add such configuration in place then we can solve the issue with code modification on CMSSW or WM sides and WMArchive will become a proxy for data injection into MONIT. It will continue to supply original data for backward compatibility, it will create new stream for data monitoring needs, and it will aggregate data based on external configuration file which we can adjust as we go. The benefit of this proposal is clear, it will allow to produce as many metrics from CMSSW/WM sides as we desire and provide dynamic metrics aggregation in WMArchive which will be considered as proxy server where WM will send the data. On a MONIT side we will not deal with aggregation either (as it is quite complex based on Ceyhun information) but we'll get the aggregated data and it can be configured.
@makortel , @amaltaro, @mrceyhun  please provide your feedback on this proposal, and if we agree I can transform it into set of actions.

@vkuznet I agree with 3 streams. I believe all 3 will include common fields to join them like fwjr id, job name, campaign etc.. You can use same WMArchive topic and set different "metadata.type" for new 2 streams, which creates different index templates automatically without asking to MONIT.

Just to make sure I am following this discussion. The 3 streams of WMArchive monitoring would be:

original WMArchive data should be intact and go to MONIT and ES/HDFS

nothing changes. I understand this will provide the backwards compatibility.

we create flat out WMArchive data for monitoring purposes and send to to separate topic

who will create this new flatted out document? Is it WMAgent, MonIT, or who will consume data from one queue and put it in another? Do I understand it right that the input would be a CMSSW FJR? Or would it be the full and nested WMArchive document?

we will add new configuration file to WMArchive to declare aggregation, e.g. take metrics A and B and create another one via specific math operation/function. This aggregated data will to to another MONIT stream.

same question as above. Who will be responsible for aggregating data? Is it WMAgent, MonIT, or who will consume data from one queue and put it in another?
In addition to these question, I think we still have to agree on which metrics we want to track and which ones we want to aggregate.
So, the original question still remains. Should we just collect EVERYTHING that CMSSW can provide us and push it upstream? Or should we collect only information that we deem necessary to push upstream and use in monitoring?

The WMArchive will become proxy between WM and Monit. As such, this service will consume unfiltered WM data and perform flattering and aggregation. Therefore, WM will collect CMSSW, and send all possible metrics. It can be part of WMArchive schema or not. Then WMArchive will perform filtering of data, and split it into 3 streams. The second item in my list is to convert existing WMArchive FWJR record into flatten form. This will be done by WMArchive. And, we only need to agree where to include CMSSW metrics, I suggest to add them to FWJR record we send to WMArchive as independent section. Doing this way we'll have everything presented in a record, and then WMArchive will perform the job of splitting, flatten, aggregation and send it over to MONIT.

we will add new configuration file to WMArchive to declare aggregation, e.g. take metrics A and B and create another one via specific math operation/function.  This aggregated data will to to another MONIT stream.

Where is the code or logic for the aggregation hosted? How is it deployed? How can it be tested?

Matti, there is no yet code for aggregation and needs to be added to WMArchive. Said that, I would like to propose simple JSON syntax to specify common (or custom aggregation) function along with list of metrics to consume, e.g.
[
   {"aggregated_metric": "its_name", "metircs": [list of metrics to use], "function": "sum"},
...
]

Doing this way, the most common math function can be easily added, while if we need a custom function we'll add it manually to aggregated module. Since we host WMArchive on k8s, the deployment will only require new configuration for aggregated metrics (if we use common math functions) and new aggregated module if we'll add new custom function. And, we'll redeploy WMArchive service and neither touch WM/CMSSW or MONIT side. Then the data flow will contain new aggregated records which we may consume in different form.

The WMArchive codebase is here: https://github.com/dmwm/WMArchive, the original implementation was python, the new one is Go based which only couple of pages of code for entire service, see https://github.com/dmwm/WMArchive/blob/master/src/go/wmarchive.go
The addition will require to produce new aggregate module to parse JSON structure and manipulate the data, the rest is already in WMArchive server code, e.g. how to handle incoming data streams and pass it to MONIT via Stomp AMQ.
Your understanding of fix logic is correct, if we need to change something it will be applied to new data, the previous data which reside in MONIT will stay intact. Said that, if we need to back-port the logic we'll need to write additional code which will get data from MONIT (HDFS) and inject it back to WMArchive or OpenSearch/ES. This is the way as do other aggregation (like site aggregation via Spark job on HDFS which run periodically on CMS Monitoring cluster). Therefore, the process will not be new and Monitoring team can extend it.
The main advantage of storing raw metrics is that it can be reprocessed again and again, while if we'll add aggregation to CMSSW or WM, then we can't recover anything since original information will be lost forever.

Thanks @vkuznet. From the CMSSW framework side I think the overall plan sounds good. We'll certainly need some help when getting started with the definitions of the high-level aggregations. And for that we still need to go through the code to understand/remind the meaning of the "raw metric numbers", and which high-level aggregations would make sense. Until then I'd ask the raw numbers to not to be propagated forward from WMArchive (except maybe something for testing purposes to make sure the whole chain works).
In addition of the I/O metrics (let me know if these would warrant their own issue), in order to better understand the various CPU efficiency numbers, it would be useful to record and show in the monitoring

begin and end timestamps (or begin timestamp and duration) of the WM program (sorry for the wrong term) that is run in the HTCondor job, and that calls cmsRun
begin and end timestamps (or begin timestamp and duration) of each cmsRun invocation
(bonus) total CPU time spent by the WM program (again sorry for the wrong term)

I'm not sure if any of these quantities are being recorded already, but if not, would it be feasible to add? (we'd also be interested of similar timekeeping by the pilot that runs the HTCondor jobs, but that's likely for other folks to talk to?)

I think we have identified the short and longer term developments, and the relevant follow up tickets have also been created. With that, we can now move this issue back to "In Progress" (from Waiting) and resume working on the already opened PR, such that it comes in in the next WMAgent stable release (beginning/mid of July). @vkuznet

This issue has been in waiting since May 22.  @vkuznet, can you please summarize what this is waiting on?

@klannon , I had 2 slots of vacations, and Matti was on vacation too, then we had several iterations on implementation. This was discussed during today's WM meeting and action item on me was to prepare summary doc about these and other metrics. You can find its first draft over here. Right now, I'm awaiting final review of my PRs (from Alan) and approval from CMS Monitoring to propagate these metrics to MONIT.

I moved its status in the project board to "In Progress". If we get to a point that we cannot merge and/or deploy it because it has external dependencies (e.g. psutils in the worker nodes), we should move it back to Waiting.

Alan, I think closing of this issue was premature and should be done only when https://github.com/dmwm/WMCore/pull/11663 is merged, otherwise we can't get CMSSW metrics.

For the record, here is new issue https://github.com/dmwm/WMCore/issues/11758 to refer to required clean-up of performance metrics.

Yes, there were multiple PRs closing this issue, so once one of them got merged, it automatically closed this ticket. Now we have both PRs merged and this issue is correctly closed.
For clarity, this issue has been closed with https://github.com/dmwm/WMCore/pull/11696 and https://github.com/dmwm/WMCore/pull/11663


Update CameraManager.swift

hi
The shouldEnableTapToFocus function didn't work, so I fix it.
Can you check it ?

@saegyeol
yes, somehow must have been changed in a previous version. It's resolved on version 4.0.1.
Thanks 👍


Migrate to federated plugin

I was thinking it would be good to migrate the plugin to the federated plugin structure to facilitate contribution. As I understand, this would enable developers to contribute platform implementations in their own repositories, without having to go through PRs. Also, it's how the packages maintained by the Flutter team (such as url_launcher and shared_preferences) are structured now.
The necessary steps are detailed here:

Create platform interface package
Publish to pub.dev
Refactor base package to use platform interface package

I think I'm done with step 1, you can preview it here. I'm not sure how to proceed though, since I can't publish under your name (of course). Let me know if you're interested in doing this. If you don't have the time / resources to do it, I understand, too!

Interesting. I need to learn a bit more about federated plugins. I thought it's just another way to organize the codebase 🤔

I am also just learning about it. The concept is fairly new and the Flutter team hasn't finalized it yet, but it seems it's the way they want to go in the future.
Since federated plugins involve a lot more publishing, I think what I want to do is publish a "federated" version of the plugin using my account to save us both a lot of back and forth. Also, I have a few major changes in mind that will be easier to carry out that way.

From what I read it doesn't sound like a lot of publishing. After the plugin is converted to a federated plugin, I need to endorse your plugin by adding it to  pubspec and then publish a new version. Doesn't sound too me like a lot of hassle. I think it would be better to have the community working on one repo together instead of splitting.


fix: Updated provder configuration in examples/complete and Added log…

what

Added Log Analytics Resource arguments in example/complete
Added Provider configuration in key-vault module call in example/complete

why

Provider Configuration Allow to deploy with different subscriptions

Terraform Security Scan Failed
Show Output

Terraform Security Scan Failed
Show Output


infinite loading screen

After the newest update this one freezes me up at the loading screen. I've gone through every mod combination i had to see if it was interacting poorly with something else. it just won't load up to the main menu anymore

Have you installed the latest version? I raised this issue and I think it's fixed.
https://github.com/Vek17/TabletopTweaks-Base/issues/487

ah i didn't check the discord. i checked everywhere else and didn't see one. completely slipped my mind. thank you

Nope. Still didn't do it for me. I tried disabling bloodline stuff, still won't work

ok I found the right bloodline thing to disable to get it working, but with that running it still doesn't function

If you aren't able to load with v2.3.2 then the issue is almost certainly
not TTT-Base
On Fri, Sep 23, 2022, 11:33 AM Gwendovescry @.***>
wrote:

ok I found the right bloodline thing to disable to get it working, but
with that running it still doesn't function
—
Reply to this email directly, view it on GitHub
https://github.com/Vek17/TabletopTweaks-Base/issues/491#issuecomment-1256424992,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAWSQPF7CCC2473U3PALFHLV7XLVVANCNFSM6AAAAAAQUDMLGM
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


[Feature] Make dispatch_kwargs accept different separators

Description
Makes dispatch_kwargs accept different separators

Codecov Report
Patch coverage: 93.61% and project coverage change: +0.02 :tada:

Comparison is base (9f2442e) 87.95% compared to head (499ab3a) 87.98%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #253      +/-   ##
==========================================
+ Coverage   87.95%   87.98%   +0.02%
==========================================
  Files          28       28
  Lines        9537     9560      +23
==========================================
+ Hits         8388     8411      +23
  Misses       1149     1149

Flag
Coverage Δ

linux-cpu
86.01% <93.61%> (+0.03%)
:arrow_up:

linux-gpu
87.91% <93.61%> (+0.02%)
:arrow_up:

linux-macos-cpu
86.03% <93.61%> (+0.03%)
:arrow_up:

linux-stable-cpu
86.04% <93.61%> (+0.03%)
:arrow_up:

linux-stable-gpu
87.91% <93.61%> (+0.02%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

tensordict/nn/common.py
74.01% <91.66%> (+2.71%)
:arrow_up:

test/test_tensordictmodules.py
99.31% <100.00%> (+<0.01%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Fix dockerfile to build with gulp4/node10.

@jsaalfeld Please review my changes. Note: It was required to run COPY . /app before yarn command. Otherwise yarn failed because file from gulp task (e.g. pdf-worker.js) not found. With this dockerfile changes docker build .runs successfully for me.

It should have run before because before the execution of yarn the files were copied (COPY package.json yarn.lock bower.json gulpfile.js /app/). But this way around it is much nicer.