is yt-dlp setting the wrong language on the extracted audio track?

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm asking a question and not reporting a bug or requesting a feature
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2023.10.13 (update instructions) or later (specify commit)
[X] I've searched known issues and the bugtracker for similar questions including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Please make sure the question is worded well enough to be understood
I'm downloading an audio track in a specific language (Italian) but when I use mediainfo on the output file I see Language = English. Am I missing a parameter?
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
# yt-dlp -vU -f "ba[language=it]" https://www.youtube.com/watch?v=21LWzguLAFE

[debug] Command-line config: ['-vU', '-f', 'ba[language=it]', 'https://www.youtube.com/watch?v=21LWzguLAFE']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>[b634ba742] (pip)
[debug] Python 3.11.6 (CPython x86_64 64bit) - Linux-6.5.9-1-default-x86_64-with (OpenSSL 3.1.3 19 Sep 2023)
[debug] exe versions: ffmpeg 6.0 (setts), ffprobe 6.0
[debug] Optional libraries: Cryptodome-3.19.0, brotli-1.1.0, certifi-2023.07.22, mutagen-1.47.0, sqlite3-3.41.2, websockets-12.0
[debug] Proxy map: {}
[debug] Loaded 1890 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version<EMAIL_ADDRESS>Current version<EMAIL_ADDRESS>yt-dlp is up to date<EMAIL_ADDRESS>[youtube] Extracting URL: https://www.youtube.com/watch?v=21LWzguLAFE
[youtube] 21LWzguLAFE: Downloading webpage
[youtube] 21LWzguLAFE: Downloading ios player API JSON
[youtube] 21LWzguLAFE: Downloading android player API JSON
[youtube] 21LWzguLAFE: Downloading m3u8 information
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[info] 21LWzguLAFE: Downloading 1 format(s): 251
[debug] Invoking http downloader on "https://rr1---sn-npa8xgp2-vgqe.googlevideo.com/videoplayback?expire=1699822780&ei=XOhQZevGB-2ZybgPp6a94AU&ip=<IP_ADDRESS>&id=o-ALfyWq8hY0b_gUm5H3MrpzhdTB_fKLtpkUtSlgI7Psyk&itag=251&source=youtube&requiressl=yes&mh=vE&mm=31%2C29&mn=sn-npa8xgp2-vgqe%2Csn-vgqsknld&ms=au%2Crdu&mv=m&mvi=1&pl=24&initcwndbps=985000&spc=UWF9f93A4sz-SUlboV7wfdByXJa8VXk&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=33634497&dur=2181.901&lmt=1677805644166615&mt=1699800787&fvip=5&keepalive=yes&fexp=24007246&beids=24350018&c=ANDROID&txp=5432434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ANLwegAwRQIgMx9TPfYOAH1w66jIltWjNCgmmLIcQtHEDvjWEi2s7lECIQChBITnkriDOqnplZhrfj4IaJrOR8nN03gMbdmS1wLHUA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AM8Gb2swRQIhAI3KZjQhcEikKWcdRIAPdTnbqx3-ndCyvp9Oe4cBFH_bAiA7cs-VB46KINJZwajbkyb_Mb2HyZx0l5saGvVyvj8J6g%3D%3D"
[download] Destination: Veppo, il paese dei migranti [21LWzguLAFE].webm
[download] 100% of   32.08MiB in 00:00:00 at 40.04MiB/s

# mediainfo Veppo,\ il\ paese\ dei\ migranti\ \[21LWzguLAFE].webm

General
Complete name                            : Veppo, il paese dei migranti [21LWzguLAFE].webm
Format                                   : WebM
Format version                           : Version 4
File size                                : 32.1 MiB
Duration                                 : 36 min 21 s
Overall bit rate                         : 123 kb/s
Writing application                      : google/video-file
Writing library                          : google/video-file

Audio
ID                                       : 1
Format                                   : Opus
Codec ID                                 : A_OPUS
Duration                                 : 36 min 21 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Compression mode                         : Lossy
Language                                 : English
Default                                  : Yes
Forced                                   : No

yt-dlp isn't doing any embedding or remuxing in your log; the wrong language metadata is coming directly from youtube.
I was about to suggest that you could manually correct this by adding --embed-metadata to your command, but when I tested it, it didn't work.
I looked into why that was, and it turns out that the embed metadata postprocessor only adds language metadata to streams if multiple formats are being merged (and requested_formats exists in the info dict). It will not do so when a single format is downloaded.
I think this might be the fix for that:
diff --git a/yt_dlp/postprocessor/ffmpeg.py b/yt_dlp/postprocessor/ffmpeg.py
index 323f4303c..7c904417b 100644
--- a/yt_dlp/postprocessor/ffmpeg.py
+++ b/yt_dlp/postprocessor/ffmpeg.py
@@ -780,7 +780,7 @@ def add(meta_list, info_list=None):
             yield ('-metadata', f'{name}={value}')

         stream_idx = 0
-        for fmt in info.get('requested_formats') or []:
+        for fmt in info.get('requested_formats') or [info]:
             stream_count = 2 if 'none' not in (fmt.get('vcodec'), fmt.get('acodec')) else 1
             lang = ISO639Utils.short2long(fmt.get('language') or '') or fmt.get('language')
             for i in range(stream_idx, stream_idx + stream_count):

Thank you for the prompt fix


License?

Hi:
Have you considered using the same license as yara proper, i.e. Apache 2.0?
Thanks for your kind consideration!

the reason why I am asking is that I wrote a generic unarchiver that builds on libarchive and 7zip here https://github.com/nexB/scancode-toolkit/tree/develop/src/extractcode

Hey, thx for the note and apologies for taking so long to respond. Been head down working ...
Give us a few days to digest the change and I will let you know here. I don't see it as a big deal but I need to run it by a few people here at Bayshore in order to make sure everyone is ok with the change.

@dre thanks for chiming in!

@dre FWIW, in a related domain (but Python-based) https://github.com/Changaco/python-libarchive-c was recently re-released from GPL to Public Domain

@dre any update?

License has been updated and should be reflected on next release

@dre Thank you very much for following up my Lord!


feat: bump release level to production/stable

Fixes #11 🦕
Release-As: 1.0.0

Wait for August 25

Adding Do Not Merge label until b/192425474 and b/184600669 are resolved.


Do you find contributors?

I've openned PR a week ago (https://github.com/cloocher/xmlhasher/pull/13). I don't really wont to support fork or copy of your gem. In the same time I'm able to help you with support of your gem. Can you give me access for merges or can we discuss that somewhere else?

thanks for the PR.  Just made you a collaborator.


WSL changed to version 1 after Win 10 Pro upgrade and got stuck in this version

Environment
Windows build number:
Microsoft Windows [versão 10.0.19042.867]

Your Distribution version:
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

Whether the issue is on WSL 2 and/or WSL 1:
Linux version 4.4.0-19041-Microsoft<EMAIL_ADDRESS>(gcc version 5.4.0 (GCC) ) #488-Microsoft Mon Sep 01 13:43:00 PST 2020

Steps to reproduce

I had this Windows update on queue since last Friday and restarted the computer today with the intend to let it happen. I'm not sure what build number it was before this.
Was using WSL2 normally with vscode extension while building this site (front-end dev) right after the update.
Once it was done, Windows restarted normally with everything seeming right.
Started Windows Terminal Preview (which defaults to Ubuntu) and opened vscode on the previously used folder.
Vscode opened with this warning telling me to use WSL2.
Closed it and checked the version in powershell with wsl -l -v and it returned version 1 to my Ubuntu distro (the only one I have installed).
Tried resetting it to version to with no luck. wsl --set-default-version 2 returns a standard message telling me (simply translated from pt-br) 'For more information on the major differences of WSL 2, visit...' and wsl -l -v will tell me again that it's on version 1.

Done the procedure to re-install WSL1 and upgrade it to WSL2 again, like it's said on https://docs.microsoft.com/en-us/windows/wsl/about, using admin powershell to enable wsl and virtual machine feature. Reinstalled the kernel too.
Then, once again tried setting up --set-default-version and no changes were made. I'm stuck with version 1.
Not sure on how to document this issue. Please, let me know if I can improve this.
Expected behavior
WSL2 to be fully functional and set as default version, after win 10 update.
Actual behavior
WSL1 is the default and I can't change it

Discussion at #6505.


Debug for Starlark not Always Available

See comment here
https://github.com/bazelbuild/intellij/issues/1706#issuecomment-1179635807
Repro:

clone https://github.com/bufbuild/rules_buf
create bazel project by opening the dir that containsexamples/workspace/WORKSPACE
attempt to debug buf_breaking_test / label fooapis_proto_breaking in fooapis/BUILD.bazel

Hello @abhillman, based on this check Starlark debugging is only available for build commands. So you need to create a run configuration with the bazel command set to build and the debugging should be enabled for it.

I'll close the issue. If you still can reproduce it, please reopen.


Keras and deeplearning4j compatibility policy

Keras and deeplearning4j compatibility policy
Is there a documentation / policy, which defines the version of Keras is guaranteed to be compatible with deeplearning4j?
We train the model with the python library Keras and import the model in our Scala infrastructure. We had a case that deeplearning4j couldn't import the Keras model due to newer version of Keras. (we still have old version of deeplearning4j running - it would fix if we upgrade the deeplearning4j, but we need to check if the deeplearning4j is still able to import the other models being built by older Keras version.).

@fspillner this has been fixed already with beta7. The "official" support can be documented here:  https://deeplearning4j.konduit.ai/keras-import/overview


change width of widget?

How can I change the width to 100%?

There is a default width attributes assigned to widget form if you are using $widget->getHtmlCode();.
You can use $widget->getUrl(); to obtain the original payment url and create your own iframe to embed it.


Codeforces Round 732 Problem A. AquaMoon and Two Arrays

Aim

I would like to add Codeforces Round 732 Problem A. AquaMoon and Two Arrays.

Details

https://codeforces.com/contest/1546/problem/A
LGM-SOC'21

Programming language

[ ] C
[x] C++
[ ] Java
[ ] Python

@ravikr126 @Ayush7614 @Amit366 Please Assign me this issue. I will solve it in C++.
LGM-SOC'21
Thank You


OnEvent decorator doesn't create listener in testing environment

I'm submitting a...

[ ] Regression
[x] Bug report
[ ] Feature request
[x] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead post your question on Stack Overflow.

Current behavior

When trying to use the event emitter inside a TestingModule the OnEvent decorator doesn't add a listener.
Expected behavior

OnEvent should add the listener.
Minimal reproduction of the problem with instructions

describe('LoremService', () => {
  let service: LoremService;
  let eventEmitter: EventEmitter2;

  beforeEach(async () => {
    const module: TestingModule = await Test.createTestingModule({
      imports: [EventEmitterModule.forRoot()],
      providers: [LoremService],
    }).compile();

    service = module.get(LoremService);
    eventEmitter = module.get(EventEmitter2);
  });

  it('should perform some operation when user created', async () => {
    jest.spyOn(service, 'someOperation');

    await eventEmitter.emitAsync('user.created');
    // eventEmitter.hasListeners() returns false

    expect(service.someOperation).toHaveBeenCalled(); // test fails
  });
});

@Injectable()
export class LoremService {
  @OnEvent('user.created')
  async someOperation(): Promise<any> {
    // ...
  }
}

What is the motivation / use case for changing the behavior?

It should be possible to test that some logic is called when a certain event is triggered.
The documentation should be extended to describe how to write unit tests with the event emitter.
Environment

Nest version: 7.5.5

For Tooling issues:
- Node version: v12.18.0
- Platform: Linux

Others:

Make sure to call await app.init() to trigger lifecycle hooks (this is when event listeners binding occurs).

Hi, I would suggest to mention it in the docs also! Thank you


Load Last Session Multiple Tab "0"

Config file contains multiple tabs called 0. These are not being deleted.

Looks like .forget() method still keeps tab under notebook.children

Resolved V0.33.5


Use pre-trained model

Hello, I am not sure how to use the pretrained model, including where to save it and how the pretrained model is read and used. Also, in order to visualize the result, do I have to train by myself?
If anybody can help me with this issue, it would be grateful. Thank you very much.

Hello DioMou, I hope I can help you.
Do you mean how to run a demo using the pre-trained model?
Firstly, you should download the pre-trained model and prepare a demo video. Then you should rename the video as 'demo.mp4' and put it in the root dir.
Secondly, If you open demo.py, you can see that in line 178:
   _, _ = model.load_state_dict(checkpoint['model'], strict=False)
and then you can find the arg in line115:
**    parser.add_argument('--resume', default='', help='resume from checkpoint')`**
so you should run as:(for example, the checkpoint file I downloaded is 619mot17_mot17.pth)
python demo.py --resume=models/619mot17_mot17.pth


OpenTracing Instrumentation for redis?

When can support redis

@qw2283345 hi,
what API are you using to access Redis storage? RedisTemplate or something else?

I can't express myself clearly,  i need  monitoring redis

At the moment we don't have any specific plans for redis instrumentation. If anybody is interested please comment or :+1:  on this issue.

Is anyone working on this already? I wouldn't mind taking a look since I need this feature

@ddcprg I am not aware of anybody. It would be great if you could have a look :+1:

I've managed to spend some time on this issue this evening. I'll try to push a PR tomorrow.

I've managed to spend some time on this issue this evening. I'll try to push a PR tomorrow.

Done in #182

thanks @pavolloffay


Issue trying NGinx docker image

I am trying to test docker support in Diego and I am facing issues because the ports used by the docker images.
FAILED
Server error, status code: 400, error code: 100001, message: The app is invalid: ports Ports must be in the 1024-65535.
I understand the issue is that NGinx by default tries to use port 80.
Thanks in advance!

Hi @debu66er, thanks for the report!
After some brief investigation, we think this may be an upstream bug with CC or routing. We have a story prioritized to investigate this further, and we'll let you know when we find out more.

Great! Thank you very much.
I tested also Tomcat, Jetty and Liberty docker containers. All of them finished with errors by timeout but the routes seem to be up. But when testing the home pages I always get 404. This doesn't happen if I test the docker containers in a local standard docker installation.
Not sure if it is related or not with NGinx docker container issue as I don't get any error about port used. But they are other docker samples that seems not working properly.
Let me know if I can share any kind of log or test that can help you with the issue.

i reviewed the docker containers you listed, i didn't have a problem started tomcat and jetty, although jetty doesn't seem to have any pages available by default so that 404 is from the app itself (could confirm with cf logs)
liberty and nginx both have multiple EXPOSE ports, which i need to do some checking to see where we're at on supporting that.  i know the routing team has been involved.  i notice on the logs for these that healthchecks are firing against both ports 80 and 443, and maybe something is happening at the cloud controller layer preventing routes to ports < 1024.

Thanks @davidwadden I revieved the tests with new v0.6.0 and:
Tomcat official Docker Image worked fine.
BTW, it is listening to port 80 although it exposes 8080. Is there anyway to specify a different port when deploying into CF?
NGinx official Docker image still failing with the same error:
Server error, status code: 400, error code: 100001, message: The app is invalid: ports Ports must be in the 1024-65535.
And WebSphere Liberty official Docker image tries to start because I get logs from it, but it seems it has a problem creating /log directory. Perhaps rights issues (although it runs fine in standard Docker runtime):
2016-02-10T21:09:22.31+0100 [CELL/0]     OUT Creating container
2016-02-10T21:09:26.62+0100 [CELL/0]     OUT Successfully created container
2016-02-10T21:09:26.62+0100 [CELL/0]     OUT Starting health monitoring of container
2016-02-10T21:09:27.54+0100 [APP/0]      OUT Launching defaultServer (WebSphere Application Server <IP_ADDRESS>/wlp-1.0.11.cl50820151201-1942) on IBM J9 VM, version pxa6480sr1fp10-20150711_01 (SR1 FP10) (en_US)
2016-02-10T21:09:27.97+0100 [APP/0]      ERR TRAS0037E: The system could not create new file /logs
2016-02-10T21:09:27.98+0100 [APP/0]      OUT CWWKE0005E: The runtime environment could not be launched.
2016-02-10T21:09:27.98+0100 [APP/0]      OUT CWWKE0018E: An exception occurred while launching the runtime environment: java.lang.NullPointerException
2016-02-10T21:09:27.98+0100 [APP/0]      ERR com.ibm.ws.kernel.boot.LaunchException: Caught unexpected exception java.lang.NullPointerException
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.kernel.boot.internal.KernelBootstrap.rethrowException(KernelBootstrap.java:524)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.kernel.boot.internal.KernelBootstrap.go(KernelBootstrap.java:207)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.kernel.boot.Launcher.handleActions(Launcher.java:245)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.kernel.boot.Launcher.createPlatform(Launcher.java:120)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.kernel.boot.Launcher.main(Launcher.java:55)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.kernel.boot.cmdline.EnvCheck.main(EnvCheck.java:61)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.kernel.boot.cmdline.EnvCheck.main(EnvCheck.java:37)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR Caused by: java.lang.NullPointerException
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.logging.internal.impl.LogProviderConfigImpl$InitConfgAttribute.getLogDirectory(LogProviderConfigImpl.java:387)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.logging.internal.impl.LogProviderConfigImpl.doCommonInit(LogProviderConfigImpl.java:173)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.logging.internal.impl.LogProviderConfigImpl.<init>(LogProviderConfigImpl.java:117)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.logging.internal.impl.LogProviderImpl.configure(LogProviderImpl.java:32)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.kernel.launch.internal.LauncherDelegateImpl.getLogProviderImpl(LauncherDelegateImpl.java:186)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.kernel.launch.internal.LauncherDelegateImpl.launchFramework(LauncherDelegateImpl.java:92)
2016-02-10T21:09:27.98+0100 [APP/0]      ERR    at com.ibm.ws.kernel.boot.internal.KernelBootstrap.go(KernelBootstrap.java:201)
2016-02-10T21:09:27.99+0100 [APP/0]      ERR    ... 5 more
2016-02-10T21:09:28.02+0100 [APP/0]      OUT Exit status 24
2016-02-10T21:09:28.02+0100 [CELL/0]     OUT Exit status 0
2016-02-10T21:09:28.04+0100 [API/0]      OUT App instance exited with guid efc25c36-8850-4c55-9517-e212807e84ee payload: {"instance"=>"b3978878-e91f-441a-4c55-1d6a2e5eb6d2", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* 2 error(s) occurred:\n\n* Exited with status 24\n* cancelled\n* cancelled", "crash_count"=>4, "crash_timestamp"=>1455134968030522274, "version"=>"9d78ff6c-2d1d-4e2d-96cc-450b145f5c25"}
`

Could you try with
cf push -o nginx ngx -u none
to disable health check?
I can't tell yet why health check does not pass but I could deploy the nginx official docker image this way.
I am curious about your use case though as Cloud Foundry provides an nginx buildpack already.
https://github.com/cloudfoundry/staticfile-buildpack

For the WebSphere Liberty I could test it onto a full Pivotal Cloud Foundry installation.
Startup took about 2 minutes given all the installation that happens in this WebSphere Liberty image (https://github.com/docker-library/docs/blob/master/websphere-liberty/tag-details.md#websphere-libertywebprofile7 ) - and that was a fast datacenter network to download from the internet.
I have used:
cf push -o websphere-liberty:webProfile7 ly
After startup I can see logs as well as the welcome page for the WAS Liberty.
I will try on PCFdev later.
Note that IBM has a WAS Liberty builpack for Cloud Foundry since some time so you may also want to try it (cf push .. -b ...)
(2013) https://blog.pivotal.io/pivotal-cloud-foundry/products/ibm-liberty-buildpack-on-cloud-foundry
https://github.com/cloudfoundry/ibm-websphere-liberty-buildpack

Closing this issue due to inactivity, and because we don't believe any remaining issues with the nginx Docker image are specific to PCF Dev.


Enable IPv6 support by default

/kind feature
Description
IPv6 support needs currently explicitly enabled by --sysctl net.ipv6.conf.all.disable_ipv6=0.
That's the same behavior like in docker and I'm wondering if we want to change it to a more sane default?
If you're fine with that I'd like to submit a PR, otherwise feel free to close this.

I like turning it on by default.  But what if a Admin wants this turned off by default?
Should we check how the system is configured and then set the settings within the container to match?

@saschagrunert Where do I see this disabled?
sudo podman run fedora cat /proc/sys/net/ipv6/conf/all/disable_ipv6
0

I think our first steps here are probably throwing IPv6 support into the default CNI config we ship, and enabling the flag for static IPv6 addresses - which, AFAIK, is still blocked on verifying CNI 0.8.1 is available everywhere and building new test images with it.

@saschagrunert Where do I see this disabled?
sudo podman run fedora cat /proc/sys/net/ipv6/conf/all/disable_ipv6
0

Interestingly, it really seems not disabled by default. I guess the main issue is what @mheon mentioned. A CNI config like this seems not to work right now:
{
    "cniVersion": "0.3.0",
    "name": "podman",
    "plugins": [
      {
        "type": "bridge",
        "bridge": "cni1",
        "isGateway": true,
        "ipMasq": true,
        "ipam": {
          "routes": [
            { "dst": "<IP_ADDRESS>/0" },
            { "dst": "1100:201::1/24" }
          ],
          "ranges": [
            [{ "subnet": "<IP_ADDRESS>/16" }],
            [{ "subnet": "1100:201::/24" }]
          ]
        }
      },
      {
        "type": "portmap",
        "capabilities": {
          "portMappings": true
        }
      }
    ]
}

> sudo podman run -it debian:stretch ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet <IP_ADDRESS>/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0@if90: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether e2:a2:86:8f:19:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::e0a2:86ff:fe8f:19e1/64 scope link tentative
       valid_lft forever preferred_lft forever

If we leverage the latest OCICNI version then it should be possible to provide the same dual stack IPv6 support like in CRI-O, right?

Yep, once we have CNI updated properly, we can update OCICNI and get v6 support enabled.

Yep, once we have CNI updated properly, we can update OCICNI and get v6 support enabled.

Alright, I see that we have another issue open on that topic: https://github.com/containers/libpod/issues/3245
I will close this one in favor of the other.

@mheon Will the future default IPv6 support be implemented through nat6, or will DHCPv6 get a subnet and assign IPv6 address to container?
Nat6 may be too evil🥶

Definitely not NAT. Right now, I think our approach might be requiring users to manually configure v6 subnets manually (maybe provide a range in the configuration file that podman network create can pick out of).


refactor: move dapp tutorail to standalone menu

@yfeng2824 it seems the menu icons are mess up since I add one new menu. where is the code to define the icon?

I'll take care of this in my PR to restructure the sidebar menus. Maybe we can close this PR as I'll do the same thing


Export control does not ask for file name

@e-gallis commented on Sat Mar 30 2019
On step 4 in the example above, it says that clicking on the Export control allows one to specify the name of the file into which the data will be exported to.
It's not !
When clicking on the Export control, the data is immediately exported to a zip file with the name of the data collection specified in the Data property of the Export control.
I'd rather have what's described in the example: that the Export control asks me for a file name before exporting the data.
(Tested on Chrome 73.0.3683.86)

Détails du document
⚠ Ne pas modifier cette section. C’est obligatoire pour docs.microsoft.com ➟ Liaison des problèmes GitHub.

ID: bb525c3c-c019-d110-76dd-1e166b77ba60
Version Independent ID: 6c1b5220-ee4c-0d91-fae8-ab76b8043a27
Content: Contrôles Export et Import : référence - PowerApps
Content Source: powerapps-docs/maker/canvas-apps/controls/control-export-import.md
Service: powerapps
GitHub Login: @fikaradz
Microsoft Alias: fikaradz

Hi team,
the user is reporting the actual "Export" action doesn't behave as described in the document

Thank you for reaching out. The content is updated to reflect updated behavior. Please allow some time to reflect changes.


Two strategies:

lazy GC: When creating a cluster , remove all potential previous garbage. When deleting a cluster, remove all potential garbage (see https://github.com/coreos/etcd-operator/pull/507#discussion_r93810299)
periodic full GC (every 10minutes or so): find out all pods/svcs that are parentless -- that doesn't have a managing cluster. For efficiency concern, we can do some caching.

periodic full GC (every 10minutes or so):

let's just do full scanning initially. we can optimize it later on.

I am less concerned on the full GC case. A normal operator would manage ~O(10) etcd clusters. The totally selected items with (app=etcd) is at most ~O(100). Finding the orphan items is trivial.
there are already other places we do full select. we should optimize all by using reflector for example.

Can you create a design doc talking about how:

we do ownership tracking
what is our GC initial strategy and why

@xiang90
Sure

closed by #519


Rpackage Version <IP_ADDRESS> crashes RStudio

Running the R package Version <IP_ADDRESS> crashes RStudio. This occurs  at the xgb.DMatrix call in the xgbtrain example. As well as when using saved xgboost objects fit under v<IP_ADDRESS> or fitting new models.
data(agaricus.train, package='xgboost')
data(agaricus.test, package='xgboost')
dtrain <- xgb.DMatrix(agaricus.train$data, label = agaricus.train$label)
R.version
platform       x86_64-apple-darwin15.6.0
arch           x86_64
os             darwin15.6.0
system         x86_64, darwin15.6.0
status
major          3
minor          5.2
year           2018
month          12
day            20
svn rev        75870
language       R
version.string R version 3.5.2 (2018-12-20)
nickname       Eggshell Igloo
RStudio Version 1.2.5033

I can't reproduce on Linux.  @hcho3 Could you please help taking a look?

Sure, glad to help.  v<IP_ADDRESS> worked fine on the described MACOS system. v<IP_ADDRESS> crashes on the xgb.train example when running xgb.DMatrix().  Also crashes when I load and predict using a v<IP_ADDRESS> xgboost object build on the below Ubuntu.
It seems to run fine under RStudio server on:
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

@tbstockton What happens if you simply run it with Rscript without using studio?

I also tried on RStudio, didn't reproduce it.

@tbstockton I just tried it on my Macbook Pro (2019, MacOS Catalina) and did not reproduce the issue.
I installed R and RStudio using Homebrew:
brew install r
brew cask install rstudio

Hi there. I have the same problem. My configuration is similar to the above.

R version 3.6.3
RStudio Version 1.2.5042
xgboost <IP_ADDRESS>
macOS Catalina 10.15.4

I'm pretty sure that I have enough ram for the dataset I use.

@lorentzenchr How did you install R? So far I was not able to reproduce the problem.

@hcho3 I installed R via R-3.6.3.pkg from https://cran.r-project.org/bin/macosx/
shasum R-3.6.3.pkg
2677aaf9da03e101f9e651c80dbec25461479f56
I don't know where this exact version went to.

I updated to R version 3.6.3 on macOS Catalina  10.15.1 and still get the crash on xgb.DMatrix
Tom

I can confirm that this is a regression. XGBoost version <IP_ADDRESS> works fine.

It's quite difficult, since I cannot reproduce the issue on my end.

Here is a minimal code example, if it helps:
library(xgboost)
xgb.DMatrix(matrix(1:2, nrow=1))

The R interpreter gives the following error mesages and quits:
OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/
zsh: abort      R

The package xgboost was installed from CRAN and compiled with clang:
clang --version
Apple clang version 11.0.3 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin19.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Full install log under details below.
I have also a real g++ compiler installed via homebrew.
g++-9 (Homebrew GCC 9.2.0_3) 9.2.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

As opposed to the g++ of clang (on macOS):
g++ --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin19.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

> install.packages("xgboost")

  There is a binary version available but the source version is later:
          binary  source needs_compilation
xgboost <IP_ADDRESS> <IP_ADDRESS>              TRUE

Do you want to install from sources the package which needs compilation? (Yes/no/cancel) Yes
installing the source package ‘xgboost’

trying URL 'https://cran.rstudio.com/src/contrib/xgboost_<IP_ADDRESS>.tar.gz'
Content type 'application/x-gzip' length 898104 bytes (877 KB)
==================================================
downloaded 877 KB

* installing *source* package ‘xgboost’ ...
** package ‘xgboost’ successfully unpacked and MD5 sums checked
** using staged installation
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking Backtrace lib...
checking for backtrace in -lexecinfo... no
checking endian...
checking whether OpenMP will work in a package... yes
configure: creating ./config.status
config.status: creating src/Makevars
** libs
Makevars:17: -DXGBOOST_STRICT_R_MODE=1
Makevars:17: -DDMLC_LOG_BEFORE_THROW=0
Makevars:17: -DDMLC_ENABLE_STD_THREAD=1
Makevars:17: -DDMLC_DISABLE_STDIN=1
Makevars:17: -DDMLC_LOG_CUSTOMIZE=1
Makevars:17: -DXGBOOST_CUSTOMIZE_LOGGER=1
Makevars:17: -DRABIT_CUSTOMIZE_MSG_
Makevars:17: -DRABIT_STRICT_CXX98_
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I./include -I./dmlc-core/include -I./rabit/include -I. -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include -Xclang -fopenmp -DDMLC_CMAKE_LITTLE_ENDIAN=1 -pthread -fPIC  -Wall -g -O2  -c xgboost_R.cc -o xgboost_R.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I./include -I./dmlc-core/include -I./rabit/include -I. -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include -Xclang -fopenmp -DDMLC_CMAKE_LITTLE_ENDIAN=1 -pthread -fPIC  -Wall -g -O2  -c xgboost_custom.cc -o xgboost_custom.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I./include -I./dmlc-core/include -I./rabit/include -I. -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c xgboost_assert.c -o xgboost_assert.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I./include -I./dmlc-core/include -I./rabit/include -I. -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c init.c -o init.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I./include -I./dmlc-core/include -I./rabit/include -I. -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include -Xclang -fopenmp -DDMLC_CMAKE_LITTLE_ENDIAN=1 -pthread -fPIC  -Wall -g -O2  -c amalgamation/xgboost-all0.cc -o amalgamation/xgboost-all0.o
In file included from amalgamation/xgboost-all0.cc:33:
In file included from amalgamation/../src/data/data.cc:13:
In file included from amalgamation/../src/data/sparse_page_writer.h:20:
./dmlc-core/include/dmlc/concurrency.h:36:22: warning: braces around scalar initializer [-Wbraced-scalar-init]
  Spinlock() : lock_(ATOMIC_FLAG_INIT) {
                     ^~~~~~~~~~~~~~~~
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/atomic:2437:26: note: expanded from macro 'ATOMIC_FLAG_INIT'
#define ATOMIC_FLAG_INIT {false}
                         ^~~~~~~
In file included from amalgamation/xgboost-all0.cc:38:
amalgamation/../src/data/ellpack_page_source.cc:33:11: warning: variable 'page' is uninitialized when used here [-Wuninitialized]
  return *page;
          ^~~~
amalgamation/../src/data/ellpack_page_source.cc:32:20: note: initialize the variable 'page' to silence this warning
  EllpackPage* page;
                   ^
                    = nullptr
amalgamation/../src/data/ellpack_page_source.cc:40:11: warning: variable 'page' is uninitialized when used here [-Wuninitialized]
  return *page;
          ^~~~
amalgamation/../src/data/ellpack_page_source.cc:39:20: note: initialize the variable 'page' to silence this warning
  EllpackPage* page;
                   ^
                    = nullptr
In file included from amalgamation/xgboost-all0.cc:75:
amalgamation/../src/common/io.cc:102:8: warning: unused variable 'ReadErr' [-Wunused-variable]
  auto ReadErr = [&fname]() {
       ^
In file included from amalgamation/xgboost-all0.cc:13:
In file included from amalgamation/../src/metric/metric.cc:6:
In file included from ./dmlc-core/include/dmlc/./registry.h:9:
In file included from /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/map:479:
In file included from /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/__tree:15:
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2338:5: warning: delete called on non-final 'xgboost::tree::QuantileHistMaker::Builder' that has virtual functions but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor]
    delete __ptr;
    ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2651:7: note: in instantiation of member function 'std::__1::default_delete<xgboost::tree::QuantileHistMaker::Builder>::operator()' requested here
      __ptr_.second()(__tmp);
      ^
amalgamation/../src/tree/updater_quantile_hist.cc:74:14: note: in instantiation of member function 'std::__1::unique_ptr<xgboost::tree::QuantileHistMaker::Builder, std::__1::default_delete<xgboost::tree::QuantileHistMaker::Builder> >::reset' requested here
    builder_.reset(new Builder(
             ^
5 warnings generated.
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I./include -I./dmlc-core/include -I./rabit/include -I. -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include -Xclang -fopenmp -DDMLC_CMAKE_LITTLE_ENDIAN=1 -pthread -fPIC  -Wall -g -O2  -c amalgamation/dmlc-minimum0.cc -o amalgamation/dmlc-minimum0.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I./include -I./dmlc-core/include -I./rabit/include -I. -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include -Xclang -fopenmp -DDMLC_CMAKE_LITTLE_ENDIAN=1 -pthread -fPIC  -Wall -g -O2  -c rabit/src/engine_empty.cc -o rabit/src/engine_empty.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I./include -I./dmlc-core/include -I./rabit/include -I. -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include -Xclang -fopenmp -DDMLC_CMAKE_LITTLE_ENDIAN=1 -pthread -fPIC  -Wall -g -O2  -c rabit/src/c_api.cc -o rabit/src/c_api.o
clang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o xgboost.so ./xgboost_R.o ./xgboost_custom.o ./xgboost_assert.o ./init.o ./amalgamation/xgboost-all0.o ./amalgamation/dmlc-minimum0.o ./rabit/src/engine_empty.o ./rabit/src/c_api.o -Xclang -fopenmp /usr/local/lib/libomp.dylib -DDMLC_CMAKE_LITTLE_ENDIAN=1 -pthread -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
installing to /Library/Frameworks/R.framework/Versions/3.6/Resources/library/00LOCK-xgboost/00new/xgboost/libs
** R
** data
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (xgboost)

OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked

This error message is quite helpful. The error message is probably because we recently enabled OpenMP (multithreading library) for XGBoost R package: #5240. Let me look at it.
An easy fix is to turn off multithreading, but that would lower training performance out of the box.

AFAIK, OpenMP is not available on OSX by default. It is possible to install it, e.g. via homebrew which I have not (yet) done. Scikit-learn therefore throws a warning and the compiles without OpenMP (at the moment, on my computer).

@lorentzenchr OpenMP is made available via the libomp package on Homebrew. You must have installed a package that depended on libomp.

I regularly compile scikit-learn. make in gives
python setup.py build_ext -i
Partial import of sklearn during the build process.
C compiler: gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g

compile options: '-c'
gcc: test_program.c
gcc objects/test_program.o -o test_program
C compiler: gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g

compile options: '-c'
extra options: '-fopenmp'
gcc: test_program.c
clang: error: unsupported option '-fopenmp'
/Users/lorentzen/github/scikit-learn/sklearn/_build_utils/openmp_helpers.py:112: UserWarning:

                ***********
                * WARNING *
                ***********

It seems that scikit-learn cannot be built with OpenMP.

- Make sure you have followed the installation instructions:

    https://scikit-learn.org/dev/developers/advanced_installation.html

- If your compiler supports OpenMP but you still see this
  message, please submit a bug report at:

    https://github.com/scikit-learn/scikit-learn/issues

- The build will continue with OpenMP-based parallelism
  disabled. Note however that some estimators will run in
  sequential mode instead of leveraging thread-based
  parallelism.

@lorentzenchr With libomp, the compiler flag is -Xclang -fopenmp, not -fopenmp. Take a look at the build log you uploaded, and you can find -Xclang -fopenmp.

@hcho3 Thanks for clarifying. Though, this is a bit above my skills.
I can confirm that I have indeed installed libomp via homebrew, contrary to what I've written before.

This error message is quite helpful. The error message is probably because we recently enabled OpenMP (multithreading library) for XGBoost R package:

In LightGBM we've recently faced the same duplicate OpenMP installation issue with R downloaded from CRAN. Here is the workaround we use:
https://github.com/microsoft/LightGBM/blob/bf99a07221d95d51a125b5cfae64908ed8d2c1e6/.ci/test_r_package.sh#L43-L56

@lorentzenchr Can you try installing this tarball?
xgboost_<IP_ADDRESS>.tar.gz This might solve the issue with the duplicated libomp.dylib.

@hcho3 That works. Thanks a lot!

@lorentzenchr That's good news! Is multithreading working in your machine?

Yes, nthread=4 works as expected.

@lorentzenchr Thanks to your help, I filed the fix https://github.com/dmlc/xgboost/pull/5701. Thanks a lot!

Hi everyone:
We will be submitting 1.1.0 version to CRAN within a week or two. It has the fix for the crashing bug (#5701). For now, you can try it out by downloading xgboost_<IP_ADDRESS>.tar.gz and running
R CMD INSTALL xgboost_<IP_ADDRESS>.tar.gz

<IP_ADDRESS> is released.

I'm late to the party. But I've got libomp installed via brew, but I still get the following:

checking whether OpenMP will work in a package... no

WARNING: OpenMP is unavailable on this Mac OSX system. Training speed may be suboptimal.
To use all CPU cores for training jobs, you should install OpenMP by running
         brew install libomp

Interestingly, if I install via cmake manually directing to brew-installed llvm, it installs successfully. But the first time I try to use XGBoost to train a mode, I get a segfault error.
I've also only ever managed to get OpenMP to work with XGBoost in R for at most 3 months. Something always breaks after a few days of struggling. My guess is there is a Catalina update or R package update that is breaking some symlink deep in the system.

Just FYI, completely removing brew and rebuilding everything in brew from from scratch fixed this.


ENH: optimize._differentiate: numerical differentiation of elementwise functions

Reference issue
gh-2035
gh-7457
gh-17059
What does this implement/fix?
This adds a private function for numerically estimating derivatives of black-box, real, univariate, scalar-valued functions. A public interface and extension to complex, multivariate, vector-valued functions is left to future work, if desired, but the immediate use is gh-15928 (fitting and testing).
I have been discussing this in https://github.com/mdhaber/jacobi/pull/1 with @HDembinski, author of jacobi and gh-17059. The purpose of this PR is to get the framework in place, and we can extend it together in future work if desired.
Features:

Adaptive step size to achieve target accuracy; returns an error estimate
Vectorized. Only one function call is made per iteration (and few iterations are needed).
Easily extended to alternative arrays (e.g. GPU arrays)

There are plenty of areas for improvement documented in the code. For instance, with little modification, we could use np.polyfit like jacobi does to approximate the derivative and the error; this would be more robust to inaccurate function evaluations. Alternatively, I've seen that high-order Richardson extrapolation from second-order formulae is numerically favorable to the high-order formulae I'm using here. (I switched to this approach from Richardson extrapolation to simplify the code and more easily accommodate one-sided stencils.) I anticipate comments about these points, but this does a reasonably good job and is theoretically straightforward, so let's start with the simple thing and improve from there.
Additional information
Here is a performance comparison with Jacobi when differentiating the CDFs of stats distributions at a point near the median, using the PDF as a reference value. The "execution time" is measured in seconds, "true error" is the relative error compared to the PDF, and "error ratio" is the ratio of the error estimate to the true error.

With central differences, rtol=1e-12, and otherwise default settings:

With forward differences, rtol=1e-12, and otherwise default settings:

The same plots when x is an array of 1000 points sampled from the distribution:
Central differences:

Forward differences:

One notable thing is that this function's error estimates are conservative with central differences, but I think I have an easy way to improve that.

CI on last commit was green except for lint issue fixed in 60addb1 and an unrelated error fixed in gh-18819.

@mdhaber I think it's fine to xfail the failing saddlepoint case and we can just treat it as something to address in a followup.

Actually it's not that. I need to merge main and update an input validation test.

W00t! _scalar_optimization_compress_work and CI is green. Thanks for the review!

There were some concerns in https://github.com/scipy/scipy/issues/17059#issuecomment-2062797122 about _differentiate and the jacobian and hessian drafts in gh-20063.

I think there's a misunderstanding. jacobian would be for $\mathbf{R}^n \rightarrow \mathbf{R}^m$ functions.

ok. The misunderstanding arose from reading the differentiate docstring.
"func(x)[i] must equal func(x[i]) for all indices i"
There's no way that this can be the case for R^n -> R^m where x.shape == (n,).

Sure. The documentation of jacobian isn't written yet, but I don't think we can expect the documentation of _differentiate to apply exactly to jacobian. This brings up a relevant point, though. The "elementwise" functions were originally stricter about their definition of "elementwise". When this requirement was relaxed, @steppi and I agreed to leave the documentation alone while they are private (https://github.com/scipy/scipy/pull/19545#pullrequestreview-1854831300). We will need to revisit this part when we make functions public.

I had a look at the https://github.com/scipy/scipy/pull/20063 and compared the jacobian function given there against approx_derivative for a simple example. The output from jacobian is...an order of magnitude slower.

I'm actually surprised it's not worse than that. Starting with a default step size of 0.5, it takes a few iterations to get an accurate answer and form an error estimate. This won't be as fast as a fixed-step algorithm.

The output from jacobian is less accurate.

Here's some code:

import numpy as np
from scipy.optimize._numdiff import approx_derivative
from scipy.optimize._differentiate import _differentiate

rng = np.random.default_rng(234823492323)

def jacobian(f, x):
    int_dtype = np.issubdtype(x.dtype, np.integer)
    x0 = np.asarray(x, dtype=float) if int_dtype else np.asarray(x)
    m = len(x0)
    i = np.arange(m)

    def wrapped(x):
        n = () if x.ndim == x0.ndim else (x.shape[-1],)  # number of abscissae
        new_dims = (1,) if x.ndim == x0.ndim else (1, -1)
        new_shape = (m, m) + x0.shape[1:] + n
        xph = np.expand_dims(x0, new_dims)
        xph = np.broadcast_to(xph, new_shape).copy()
        xph[i, i] = x
        return f(xph)
    res = _differentiate(wrapped, x, preserve_shape=True)

    return res

def f(x):
    return np.array([np.sin(x[0]), np.cos(x[1]), np.sin(x[1]) * np.cos(x[2])])

def df(x):
    return np.array([[np.cos(x[0]), 0, 0],
                     [0, -np.sin(x[1]), 0],
                     [0, np.cos(x[1]) * np.cos(x[2]), -np.sin(x[1]) * np.sin(x[2])]])

x = rng.random(3)

res = jacobian(f, x)
res2 = approx_derivative(f, x)
ref = df(x)

print('jacobian error')
print(res.df - ref)
print()
print('approx_derivative error')
print(res2 - ref)

and the output:
jacobian
[[-9.65894031e-15 -8.70477479e-14 -8.70477479e-14]
 [ 2.01118444e-13  9.76996262e-15  2.01118444e-13]
 [-2.06182278e-13 -4.55191440e-15  7.43849426e-15]]

approx_derivative
[[-1.01466613e-11  0.00000000e+00  0.00000000e+00]
 [ 0.00000000e+00  6.00963723e-12  0.00000000e+00]
 [ 0.00000000e+00 -1.15374377e-12 -2.24353869e-12]]

It depends on how we assess "less accurate".
The true jacobian is:
array([[ 0.94788058,  0.        ,  0.        ],
       [ 0.        , -0.76512863,  0.        ],
       [ 0.        ,  0.4705355 , -0.52228332]])

So jacobian is a few orders of magnitude more accurate on the nonzero elements, and the maximum absolute error on the zero elements is 2.01118444e-13. And it tells you it's not confident about those zeros. The error estimate is:
array([[2.47424303e-12, 4.35238740e-14, 4.35238740e-14],
       [1.00559222e-13, 1.99640304e-12, 1.00559222e-13],
       [1.03091139e-13, 1.22873933e-12, 1.36191058e-12]])

If we relax the absolute tolerance (it's essentially zero by default because a nonzero absolute tolerance really needs to be set by the user), then jacobian converges much faster (~4x that of approx_derivative). And since it doesn't go to higher orders trying to get good relative accuracy on zero values ("thrashing"), the error of the zero elements gets better.
jacobian error
[[-9.65894031e-15 -3.40030265e-16 -3.40030265e-16]
 [ 7.85618923e-16  9.76996262e-15  7.85618923e-16]
 [-8.05399525e-16 -4.55191440e-15  7.43849426e-15]]

So I think it's doing its job here. To make it even better, we can add a simple anti-thrashing heuristic that stops iteration when the error estimate isn't improving.

Also, it's often not the case that func is vectorisable, allowance should be made for that.

Seems doable. Most things that can be done in a single vectorized call can be done with for loops. It's going the other way that can be hard.

Thanks for that more detailed explanation @mdhaber.
On a side note, there are a few disadvantages to the current implementation that uses an absolute initial step.
e.g. What about this simple example:
def f(x):
    return np.array([np.sin(x[0]), np.cos(x[1]), np.sin(x[1]) * np.log(x[2])])

x = np.array([0.5, 0.5, 0.1])

this will result in a NaN for df[2, 2]. This is because several of the step locations for x[2] are less than 0, which the log can't evaluate.
Using relative steps would've ensured that the log value wouldn't have gone out of bounds. For fixed point, non-iterative calculation (a la approx_derivative), it also ensures that the step is scaled to the problem correctly.
So for production code a relative step, falling back to a (smaller) absolute step is better.

I think I discussed this with @HDembinski in https://github.com/mdhaber/jacobi/pull/1 or a spin-off issue. IIUC, this problem happens to be avoided in this case, but it is not solved in general by relative steps (e.g. suppose the function were not np.log(x[2]) but np.log(x[2] - 0.0999). The reason I didn't choose a relative step by default is that I can't see any theoretical justification for it - it's just a heuristic that the step size "should" be larger for larger x. If we want to differentiate np.sin(x) at x = 1e10, there's no reason that the step size should be a multiple of x.
I think the better solution is one-sided differences, which are implemented. Right now it just fails (for the affected elements) when the function returns a NaN, but in principle it could resort to one-sided differences.

The general rule is to use relative steps, and only use absolute steps when the initial value is zero.
The reasoning is the following: floating point arithmetic does not care about the scale of your value. The relative accuracy of computing 1 + 1e6 is the same as 1e6 + 1e12. The round-off happens in the mantissa, not the exponent.
When we compute numerical derivatives, we want to use a step size dx which is small compared to the curvature of the function (which we don't know), but not so small that x + dx becomes equal to x. Since we don't know the curvature, the next best thing is to use dx = f x, with f = 1e-3 or so. With this we can at least make sure that dx never becomes too small for x + dx to become equal to x.

there are very reasonable counterexamples

Sure there are, every heuristic has counter-examples. But relative steps work in infinitely more cases than absolute steps, that's why relative steps are the default.


Source Rotation

Capacity to rotate sources along the X, Y, Z axis (individually or several at the same time) is required to match mesh rotations.

Initial implementation of source rotation for InitialField class.


Added environment variables plugin to automatically set/unset envvars when activating/deactivating a virtualenv

See the added docs for how the functionality works. I must admit though that I'm quite new to fish scripting / virtualfish so I may have done things the 'wrong' way. Please see this pull request as a proposal, I'll happily change code based on feedback. Thanks for making virtualfish!

Hi!
I'm so sorry I didn't notice this until now! You'll need to merge the upstream changes that have been made since into your branch, or rebase your branch on top of them, for me to be able to merge this now. Note that plugins go in the virtualfish/ subdirectory now rather than in the top level of the repo like they used to.
Without having actually run it yet, the only issue I can spot is this: functions with names starting with __vf_ are picked up as vf subcommands (e.g. __vf_new is called when you run vf new, plus it shows up in the autocomplete, help output, and so on), which it looks like is not what you want. My suggestion would be to name them starting with __vfsupport_ instead. (I should probably document this, sorry.)
Anyway, thanks for helping make virtualfish better, and once again sorry for missing your PR for so long.

Since this PR seems to have languished, and this seems like a genuinely useful feature that I want to use, would either of you object to me forking @shanx's repo, applying the required markups, and then submitting as a new PR?

Not at all, go ahead! I ended up ditching this approach in favor of https://pypi.python.org/pypi/envdir

I believe this has been rendered moot, since similar functionality was merged via #119.


Bump go version

@soltysh @bparees You guys know what's going on with Travis:
hack/../hack/common.sh: line 416: STI_GIT_MAJOR: unbound variable
Note that:
$("${git[@]}" describe "${STI_GIT_COMMIT}^{commit}" 2>/dev/null)
is returning:
fatal: No annotated tags can describe '8791e3edfa1c9d19d48855d8151cbdf132e759c4'

No, I peeked at it last night and saw that until about a week ago Travis
was working, and nothing obvious was committed since, but that's as far as
i dug. Will spend some more time on it... Obviously it's not failing
outside of Travis.
Ben Parees | OpenShift
On Nov 7, 2015 11:20, "Dan McPherson"<EMAIL_ADDRESS>wrote:

@soltysh https://github.com/soltysh @bparees
https://github.com/bparees You guys know what's going on with Travis:
hack/../hack/common.sh: line 416: STI_GIT_MAJOR: unbound variable
Note that:
$("${git[@]}" describe "${STI_GIT_COMMIT}^{commit}" 2>/dev/null)
is returning:
fatal: No annotated tags can describe
'8791e3edfa1c9d19d48855d8151cbdf132e759c4'
—
Reply to this email directly or view it on GitHub
https://github.com/openshift/source-to-image/pull/334#issuecomment-154719570
.

Let's merge this as is (LGTM), we'll tackle travis down the road.
[merge]

@soltysh we should fix the unbound variable asap

@mfojtik let us first have the build pieces in and I'll tackle the unbound variable afterwards.


The BindablePicker backgound is mess in Android.

As you see the background is very mess, like a lot of pieces from other place.
I test the Android Phone and Table on the simulator, all same. (UWP all fine)
There are some pictures:

Does this happen in the normal Xamarin.Forms picker?

@rid00z , Picker from Android has same problems in Xamarin.Forms

So we can assume this is a Xamarin.Forms issue?

@rid00z  I think so.  I report the bug : https://bugzilla.xamarin.com/show_bug.cgi?id=43656
Thank you.


Feature: Add Lucid Link cloud drive support

What feature or improvement do you think would benefit Files?
LucidLink is buried in Volumes. Why couldn't it be displayed the same as Explorer?

Requirements

This proposal will accomplish X
This proposal will accomplish Y
This proposal will accomplish Z

Files Version
<IP_ADDRESS>
Windows Version
10.0.22631.3085
Comments
No response

Thanks for the feedback, I've added this to the project board.

Wow, that was fast. Thanks!

This documentation might be helpful https://support.lucidlink.com/hc/en-us/articles/4416950355085-Where-is-my-LucidLink-folder-and-data

I read the document but I'm thinking from a typical user's point of view not an enterprise or sysadmin scenario.
LucidLink is used by many remote video editors. They are not IT experts. They just want access to their files and Explorer does that with no extra setup (I didn't have to use a CLI). Having that appear in Files would be a good thing too.

@Colin-VideoRevealed that comment was for whoever decides to implement this, sorry for the confusion.


Upgrading tf to 2.4.0

Have you tried our tests with tf2.4? I just tried and I get the following failure
AttributeError: module 'tensorflow' has no attribute 'python'

because we are using oom_error = tf.python.framework.errors_impl.ResourceExhaustedError in some exceptions in order to raise our own OOM error. If this is the only issue, I guess we could just find what is the corresponding error for tf2.4 and update accordingly, but we should do it in a way that does not break compatibility with older tf versions.
Note that pip still installs tf2.3.1 by default (unless we explicitly do pip install tensorflow==2.4.0rc1), which means that CI is not testing for tf2.4. We may want to update that as well.

Yes, sorry for not being more explicit, this should be a draft PR to fix this https://github.com/Quantum-TII/qibo/network/alert/requirements.txt/tensorflow/open. As soon as pypi uploads the final tf release we can go ahead and fix/test the changes.

Yes, sorry for not being more explicit, this should be a draft PR to fix this https://github.com/Quantum-TII/qibo/network/alert/requirements.txt/tensorflow/open. As soon as pypi uploads the final tf release we can go ahead and fix/test the changes.

@stavros11 this should be ready to go.

@stavros11 I have checked using anaconda with python38, tf2.4.0 cuda11. Tests are passing on GPU. I will retry on other systems and then merge.

I confirm that tests are passing even with python3.9 and cuda 11.1.


Add structured metadata for event

Attempt to get search engines to show the South Dakota Code Camp as an
event in search results by including Google-recommended JSON-LD event
metadata in the head.

@bryanburgers could you make this request to the primary repo?


:robot: Add bochs.vm

Automated PR from https://github.com/mandiant/VM-Packages/issues/897 :rocket:

I have tested the package locally and it does not work. Closing and documenting what needs to be done in #899


Bump minimum required ruby version to 3.0

See #1612
Closes #1614
I would suggest against yanking the affected version. Doing so will result in installs for those that already upgraded not working anymore so you get another disruption for free. It's been a month since that release and it would have been better to do this as early as possible instead.

Doh! Thank you for catching this.


TypeError: load_inference_model() missing 1 required positional argument: 'path_prefix'

TypeError: load_inference_model() missing 1 required positional argument: 'path_prefix'

同问题

同问题

/home/aistudio/PaddleGAN/applications
You are using Paddle compiled with TensorRT, but TensorRT dynamic library is not found. Ignore this if TensorRT is not needed.
Model EDVR proccess start..
Traceback (most recent call last):
File "tools/video-enhance.py", line 113, in
predictor = EDVRPredictor(args.output, weight_path=args.EDVR_weight)
File "/home/aistudio/PaddleGAN/ppgan/apps/edvr_predictor.py", line 153, in init
self.build_inference_model()
File "/home/aistudio/PaddleGAN/ppgan/apps/base_predictor.py", line 43, in build_inference_model
params_filename=param_file)
TypeError: load_inference_model() missing 1 required positional argument: 'path_prefix'

EDVR DAIN same issue

/home/aistudio/PaddleGAN/applications
You are using Paddle compiled with TensorRT, but TensorRT dynamic library is not found. Ignore this if TensorRT is not needed.
Model EDVR proccess start..
Traceback (most recent call last):
File "tools/video-enhance.py", line 113, in
predictor = EDVRPredictor(args.output, weight_path=args.EDVR_weight)
File "/home/aistudio/PaddleGAN/ppgan/apps/edvr_predictor.py", line 153, in init
self.build_inference_model()
File "/home/aistudio/PaddleGAN/ppgan/apps/base_predictor.py", line 43, in build_inference_model
params_filename=param_file)
TypeError: load_inference_model() missing 1 required positional argument: 'path_prefix'

EDVR DAIN same issue

32G的机器没问题，重新启动一下环境，换台机器就好了

32G的机器没问题，重新启动一下环境，换台机器就好了


1.6 rc2, fc30, rootless: warnings about missing crun

(Similar to, but not the same as, #4012)
podman-1.6.0-0.2.rc2.fc30 rootless outputs copious warnings by default:
$ podman images
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
REPOSITORY                   TAG       IMAGE ID       CREATED        SIZE
....

This is fc30, not fc31. I made the warnings go away by hand-editing ~/.config/containers/libpod.conf but am concerned that this is going to come back and haunt us; in particular, regular users will end up doing the same (removing runc from config) and then ending up with nonworking podman when they upgrade to fc31.

This repros as rootless, but not as root. I'm really confused as to why default log level is not the same between the two.

We explicitly set ERROR level logging by default for Podman, unconditionally, unless overridden by CLI.

$ podman info --log-level=error
WARN[0000] cannot find mappings for user mcs: No subuid ranges found for user "mcs" in /etc/subuid
WARN[0000] using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding subids

Well there's our problem. logrus as rootless is ignoring log-level=error. Maybe something about how we exec a new process?

I think it's definitely the re-exec part of rootless.

Nice. Fixing this bug may then fix #4012 and get gating tests to pass on rawhide.

@giuseppe Poke - where does rootless re-exec happen? I'm trying to trace the code, but getting lost in the make pause process vs join pause process bit.

Alright, we seem to be passing literal argv into the reexec'd process... So I don't see how we're losing the log level.

Does not seem directly related to reexec... I can get it to happen both inside and outside the rootless user namespace.


Using WarningsLibrary to query for warnings

Pull Request Description
Fixes #6616 by using WarningsLibrary.
Checklist
Please ensure that the following checklist has been satisfied before submitting the PR:

[x] All code follows the
Scala,
All code has been tested:

[x] Unit tests updated.

I hope the performance won't degrade by using uncached library? I guess we don't have any other option though.

This is not PE code. We have to use uncached version or we would need to do a more complicated rewrite.

There is a failure in RuntimeServerTest - should the expected message just be updated, @4e6?

Failure in RuntimeVisualizationsTest fixed in 224ae5c4e1


Add secp256k1 keyTmplate and primitive support

currently secp256k1 is not supported in Tink. Add support for it (in aries-framework-go)

@Baha-sk FYI both secp256k1 and ES256K have been added to the IANA JWA registry as JSON Web Key Elliptic Curve and signing algorithm respectively.

@Baha-sk FYI both secp256k1 and ES256K have just been added to the IANA JWA registry as JSON Web Key Elliptic Curve and signing algorithm respectively.

It's good to know @llorllale , it is not going to be supported natively in go for a while though, unless someone is currently working on updating the standard library.


Date(Time) of versions is not globalized

Describe the bug
The detail page for a package does not use a globalized representation as requested by the browser.
To Reproduce

open a detail page
look at the Versions section in the Last Updated column

Expected behavior
A globalized representation as requested by the browser.
Screenshots
hosted as docker container

running local

Additional context
It seems as the docker hosted version does not check the request header.
I added the following code to the Package.cshtml file.
@using System.Globalization;

    CurrentCulture: @System.Globalization.CultureInfo.CurrentCulture

    AcceptLanguage Header: @HttpContext.Request.Headers.AcceptLanguage.ToString();

    Last updated (ToShortDateString): @Model.Versions[0].LastUpdated.ToShortDateString();

Did you set the wished timezone to the Docker container as environment variable like so
docker run -p 5000:8080 -e TZ=Europe/Berlin bagetter/bagetter

or does this never work in Docker?

I did not know about this feature....
But that also does not fix the issue.

Unfortunately, even with the new Dockerfile it looks like this for me:
CurrentCulture:

AcceptLanguage Header: de,en-GB;q=0.7,en;q=0.3;

Last updated (ToShortDateString): 02/12/2024;

Putting something like this in the request pipeline works correctly in my small test, but I'm not entirely sure what else is affected by that and also it feels like the framework should already do something like this for us:
    app.Use(async (context, next) =>
    {
        var preferredLanguage = context.Request.Headers["Accept-Language"].ToString().Split(',').FirstOrDefault();
        if (!string.IsNullOrEmpty(preferredLanguage))
        {
            var culture = new System.Globalization.CultureInfo(preferredLanguage);
            System.Globalization.CultureInfo.CurrentCulture = culture;
        }

        await next.Invoke();
    });

As per #169 I'm seeing this as I'm expecting dd/mm/yyyy.


Feature Request: Generic Policy Example

Request for a very generic example

by specifying a JMSEPath (or similar) type query

and expected query value.

E.g along similar lines to the older KE scanner examples here where a key and value are used
https://github.com/forseti-security/forseti-security/blob/v2.14.0/rules/ke_scanner_rules.yaml

This is a good idea, I just added PR #76 which has very basic support for something like this.  I have some thoughts on how to take it much further.  For what you are asking for it might be better served as a plugin or PR directly to OPA.  However, I assume performance might be an issue.

@t12g @ddremund  @morgante I created a basic POC for a more generic constraint template
https://github.com/abombss/policy-library/tree/feature/generic-template
Its inspired by the Cloud Custodian generic filter syntax.
It's not feature complete yet, but I think it could be a very powerful base template for people to quickly develop constraints and iterate on them without having to dive deep into rego and opa.  Modeling one of the new bq location constraints looks like this.
https://github.com/abombss/policy-library/blob/a8c6bcd8aa914dd60ff3b95a4770a6806e32fd72/validator/test/fixtures/generic_constraint/constraints/bq_dataset_location/allowlist_one_exemption/data.yaml#L15-L36
Here is another sample as well.
https://github.com/abombss/policy-library/blob/7a586e33a4d82bf412162432ffe4e8ba1fa43fe5/validator/test/fixtures/generic_constraint/constraints/subnets/data.yaml#L15-L54
Before I take this further I would like to open the discussion to get feedback and ideas.  There are certainly some challenges with doing this in native opa and rego.  Currently there is no support for recursion so we need to hard code the levels of nesting for the logical operators.  I currently stopped at 3 levels but we could do something more reasonable like 8.  Projections are also a weak spot.  Today I only support very direct keys "resource.data.labels.foo", dealing with collections and aggregations will be tricky.
Assuming we like this approach and move forward, I would consider closing #76 in favor of this since its far more powerful.
Status

[ ] Design

[ ] Syntax
[ ] Rego Rule Model
[ ] Projections
[ ] Performance

[ ] Value Filter

[x] Comparison Operators

[x] equal
[x] not-equal
[x] greater-than
[x] gte
[x] less-than
[x] lte

[ ] Other Operators

[x] absent
[x] present
[x] not-null
[x] null
[ ] empty
[ ] contains

[x] Logic Operators

[x] or
[x] and
[x] not

[ ] List Operators

[ ] in
[ ] not-in
[ ] intersect

[ ] Special Operators

[ ] glob
[x] regex
[x] regex-case

[ ] Transformations
[ ] Value Regex

[ ] Age Filter

I have some concerns about the testability of this approach. Looking at the example, the conditional logic is moved from template to constraint. I think we'd still want to test it (essentially shifting the testing focus from templates to constraints). We can reuse the existing unit test framework, but that still requires some understanding of rego and test setup. Does that diminish the usefulness of this generic policy? Can we simplify/streamline the test setup for each generic policy constraint?

@t12g  That is a valid concern and something we should probably have a solution for regardless of this template.  I think we can spin up a simple little framework so end users can write tests for their constraints by just supplying a json file of the assets and the expected names of assets for violations.
While this does shift the burden of logic from the template to the constraint, another advantage is the transparency and readability of the constraint.  Currently templates are black boxes and unless you are a rego expert you won't understand what it is actually doing.  There is no way I could show an auditor or one of my IS people the rego and expect them to understand, I would have a lot better luck walking them through the simple boolean filters.  In both instances test cases validating the functionality of the constraint and not just the template would be important.
The only other thought I had to reduce the overhead and learning curve of adding constraints was creating a SQL like syntax and then doing rego codegen for the template and constraint.  I am not sure which approach would be better.


Peering behavior - baseline

Description
As a Besu dev, I want to establish a baseline for peering behavior so that when we make changes to peering code, we know whether we are making it better.
Acceptance Criteria
eg Release 22.1.2 is a good opportunity to observe the behaviour of new nodes as they start up - eg how long it takes a new besu node to sync goerli. - make a doc to paint a picture of what normal looks like for each testnet/mainnet. Screenshots if helpful
Example of what may be helpful to track:

Time to sync from scratch (where appropriate)
Time for an existing node to catch up
Time to gain N peers

This log means the P2P agent started
2022-03-29 08:31:27.174+10:00 | vert.x-eventloop-thread-2 | INFO | PeerDiscoveryAgent | P2P peer discovery agent started and listening on /0:0:0:0:0:0:0:0:30303
And then once sufficient useful peers are obtained, full sync starts with something like this
2022-03-29 09:08:26.831+10:00 | nioEventLoopGroup-3-3 | INFO | SyncTargetManager | Found common ancestor with peer Peer 0x0b7150c1adbaec3bc6... at block 0
fast sync is a bit different
once it has (minimum) 5 peers
2022-03-29 09:22:10.702+10:00 | EthScheduler-Timer-0 | INFO  | FastSyncActions | Selecting block number 5062554 as fast sync pivot block.
Then it waits for 5 confirmations of pivot block which can take a while and obviously depends on quality of peers
2022-03-29 09:37:16.549+10:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Confirmed pivot block at 5062554: 0xd4e574268148d0544f7a74402fe80dde2cfc8cf2ca5901f60d373eeb243b18d5
then can download world state
2022-03-29 09:37:16.558+10:00 | nioEventLoopGroup-3-5 | INFO  | FastWorldStateDownloader | Begin downloading world state from peers for block 5062554 (0xd4e574268148d0544f7a74402fe80dde2cfc8cf2ca5901f60d373eeb243b18d5). State root 0x7ca542145e6c33e8aea6142133d1d195fb02ef38a63ff3450a8011a00ef2c953

Is the "time to sync from scratch" a good thing to measure, since on a network like goerli, the number of blocks is always increasing, the amount of time expected will also increase - it's a moving target.

If we focus on Fast sync as our use case, I understand that we have 3 checkpoints :

P2P peer discovery agent start
Selecting fast sync pivot block
Confirming pivot block (the same as step 2)

Do you agree with this ?
I noticed the same thing regarding the number of blocks, I think the difference is negligible regarding the total number of blocks, but still important to mention it. For my first baseline on Fast Sync, I display the number of imports (each import = 200 blocks) to keep an eye on it

Yes and the third step "Confirming the pivot block" will be highly dependent on gaining "good" peers quickly

I noticed two behaviours during baseline calculation, either peering is very fast (less than 12 s), either peering is much slower (> 5 minutes). I guess this related with the quality of peers.

This calculation is done on Goerli fast sync nodes, started with the same parameters on i3.xlarge AWS instance (ohio), besu version 22.1.1

@ahamlat how can we generate these numbers - eg if we spin up a new node off a branch to test some new code?

@macfarla for the moment, this is done manually by using logs timestamps but it can be automated. I use this kind of requests to display all the timestamps : ("Selecting block number" or  "Confirmed pivot block" or "P2P peer discovery agent started and listening")

I think it's useful to look at the time elapsed between the 'Selecting block number xx as fast sync pivot block' so we get a better understanding of how long a complete pivot download takes


add environment.yml which actually makes it work

Is this project still maintained?   Not sure what an envorionment.yml file is for though.

environment.yml is like requirements.txt, but for conda environments (a more general virtualenv, very popular in scientific python).

For virtualenvironments, I think you can just enter and pip install.  Can you say what functionality this enables and for which OSes, distributions, etc?

I can't say it adds a crucial functionality, in conda you can pip install the requirements now if you want to. I did this mainly because the requirements file was un-maintained when I hacked with it so I shared back how to make it work :)
Closing the PR


Ported Banner to Java.

This port compared well to the FreeBASIC compiler output.

More Java, yes! Appreciate it! 👏


Maistra 2.4 sec

Updates from upstream

@twghu replaced by https://github.com/maistra/envoy/pull/266 , safe to close?

Replaced by https://github.com/maistra/envoy/pull/266


Html3D: move to its own repository

Html3D component need its own repository.
This way people can import it in their own R3R project.

Done.
See react-three-renderer-html3d package on github.


Received order from JLCPCB boards not trimmed

Just received my order from JLCPCB but it seems as though they didnt remove the excess material from the top and the bottom of the boards.. Its "scored" on both sides if you would call it that but not removed. pretty firm still wont easily snap off ive never dealt with this before what is recommended? I could exert more force and use a vice OR use a box cutter to further score to remove.. what would folks recommend?

Hi, you can break it with your hands. That part of pcb is needed for the PCB Assembly. All the assembled boards have it.


feat: Setup InfiniteScroll callbacks for Sqllab result sets

SUMMARY

Setup initial callback to allow us to add infinite scroll functionality later on. Right now we currently still loading the entire payload into the browser, but only rendering rows that the user scrolls over with rowIdx variable. This variable is updated whenever the user crosses over the initial 100 row boundary.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

/testenv up

This feels much better than before!
@hughhhh were you going to update the progress bar as well to be a Superset spinner?

This feels much better than before!
@hughhhh were you going to update the progress bar as well to be a Superset spinner?

I shipped the progress bar into a separate PR

closing for now

@hughhhh this is great work... did we merge the front end pagination improvements elsewhere?


Activity links

Type of issue
Missing information
Description
It would be nice to have a ballpark figure of how many links is considered "too many". Docs mention a big batch of work - see below - is it hundreds of activities? thousands? tens of thousands? etc
void DoBatchWork(ActivityContext[] requestContexts)
{
    // Assume each context in requestContexts encodes the trace-id that was sent with a request
    using(Activity activity = s_source.StartActivity(name: "BigBatchOfWork",
                                                     kind: ActivityKind.Internal,
                                                     parentContext: default,
                                                     links: requestContexts.Select(ctx => new ActivityLink(ctx))
    {
        // do the batch of work here
    }
}

Page URL
https://learn.microsoft.com/en-us/dotnet/core/diagnostics/distributed-tracing-instrumentation-walkthroughs
Content source URL
https://github.com/dotnet/docs/blob/main/docs/core/diagnostics/distributed-tracing-instrumentation-walkthroughs.md
Document Version Independent Id
dc4689df-b03a-e744-dbed-f23d8defcb6e
Article author
@tommcdon
Metadata

ID: 3f30f3b0-90cb-d2eb-ee08-4319ffcb7686
Service: dotnet-fundamentals

@tarekgh @noahfalk

According to the specs the recommended limit is 128. https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#span-limits.
@rcollina are you interested to submit a doc PR for that?

@tarekgh I'd love to! what are the steps?

@rcollina thanks for willing to help.
Simply clone the repo https://github.com/dotnet/docs/. In the cloned copy create a branch, edit the file docs/core/diagnostics/distributed-tracing-instrumentation-walkthroughs.md and push your created branch. Open a PR request to push your branch to the dotnet main branch. Let me know if this is not clear so I can send more details as needed.

@rcollina Remember that you first have to fork this repository, and only after that clone your fork of this repo and push your created branch to your fork. You won't be able to push any branches to clone of this repo without write permissions.

Thanks @BartoszKlonowski. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork may be a useful link to explore.

Thank you all, I'll get to it as soon as I can.


Unable to add a New step after two or more parallel branches.

Describe the Bug with repro steps
In the new designer I'm unable to add a new step after I have two or more parallel branches used.
I need to switch back to the legacy designer to add a new step (to bring the branches together).
What type of Logic App Is this happening in?
Consumption (Portal)
Which operating system are you using?
Windows
Are you using new designer or old designer
New Designer
Did you refer to the TSG before filing this issue? https://aka.ms/lauxtsg
Yes
Workflow JSON
No response
Screenshots or Videos

Browser
Microsoft Edge
Additional context
No response

This is by design. The right way to do this would be to add an action after a single path and then go to run after settings and add the other paths there. The reason for this is because while your example is balanced and simple, there were many many cases where this isn't the case and the + New Step button would be placed in an extremely confusing location that didn't properly show which paths would be combined. The new method takes a few extra steps but it's more explicit and more likely to be "correct".


Design: OADP non-admin use cases

Context:
The work here should be build a  CRD's and controllers that would intercept all backup, restore, bsl creation and check if the user has the correct permissions to do the action.
New CRD's

OADP/velero non-admin Backup, superset of https://velero.io/docs/v1.11/api-types/backup/

includes user info
anything else?

OADP/velero non-admin Restore, superset of https://velero.io/docs/v1.11/api-types/restore/

includes user info
anything else?

OADP / velero non-admin BSL? https://velero.io/docs/v1.11/api-types/backupstoragelocation/
user info
OADP / velero non-admin VSL? https://velero.io/docs/v1.11/api-types/volumesnapshotlocation/

user info

Controller:

All non-admin cr's are routed to new controller, user info taken.

Checks permission for user A to do Action X
For example User A wants to backup namespace foo.  If user A has view/edit access to namespace foo, create the velero backup CR and send to velero.
If the user does NOT have permission, send an error back on the creation of the OADP/Velero non-admin cr.

Spec:

[ ] Create / Delete backups by non-admin
[ ] Create / Delete restores by non-admin
[ ] Create BSL by non-admin
[ ] Provide Status ( but not logs )?

Related work:
https://issues.redhat.com/browse/OADP-203
https://github.com/weshayutin/oadp-operator/tree/tekton-non-admin/docs/tekton-oadp-nonadmin
https://github.com/shawn-hurley/prototype-non-admin

We should discuss more about what kind of user info do we need ?

Should we make use of admission webhooks for backup/Restore CR creations ?
Will there be a different BSL per user or a shared one ? How will this scenario work ?
What if there is a change in during the Backup/Restore operation, how do we handle this scenario ?

I would suggest that a potential to consider here is that a new CR that is specifically for this purpose is used. Something like NamespacedBackup, or something similar, that when a user has access to that CR they are able to backup the Namespace. this will allow you to create a NamespaceRestore that must reference the NamespacedBackup as a local reference.
I think that some things to be very concerned with, is that Velero is single thread AFAIK and I think it needs to be a prequisite to make it so that it can backup multiple namespaces at a time. If this is not fixed, then no amount of good secure implementation here is going to matter IMO because it will be unusable.

@shubham-pampattiwar Answes as I see them:

We should discuss more about what kind of user info do we need ?

if you use the webhook, you can do the SAR requests to the objects that you want to make sure the user has access to. The user info that you get in the webhook is all you need to ask these questions of the RBAC system.

I would think that we make it clear that a user who has access to this resource in this namespace basically gets access to all the resources in the namespace, similar to admin. One thing that we need to make sure of, is if a role binding is bound to a cluster role, we need to either downgrade to a role, OR we need to assume that something else can restore that. and that we don't back that up and that you can't restore it. From a user/non-admin restore operation.

Should we make use of admission webhooks for backup/Restore CR creations ?

Yeah I agree, this needs to be there to at least do some permissions, if you don't create new CR's and attempt to re-use the existing ones.

@shubham-pampattiwar Answes as I see them:

We should discuss more about what kind of user info we need ?

if you use the webhook, you can do the SAR requests to the objects that you want to make sure the user has access to. The user info that you get in the webhook is all you need to ask these questions of the RBAC system.

I would think that we make it clear that a user who has access to this resource in this namespace gets access to all the resources in the namespace, similar to admin. One thing that we need to make sure of is if a role binding is bound to a cluster role, we need to either downgrade to a role OR we need to assume that something else can restore that. and that we don't back that up and that you can't restore it. From a user/non-admin restore operation.

I agree that access is namespace based and it's 100% access to the namespace or no access.

Should we make use of admission webhooks for backup/Restore CR creations ?

Yeah, I agree; this needs to be there to at least do some permissions if you don't create new CRs and attempt to re-use the existing ones.

I'm not sure if you are referring to dynamic admission control, I think you are.   If the admission control is scoped to only check api requests for velero than we might be ok.   A large customer already nacked using dynamic admission control via policy, the context here is a little different I think.   Needs discussion.

Will there be a different BSL per user or a shared one ? How will this scenario work ?

I think that a Namespaced Backup can only use a BSl that is defined in the namespace of the backup, OR we create a ClusterBackupStorageLocation, that an admin can define for everyone.

Let's talk about this one, I think we want a non-admin to create their own bsl and only have access to bsl's they create. Not 100% sure, needs discussion.

What if there is a change during the Backup/Restore operation? How do we handle this scenario ?

I need more context, but do you mean like the RBAC system? I think this is an interesting use case, I think that if they are removed they would be unable to use the objects anymore so I don't think there is a escalation concern but we should make sure.

more context..  but supposing.
A backup is running and permission to the namespace is removed, it would be too late to stop the backup.
what other scenarios are you thinking about here?

I hope that this helps, I have thought about this in the past, and we are having similar concerns that we had with OLM, where some stuff is just ClusterScoped by nature and tried to make it non-admin.

A backup is running and permission to the namespace is removed, it would be too late to stop the backup.
what other scenarios are you thinking about here?```

If that does happen things should be fine, because the user would be unable to use the backup in anyone once finished right? And they wouldn't be able to restore etc. I think this should be fine and there is nothing to do IMO.

If there is another scenario that someone can come up with though would be interested in hearing about it.

A large customer already nacked using dynamic admission control via policy, the context here is a little different I think. Needs discussion.

I don't understand what this means or what they nacked, maybe we sould have a seperate discussion on this.

discussion has moved to https://hackmd.io/qbYw8wq4RZSWEe2KZpzLaw
Once we have solid design written out in the hackmd we'll copy it back to this issue.


e2e fix (#4707)

Pull request opened by the merge tool on behalf of #4707

Merged into branches iso5 in 9fc9b150c2123ba1efce83b2886f7d6cf281adb5


Fix inaccurate docs for Commands::spawn_empty

Objective
Commands::spawn_empty docs say that it queues a command to spawn an entity, but it doesn't. It immediately spawns an Entity, which is possible because Entities::reserve_entity() takes &self and no components are added yet.
Solution
Fix docs.

I'm not sure if "immediately spawns ..." is accurate to how the ECS works internally, but it's definitely not queueing a command. Open to better wording suggestions.

It immediately reserves the entity with Entities::reserve_entity() but the entity only gets "created" in the next call to World::flush_entities() which is usually the next time commands are applied as it requires &mut World.
There are no literal commands though.

Adjusted wording to better reflect that.


Italian language leader

Hi,
I'm Antonio and I'm an Italian Mechatronic Engineer and I love home assistant. I use it for everything since 4 years in two different homes. It would be really amazing to contribute in the community.
Thank you,
Antonio

Thanks @jjantoine. Current Italian language leaders are @auanasgheps and @xraver. Please coordinate with them here: https://github.com/home-assistant/intents/discussions/370


fix: correct condition to downweigh track-feature packages

We use track features to de-prioritize packages.
Sometimes, the solver is forced to select a given package, e.g. when selecting numpy-XY-<hash> it might have a dependency on pypy. Numpy variants themselves are not down-prioritized, though.
If we detect that a package forces only dependencies that have track features (such as the case for the numpy variant that only works together with pypy) we also "inherit" the down-weighing to numpy (otherwise the solver would be forced to use pypy later on).
We already had the code for this but it was missing a ! – this PR corrects that.

So everything worked fine if there is only a single package with the track feature. I had to change the repodata to contain 2 builds of
    "python_abi-3.9-2_cp39.tar.bz2": {
      "build": "2_cp39",
      "build_number": 2,
      "constrains": [
        "pypy <0a0"
      ],
      "depends": [
        "python 3.9.*"
      ],
      "license": "BSD-3-Clause",
      "license_family": "BSD",
      "md5": "39adde4247484de2bb4000122fdcf665",
      "name": "python_abi",
      "sha256": "67231829ea0101fee30c68f788fdba40a11bbee8fdac556daaab5832bd27bf3d",
      "size": 4017,
      "subdir": "linux-64",
      "timestamp":<PHONE_NUMBER>037,
      "version": "3.9"
    },

And that way I could reproduce the issue + make sure the fix works by adding snapshots.


Moved gedcomdiff HTML elements to html package

This is so gedcomdiff functionality can be used outside of the binary.

This change is

Codecov Report

Merging #195 into master will increase coverage by 2.84%.
The diff coverage is 70.96%.

@@            Coverage Diff             @@
##           master     #195      +/-   ##
==========================================
+ Coverage    56.6%   59.44%   +2.84%
==========================================
  Files         170      170
  Lines        5083     5084       +1
==========================================
+ Hits         2877     3022     +145
+ Misses       2131     1982     -149
- Partials       75       80       +5

Impacted Files
Coverage Δ

html/text.go
100% <ø> (ø)
:arrow_up:

gedcomdiff/main.go
0% <0%> (ø)
:arrow_up:

html/diff_row.go
73.52% <100%> (ø)

html/individual_compare.go
51.57% <56.25%> (ø)

html/diff_page.go
84.09% <87.5%> (ø)

html/link.go
100% <0%> (+41.66%)
:arrow_up:

html/google_analytics.go
42.85% <0%> (+42.85%)
:arrow_up:

html/individual_name.go
60% <0%> (+60%)
:arrow_up:

html/individual_dates.go
71.42% <0%> (+71.42%)
:arrow_up:

html/event_date.go
75% <0%> (+75%)
:arrow_up:

... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3177035...2227694. Read the comment docs.


MDTZ-1571: Resolve the issue with an inconsistent AtlassianDesign.displayName truncation

Changes

fix: Resolves the issue with AtlassianDesign with a not truncated displayName being returned from the /entities/associateEntity endpoint (and causing the failure in the upstream service). Truncates AtlassianDesign.displayName on the entry points to the application to ensure a consistent truncation among use cases.
fix: Resolves the issue with AtlassianDesign.displayName being truncates based on bytes instead of UTF-16 code units (as expected by Jira).
feat: Improves the truncate utility method to exclude lone surrogates that can appear due to the truncation (e.g., when "🐱", which contains 2 code units, to 1).

TIL the term lone surrogate. Nice find @akostevich-atlassian!
Also apologies @elainefigma for missing this originally, I suspected the upstream errors were due to encoding issues but this wasn't the case.