🐛 json schema for noLabelWithoutControl incorrectly requires inputComponents labelAttributes labelComponents

Environment information
CLI:
  Version:                      1.9.2
  Color support:                true

Platform:
  CPU Architecture:             aarch64
  OS:                           macos

Environment:
  BIOME_LOG_PATH:               unset
  BIOME_LOG_PREFIX_NAME:        unset
  BIOME_CONFIG_PATH:            unset
  NO_COLOR:                     unset
  TERM:                         "xterm-256color"
  JS_RUNTIME_VERSION:           "v22.6.0"
  JS_RUNTIME_NAME:              "node"
  NODE_PACKAGE_MANAGER:         "npm/10.8.2"

Biome Configuration:
  Status:                       Loaded successfully
  Formatter disabled:           false
  Linter disabled:              false
  Organize imports disabled:    false
  VCS disabled:                 true

Workspace:
  Open Documents:               0

What happened?

We configured inputComponents for our components
IDE says labelAttributes and labelComponents are required attributes, but biome cli correctly parses the configuration JSON so I think this is false-positive

Expected result
I think labelAttributes and labelComponents should not be marked as required in json schema, if biome cli accepts json without them
Code of Conduct

[X] I agree to follow Biome's Code of Conduct

This is similar to #3763 and there might be several similar. Should I create issue for each time I found or send new ones as comment for some existing issue?

This is similar to #3763 and there might be several similar other places. Should I create issue for each time I found or send new ones as comment for some existing issue?

Yes please. Every option is different


Open With Workspace…

Add an option to the File menu, with a key command, to open a document with a specific workspace. Use teh standard open file sheet and have a pop up button with the available workspace names. Maybe even remember the most recently used workspace and have that preselected.

key command: shift-control-command-o


Unable to fetch metrics

Hello.
I have a metric in Prometheus:
nginx_ingress_controller_requests{controller_class="nginx",
  controller_namespace="default",
  controller_pod="my-nginx-nginx-ingress-controller-5b858c69c9-z4sns",
  endpoint="metrics",
  exported_namespace="default",
  ingress="hello-world-ingress",
  instance="<IP_ADDRESS>:10254",
  job="my-nginx-nginx-ingress-controller-metrics",
  namespace="default",
  pod="my-nginx-nginx-ingress-controller-5b858c69c9-z4sns",
  service="my-nginx-nginx-ingress-controller-metrics",
  status="200"}

Created an adapter rule:
- seriesQuery: 'nginx_ingress_controller_requests{status="200"}'
  resources:
    overrides:
      ingress: {resource: "ingress"}
  name:
    matches: "^(.*)"
    as: "${1}_per_second"
  metricsQuery: 'rate(<<.Series>>{<<.LabelMatchers>>}[5m]) by (<<.GroupBy>>)'

And I even can see it appearing in APIs:
$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "ingresses.extensions/nginx_ingress_controller_requests_per_second",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

But I (and HPA too) can't fetch metrics from API
$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespace/default/ingresses.extensions/*/nginx_ingress_controller_requests_per_second" | jq .
Error from server (NotFound): the server could not find the requested resource
$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespace/default/ingresses.extensions/hello-world-ingress/nginx_ingress_controller_requests_per_second" | jq .
Error from server (NotFound): the server could not find the requested resource
$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/ingresses.extensions/hello-world-ingress/nginx_ingress_controller_requests_per_second" | jq .
Error from server (InternalError): Internal error occurred: unable to fetch metrics

Maybe I just don't understand something. Because when I enable default metrics in the adapter, I can see nginx_ingress_controller_requests metrics and able to couple it with HPA.
P.S. Sorry, I didn't find any usergroup or main list, so I'm writing here.

I have a similar issue except with istio_request metrics. Everything looks good until I try to pull metrics from API, they don't exist.
rules:
  custom:
  - seriesQuery: '{__name__=~"^istio_request_.*"}'
    resources:
      template: <<.Resource>>
    name:
      matches: ^istio_request_(.*)$
      as: ""
    metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)

You'll want a resource for namespace, too.  That's probably the issue.  If you have a namespaced resource, you need to have mappings for both namespace and the resource (and ingress is namespaced).

@danisabad, were you able to solve this problem?
@DirectXMan12,

You'll want a resource for namespace, too. That's probably the issue. If you have a namespaced resource, you need to have mappings for both namespace and the resource (and ingress is namespaced).

How about this configuration? See the mapping. Nginx prometheus metrics has exported_namespace, exported_service and ingress for mapping.
    - seriesQuery: '{controller_class="nginx",ingress!="",exported_namespace!="",exported_service!=""}'
      seriesFilters:
        - is: ^nginx_ingress_controller_request_duration_seconds_bucket$
      resources:
        overrides:
          ingress:
            resource: ingress
          exported_namespace:
            resource: namespace
          exported_service:
            resource: service
      name:
        as: nginx_ingress_controller_request_duration_seconds_rate_99_pct
      metricsQuery: 'histogram_quantile(0.99, sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (le, ingress))'

This is the example nginx ingress controller prometheus metrics:
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="ingress-controller-nginx-ingress-controller-64fd4f454d-hkh5c",endpoint="10254",exported_namespace="production",exported_service="foo-api",host="t.fstrk.net",ingress="foo-api-ingress",instance="<IP_ADDRESS>:10254",job="ingress-controller-nginx-ingress-controller",le="0.5",method="GET",namespace="nginx-ingress",path="/api",pod="ingress-controller-nginx-ingress-controller-64fd4f454d-hkh5c",service="ingress-controller-nginx-ingress-controller",status="503"}

nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="ingress-controller-nginx-ingress-controller-64fd4f454d-hkh5c",endpoint="10254",exported_namespace="production",exported_service="foo-api",host="t.fstrk.net",ingress="foo-api-ingress",instance="<IP_ADDRESS>:10254",job="ingress-controller-nginx-ingress-controller",le="0.5",method="GET",namespace="nginx-ingress",path="/api",pod="ingress-controller-nginx-ingress-controller-64fd4f454d-hkh5c",service="ingress-controller-nginx-ingress-controller",status="499"}

Testing the metrics:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/foo/services/production/nginx_ingress_controller_request_duration_seconds_rate_99_pct" | jq '.'

gives nothing in the metrics:
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/services/%2A/nginx_ingress_controller_request_duration_seconds_rate_99_pct"
  },
  "items": []
}

I will appreciate if you can enlighten us what we have been doing wrong. Thank you!

Any update on this issue. I am facing exactly the same problem.
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq {   "kind": "APIResourceList",   "apiVersion": "v1",   "groupVersion": "custom.metrics.k8s.io/v1beta1",   "resources": [     {       "name": "services/service_response_time",       "singularName": "",       "namespaced": true,       "kind": "MetricValueList",       "verbs": [         "get"       ]     }   ] }
But when I run the following command, I get error resource not found.
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespace/default/service/*/service_response_time | jq Error from server (NotFound): the server could not find the requested resource

getting the same issue here, any chance of getting some docs on how to install this? using namespaces should hardly be treated as an edge case?

Faced with the same issue, I'm trying to set custom metric
- metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,reporter="destination"}[1m]))
      name:
        as: http_requests
      resources:
        overrides:
          destination_service_name:
            resource: service
          destination_workload_namespace:
            resource: namespace
      seriesQuery: istio_requests_total

I get the following log:
I0708 14:35:28.203020       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (283.178µs) 200 [Go-http-client/2.0 <IP_ADDRESS>:27608]
I0708 14:35:28.203048       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (235.767µs) 200 [Go-http-client/2.0 <IP_ADDRESS>:27608]
I0708 14:35:28.203020       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (242.222µs) 200 [Go-http-client/2.0 <IP_ADDRESS>:27608]
I0708 14:35:28.203025       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (208.974µs) 200 [Go-http-client/2.0 <IP_ADDRESS>:27608]
I0708 14:35:28.203028       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (273.148µs) 200 [Go-http-client/2.0 <IP_ADDRESS>:27608]
I0708 14:35:28.631951       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1/namespaces/devops/services/k8s-workshop-service/http_requests: (1.992389ms) 404 [kube-controller-manager/v1.15.11 (linux/amd64) kubernetes/d94a81c/system:serviceaccount:kube-system:horizontal-pod-autoscaler <IP_ADDRESS>:59492]

Does anyone come across and know how to solve it?

@danisabad, were you able to solve this problem?
@DirectXMan12,

You'll want a resource for namespace, too. That's probably the issue. If you have a namespaced resource, you need to have mappings for both namespace and the resource (and ingress is namespaced).

How about this configuration? See the mapping. Nginx prometheus metrics has exported_namespace, exported_service and ingress for mapping.
    - seriesQuery: '{controller_class="nginx",ingress!="",exported_namespace!="",exported_service!=""}'
      seriesFilters:
        - is: ^nginx_ingress_controller_request_duration_seconds_bucket$
      resources:
        overrides:
          ingress:
            resource: ingress
          exported_namespace:
            resource: namespace
          exported_service:
            resource: service
      name:
        as: nginx_ingress_controller_request_duration_seconds_rate_99_pct
      metricsQuery: 'histogram_quantile(0.99, sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (le, ingress))'

This is the example nginx ingress controller prometheus metrics:
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="ingress-controller-nginx-ingress-controller-64fd4f454d-hkh5c",endpoint="10254",exported_namespace="production",exported_service="foo-api",host="t.fstrk.net",ingress="foo-api-ingress",instance="<IP_ADDRESS>:10254",job="ingress-controller-nginx-ingress-controller",le="0.5",method="GET",namespace="nginx-ingress",path="/api",pod="ingress-controller-nginx-ingress-controller-64fd4f454d-hkh5c",service="ingress-controller-nginx-ingress-controller",status="503"}

nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="ingress-controller-nginx-ingress-controller-64fd4f454d-hkh5c",endpoint="10254",exported_namespace="production",exported_service="foo-api",host="t.fstrk.net",ingress="foo-api-ingress",instance="<IP_ADDRESS>:10254",job="ingress-controller-nginx-ingress-controller",le="0.5",method="GET",namespace="nginx-ingress",path="/api",pod="ingress-controller-nginx-ingress-controller-64fd4f454d-hkh5c",service="ingress-controller-nginx-ingress-controller",status="499"}

Testing the metrics:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/foo/services/production/nginx_ingress_controller_request_duration_seconds_rate_99_pct" | jq '.'

gives nothing in the metrics:
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/services/%2A/nginx_ingress_controller_request_duration_seconds_rate_99_pct"
  },
  "items": []
}

I will appreciate if you can enlighten us what we have been doing wrong. Thank you!

check promethus scrape rule ,make sure kubernetes_pod_name （or other name exported_service |exported_namespace ）exsit
 - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: kubernetes_pod_name
    replacement: $1
    action: replace

In my case I debugged as follows:
given the following custom rule:
- seriesQuery: '{__name__=~"^nginx_ingress_controller_requests.*",namespace!=""}'
      seriesFilters: []
      resources:
        template: <<.Resource>>
        overrides:
          exported_namespace:
            resource: "namespace"

I could see the custom metric listed
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq . | grep ingresses | grep nginx_ingress_controller_requests_rate

 => "name": "ingresses.networking.k8s.io/nginx_ingress_controller_requests_rate",

but the direct api call would fail
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/my-ing-namespace/ingress/my-ing-name/nginx_ingress_controller_requests_rate"

=> error not found

To debug I pasted the query into prometheus
{__name__=~"^nginx_ingress_controller_requests.*",namespace!=""}

=> nginx_ingress_controller_requests{controller_class="nginx",controller_namespace="lorem",controller_pod="lorem",ingress="lorem",instance="<IP_ADDRESS>:10254",job="ingress-nginx-endpoints",namespace="lorem",service="lorem,status="200"}

From the query results I could see there was no key "exported_namespace" to remap, it was simply called "namespace".
Changing the "overrides" section of my custom rule fixed the issue.

To clarify if anyone gets this error. I just had to the remove the overrides key entirely because nginx now uses just namespace instead of exported_namespace. Thank you @seocahill


Find and fix spelling and grammar errors

Find and fix spelling and/or grammar errors within the CSV file

I have a feature branch with these fixes, if you'd like to allow me to send you a PR, I can.


[ISSUE #5430] Remove redundant null check

What is the purpose of the change
Remove the null check for topicData in HookUtils. Fix the typo for variable tranformRes to transformRes.
Brief changelog
Remove redundant null check and fix typo.
Verifying this change
This pr does not change logic, it only contains small code improvements.

@fndejan you can see this docment Linking a pull request to an issue.  You can link a pull request or branch to an issue to show that a fix is in progress and to automatically close the issue when the pull request or branch is merged.

@mxsm it's updated now, thanks.

Codecov Report

Merging #5467 (fb6792c) into develop (dd95a5b) will decrease coverage by 0.01%.
The diff coverage is 30.59%.

@@              Coverage Diff              @@
##             develop    #5467      +/-   ##
=============================================
- Coverage      43.17%   43.15%   -0.02%
- Complexity      8050     8059       +9
=============================================
  Files           1021     1025       +4
  Lines          71999    72193     +194
  Branches        9537     9559      +22
=============================================
+ Hits           31082    31156      +74
- Misses         36985    37088     +103
- Partials        3932     3949      +17

Impacted Files
Coverage Δ

...a/org/apache/rocketmq/broker/BrokerController.java
43.55% <0.00%> (-0.22%)
:arrow_down:

...rocketmq/broker/metrics/BrokerMetricsConstant.java
0.00% <ø> (ø)

.../rocketmq/broker/metrics/NopLongUpDownCounter.java
0.00% <0.00%> (ø)

...ocketmq/broker/processor/PeekMessageProcessor.java
3.40% <0.00%> (-0.20%)
:arrow_down:

...he/rocketmq/broker/processor/PopReviveService.java
40.56% <0.00%> (-0.92%)
:arrow_down:

.../transaction/queue/TransactionalMessageBridge.java
69.51% <0.00%> (-2.71%)
:arrow_down:

...ava/org/apache/rocketmq/broker/util/HookUtils.java
12.84% <0.00%> (+0.11%)
:arrow_up:

...he/rocketmq/common/attribute/TopicMessageType.java
91.66% <0.00%> (-8.34%)
:arrow_down:

.../apache/rocketmq/common/utils/ServiceProvider.java
0.00% <0.00%> (ø)

...org/apache/rocketmq/store/DefaultMessageStore.java
53.62% <0.00%> (+0.04%)
:arrow_up:

... and 31 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


PermissionError: [WinError 5] Access is denied

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like this: [x])
Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.08.22. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2018.08.22

Before submitting an issue make sure you have:

[x] At least skimmed through the README, most notably the FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones
[x] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

D:\>youtube-dl -v "https://www.youtube.com/watch?t=4&v=BaW_jenozKc"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/watch?t=4&v=BaW_jenozKc']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2018.08.22
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.15063
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] BaW_jenozKc: Downloading webpage
[youtube] BaW_jenozKc: Downloading video info webpage
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'https://r11---sn-xupn5a5u-4wgez.googlevideo.com/videoplayback?sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cnh%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&expire=1535059889&key=yt6&initcwndbps=422500&mime=video%2Fmp4&ipbits=0&mt=1535038234&fvip=3&ratebypass=yes&itag=22&requiressl=yes&c=WEB&signature=6193B34D9F679D5F4AA3A58844773C8839105CEF.CB7E5BC3996CD6ED07F55DC5DB3CE00AA4B1C6F2&nh=EAE%2C&mn=sn-xupn5a5u-4wgez%2Csn-hgn7rn7y&mm=31%2C26&id=o-AFeHoPDkg_55CGNEykjtPpap7vHXss7sNPq9Gn-10CAl&mv=m&source=youtube&ms=au%2Conr&ei=UdN-W4adHM7JxwKcjInIAw&lmt=1472250137564567&ip=<IP_ADDRESS>&pl=22&dur=9.891'
[download] Destination: youtube-dl test video ''_ä↭𝕐-BaW_jenozKc.mp4
[download] 100.0% of 1.74MiB at 1001.49KiB/s ETA 00:00ERROR: unable to rename file: [WinError 5] Access is denied: "youtube-dl test video ''_ä↭𝕐-BaW_jenozKc.mp4.part" -> "youtube-dl test video ''_ä↭𝕐-BaW_jenozKc.mp4"
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpsjdqsmfs\build\youtube_dl\downloader\common.py", line 200, in try_rename
PermissionError: [WinError 5] Access is denied: "youtube-dl test video ''_ä↭𝕐-BaW_jenozKc.mp4.part" -> "youtube-dl test video ''_ä↭𝕐-BaW_jenozKc.mp4"

...
<end of log>

having all permissions on the download destination .same issue is whe downloading youtube playlist.
Please help

I am on windown 10.
is there anything in missing.
checked all required configurations to run youtube-dl . download is successful but renaming file is failed.

I am not able to find below file on my computer.
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpsjdqsmfs\build\youtube_dl\downloader\common.py",

Location C:\Users\dst is not present at all


My terminology

I think what I'm meaning to ask is how do I use Postal as my MDA and Using Mailwizz as the MUA.

Is this a stray message related to https://github.com/atech/postal/issues/655?


improper handling of ima signature and hashes

Is your an issue a feature request? If so, please raise it as an enhancement
Environment

OS / version: CentOS Stream 9
Processor architecture: x86_64
TPM Manufacturer: swtpm
Keylime version: current upstream 1268609...

Description
From https://github.com/RedHat-SP-Security/keylime-tests/pull/123 I am running the following test sequence
           /setup/configure_tpm_emulator
            /setup/install_upstream_keylime
            /setup/configure_kernel_ima_module/ima_policy_signing
            /functional/basic-attestation-on-localhost

in particular the setup is made using the tmt tool
tmt -c distro=centos-stream-9 run -vvv plan -n upstream prepare discover -h fmf -t /setup/configure_tpm_emulator -t /setup/install_upstream_keylime -t /setup/configure_kernel_ima_module/ima_policy_signing -t /functional/basic-attestation-on-localhost  provision -h virtual -i CentOS-Stream-9 -c system execute login finish

These tests pass. Please note that despite configuring a system with ima_sig template the test attestation scenario does not work with signatures on tis own as it doesn't pass a signature when adding an agent.
Now I install bash RPM along with its signatures.
yum -y install rpm-plugin-ima
yum -y reinstall bash

and I execute /functional/basic-attestation-on-localhost manually once more.
cd /var/tmp/tmt/run-*/plans/upstream-keylime-tests-github-ci/discover/default/tests/functional/basic-attestation-on-localhost/
./test/sh

This time the test fails. Also note that while files from bash RPM do have evm signatures installed these files are on the excludelist.
# cat excludelist.txt
/afs(/.*)?
/bin(/.*)?
/boot(/.*)?
/dev(/.*)?
/etc(/.*)?
/home(/.*)?
/lib(/.*)?
/lib64(/.*)?
/media(/.*)?
/mnt(/.*)?
/opt(/.*)?
/proc(/.*)?
/root(/.*)?
/run(/.*)?
/sbin(/.*)?
/srv(/.*)?
/sys(/.*)?
/tmp(/.*)?
/usr(/.*)?
/var(/.*)?
/sysroot/etc/fstab
/dracut-state.sh
/keylime-tests/basic-attestation-on-localhost-UvK3j(/.*)?

The error I see is after adding an agent, its status is Invalid Quote.
2022-06-14 05:42:53.364 - keylime.tenant - INFO - Agent Info:
{"d432fbb3-d2f1-4a97-9ef7-75bd81c00000": {"operational_state": "Invalid Quote", "v": "08hACNmWhrOiMZkzNVLUXkIt9XCfIM8HJ9Y4DA1Pkn8=", "ip": "<IP_ADDRESS>", "port": 9002, "tpm_policy": "{\"22\": [\"0000000000000000000000000000000000000001\", \"0000000000000000000000000000000000000000000000000000000000000001\", \"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001\", \"ffffffffffffffffffffffffffffffffffffffff\", \"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\", \"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\"], \"15\": [\"0000000000000000000000000000000000000000\", \"0000000000000000000000000000000000000000000000000000000000000000\", \"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\"], \"mask\": \"0x408400\"}", "meta_data": "{\"cert_serial\": 2, \"subject\": \"OU=53,O=MITLL,L=Lexington,ST=MA,CN=d432fbb3-d2f1-4a97-9ef7-75bd81c00000,C=US\"}", "allowlist_len": 6, "mb_refstate_len": 0, "accept_tpm_hash_algs": ["sha512", "sha384", "sha256", "sha1"], "accept_tpm_encryption_algs": ["ecc", "rsa"], "accept_tpm_signing_algs": ["ecschnorr", "rsassa"], "hash_alg": "sha256", "enc_alg": "rsa", "sign_alg": "rsassa", "verifier_id": "default", "verifier_ip": "<IP_ADDRESS>", "verifier_port": 8881, "severity_level": 6, "last_event_id": "ima.validation.ima_hash"}}

Multiple revocation notifications are being sent, adding first two below
{"msg": "{\"type\": \"revocation\", \"ip\": \"<IP_ADDRESS>\", \"agent_id\": \"d432fbb3-d2f1-4a97-9ef7-75bd81c00000\", \"port\": 9002, \"tpm_policy\": \"{\\\"22\\\": [\\\"0000000000000000000000000000000000000001\\\", \\\"0000000000000000000000000000000000000000000000000000000000000001\\\", \\\"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001\\\", \\\"ffffffffffffffffffffffffffffffffffffffff\\\", \\\"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\\\", \\\"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\\\"], \\\"15\\\": [\\\"0000000000000000000000000000000000000000\\\", \\\"0000000000000000000000000000000000000000000000000000000000000000\\\", \\\"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\\\"], \\\"mask\\\": \\\"0x408400\\\"}\", \"meta_data\": \"{\\\"cert_serial\\\": 2, \\\"subject\\\": \\\"OU=53,O=MITLL,L=Lexington,ST=MA,CN=d432fbb3-d2f1-4a97-9ef7-75bd81c00000,C=US\\\"}\", \"event_time\": \"Tue Jun 14 05:41:49 2022\", \"event_id\": \"ima.validation.ima_hash\", \"severity_label\": \"emergency\", \"context\": \"{\\\"message\\\": \\\"IMA template hash does not match the calculated hash.\\\", \\\"expected\\\": \\\"b'\\\\\\\\xffo\\\\\\\\xe6\\\\\\\\xbaL\\\\\\\\xa4\\\\\\\\x10{>\\\\\\\\xd1t\\\\\\\\xe0}\\\\\\\\x80\\\\\\\\xf7\\\\\\\\xba\\\\\\\\xf8\\\\\\\\xd5w\\\\\\\\xee'\\\", \\\"got\\\": \\\"b'(\\\\\\\\x00\\\\\\\\x00\\\\\\\\x00sha256:\\\\\\\\x004,i2E\\\\\\\\x0b\\\\\\\\xbb\\\\\\\\x1aG\\\\\\\\x86O0\\\\\\\\xbe~S\\\\\\\\xc2-ha\\\\\\\\x83\\\\\\\\x8d>\\\\\\\\xfc\\\\\\\\x8a\\\\\\\\xbb\\\\\\\\xbb\\\\\\\\xbaG\\\\\\\\xdc\\\\\\\\xa5\\\\\\\\x80j\\\\\\\\xef\\\\\\\\x00\\\\\\\\x00\\\\\\\\x00/usr/bin/bash 0302047c435bb5670030650230425145d71be6888dad039c721970e56cf55968acb3ce1ef9d9fb523c622911b89a4814f464d6b9f02a7c29f4b4e7bf4d023100fc3e6433d05aab8690625c9cab99341fbdebb7ff42869b630ff71b695b6b8078f82499408f52ae94ecc42624f3ad1cbe\\\\\\\\x00\\\\\\\\x00\\\\\\\\x00\\\\\\\\x00\\\\\\\\x00'\\\"}\"}", "signature": "w+w1F9xrDEuEQ+rM+j5VewQEfAXZScfBDX+95XMH5lViKDZJl0/Ql6aHhRwO/iW1EARU51rxD3jOrE0lxTwwTGgYhD8++AWegA0Gn4j6dDWfpgtogfcnlXQoxBtkk5S+JaAL+QrgiQle+6ahXzMs5g+6cgKxqFhXt+AtCeLeVeaXxXyQ0ur79fGPhwSsxjJeEYs8aLrWxJz+pv0YA5HrUnuOuBd/6tO1G63lGTVdln8Zo6jZW135tFJtiftbaom/j2nCwnRl/ZHOzf6NSy1i3bC7JWYdOxxsTyazm9T2c3dEUOO02vDmVwKWbfaHoVvnyhqYO3IoYxN6gwvu9E4xFw=="}POST / HTTP/1.1

{"msg": "{\"type\": \"revocation\", \"ip\": \"<IP_ADDRESS>\", \"agent_id\": \"d432fbb3-d2f1-4a97-9ef7-75bd81c00000\", \"port\": 9002, \"tpm_policy\": \"{\\\"22\\\": [\\\"0000000000000000000000000000000000000001\\\", \\\"0000000000000000000000000000000000000000000000000000000000000001\\\", \\\"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001\\\", \\\"ffffffffffffffffffffffffffffffffffffffff\\\", \\\"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\\\", \\\"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\\\"], \\\"15\\\": [\\\"0000000000000000000000000000000000000000\\\", \\\"0000000000000000000000000000000000000000000000000000000000000000\\\", \\\"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\\\"], \\\"mask\\\": \\\"0x408400\\\"}\", \"meta_data\": \"{\\\"cert_serial\\\": 2, \\\"subject\\\": \\\"OU=53,O=MITLL,L=Lexington,ST=MA,CN=d432fbb3-d2f1-4a97-9ef7-75bd81c00000,C=US\\\"}\", \"event_time\": \"Tue Jun 14 05:41:49 2022\", \"event_id\": \"ima.validation.ima_hash\", \"severity_label\": \"emergency\", \"context\": \"{\\\"message\\\": \\\"IMA template hash does not match the calculated hash.\\\", \\\"expected\\\": \\\"b'\\\\\\\\xffo\\\\\\\\xe6\\\\\\\\xbaL\\\\\\\\xa4\\\\\\\\x10{>\\\\\\\\xd1t\\\\\\\\xe0}\\\\\\\\x80\\\\\\\\xf7\\\\\\\\xba\\\\\\\\xf8\\\\\\\\xd5w\\\\\\\\xee'\\\", \\\"got\\\": \\\"b'(\\\\\\\\x00\\\\\\\\x00\\\\\\\\x00sha256:\\\\\\\\x004,i2E\\\\\\\\x0b\\\\\\\\xbb\\\\\\\\x1aG\\\\\\\\x86O0\\\\\\\\xbe~S\\\\\\\\xc2-ha\\\\\\\\x83\\\\\\\\x8d>\\\\\\\\xfc\\\\\\\\x8a\\\\\\\\xbb\\\\\\\\xbb\\\\\\\\xbaG\\\\\\\\xdc\\\\\\\\xa5\\\\\\\\x80j\\\\\\\\xef\\\\\\\\x00\\\\\\\\x00\\\\\\\\x00/usr/bin/bash 0302047c435bb5670030650230425145d71be6888dad039c721970e56cf55968acb3ce1ef9d9fb523c622911b89a4814f464d6b9f02a7c29f4b4e7bf4d023100fc3e6433d05aab8690625c9cab99341fbdebb7ff42869b630ff71b695b6b8078f82499408f52ae94ecc42624f3ad1cbe\\\\\\\\x00\\\\\\\\x00\\\\\\\\x00\\\\\\\\x00\\\\\\\\x00'\\\"}\"}", "signature": "w+w1F9xrDEuEQ+rM+j5VewQEfAXZScfBDX+95XMH5lViKDZJl0/Ql6aHhRwO/iW1EARU51rxD3jOrE0lxTwwTGgYhD8++AWegA0Gn4j6dDWfpgtogfcnlXQoxBtkk5S+JaAL+QrgiQle+6ahXzMs5g+6cgKxqFhXt+AtCeLeVeaXxXyQ0ur79fGPhwSsxjJeEYs8aLrWxJz+pv0YA5HrUnuOuBd/6tO1G63lGTVdln8Zo6jZW135tFJtiftbaom/j2nCwnRl/ZHOzf6NSy1i3bC7JWYdOxxsTyazm9T2c3dEUOO02vDmVwKWbfaHoVvnyhqYO3IoYxN6gwvu9E4xFw=="}

Expected behavior vs. actual behavior
The agent should pass attestation because even though a signature from /usr/bin/bash cannot be verified it matches the exclude list.
Steps to reproduce problem

described above

Relevant logs
verifier log
Jun 14 05:41:49 testcloud keylime_verifier[7837]: 2022-06-14 05:41:49.315 - keylime.tpm - INFO - TPM2-TOOLS Version: 5.0
Jun 14 05:41:49 testcloud keylime_verifier[7837]: 2022-06-14 05:41:49.325 - keylime.tpm - INFO - Checking IMA measurement list on agent: d432fbb3-d2f1-4a97-9ef7-75bd81c00000
Jun 14 05:41:49 testcloud keylime_verifier[7837]: 2022-06-14 05:41:49.339 - keylime.ima - ERROR - IMA ERRORS: Some entries couldn't be validated. Number of failures in modes: ImaSig 1.
Jun 14 05:41:49 testcloud keylime_verifier[7837]: 2022-06-14 05:41:49.380 - keylime.revocation_notifier - INFO - Sending revocation event via webhook...
Jun 14 05:41:49 testcloud keylime_verifier[7837]: 2022-06-14 05:41:49.391 - keylime.cloudverifier - WARNING - Agent d432fbb3-d2f1-4a97-9ef7-75bd81c00000 failed, stopping polling

ascii_runtime_measurements file can be downloaded using
curl -O https://transfer.sh/SS8Eay/ascii_runtime_measurements.txt

I found now root cause of this issue.
Signatures are distributed in CentOS 9 Stream the following way:

Signatures are added to the RPM header (e.g. with rpmsign)
rpm-plugin-ima takes those signatures, converts them to the IMA format
signatures are attached to the file using the security.ima extended file attribute

Keylime does the following with a signature:

Parses the header
validates that the length is correct
validates the signature against the provided keyring

If the parsing of the signature fails we currently assume that we had whitespace in the path and and just add it to the filename.
The issue is that the RPM plugin did not specify the actual size of the signature, instead they just put in the maximal length. This caused our parser to fail. Now our path is now wrong because the it includes also the signature and therefore the template_hash sanity check fails.
More details and the fix is already in the RPM repo and can be found here: https://github.com/rpm-software-management/rpm/commit/07f1d3132f0c7b7ecb69a47a9930edb534a9250e

Edit: It seems to not be this bug exactly, but it is something related to the rpm plugin.

Actually it doesn't seem to be related to the IMA plugin; it just calls into RPM to get the signatures in binary format and stores them into the extended file attributes.

@dmnks thanks for taking a look at this issue.

Actually it doesn't seem to be related to the IMA plugin; it just calls into RPM to get the signatures in binary format and stores them into the extended file attributes.

Ok. Do you have an idea why the following signature then has the wrong length (26368 instead of 103) in the header?
0302047c435bb5670030650230425145d71be6888dad039c721970e56cf55968acb3ce1ef9d9fb523c622911b89a4814f464d6b9f02a7c29f4b4e7bf4d023100fc3e6433d05aab8690625c9cab99341fbdebb7ff42869b630ff71b695b6b8078f82499408f52ae94ecc42624f3ad1cbe

This is the signature for bash in from IMA log provided above.

Why don't you just get the length from the actual hex string? We do just that in RPM.

We do that and compare it against the length in the header as a sanity check.
https://github.com/keylime/keylime/blob/9a60e8218d21b7d178be9fb90bbd476ba2aa83ba/keylime/ima/ast.py#L101-L108

@dmnks thanks for taking a look at this issue.

Glad to help!

Ok. Do you have an idea why the following signature then has the wrong length (26368 instead of 103) in the header?

What do you mean by "has" here? Where is this length stored? Remember we don't store individual file signature lengths in the header.

We do that and compare it against the length in the header as a sanity check.
https://github.com/keylime/keylime/blob/9a60e8218d21b7d178be9fb90bbd476ba2aa83ba/keylime/ima/ast.py#L101-L108

I'm not familiar with the code base here, but where do you get the header data from exactly?

Looking closer at the surrounding code linked above, this really doesn't seem to have anything to do with RPM headers. I wonder if this term is used within the context of IMA? If so, that could be where the confusion comes from.

What do you mean by "has" here? Where is this length stored? Remember we don't store individual file signature lengths in the header.

I mean the IMA signature. IMA signatures have the following format
struct signature_v2_hdr {
	uint8_t type;		/* xattr type */
	uint8_t version;	/* signature format version */
	uint8_t	hash_algo;	/* Digest algorithm [enum hash_algo] */
	__be32 keyid;		/* IMA key identifier - not X509/PGP specific */
	__be16 sig_size;	/* signature size */
	uint8_t sig[];		/* signature payload */
};

See: https://elixir.bootlin.com/linux/latest/source/security/integrity/integrity.h#L126
The first five struct members are what I mean with "header". This header includes the signature length. We extract that length and compare it against the bytes that we got from the IMA log. In the provided example we expect 103, but sig_size contains 26368.

Looking closer at the surrounding code linked above, this really doesn't seem to have anything to do with RPM headers. I wonder if this term is used within the context of IMA? If so, that could be where the confusion comes from.

Yes I meant the IMA signature header, not the RPM one. Sorry for the confusion.

Oh, that explains it, thanks! I wasn't aware of the internal structure of these signatures, but of course they'd contain some kind of a header structure!
So this indeed looks like the signature size is computed incorrectly at sign time before it's stored in the corresponding RPM tag RPMTAG_FILESIGNATURES. That leads to the signFile() function in RPM where call into sign_hash() defined in imaevm.h, however it's really impossible to say (without running this through a debugger) whether the bug lies within our code or libimaemv's, I'm afraid.

If it was a bug in RPM's signing routine, it would have to be some undefined use of the sign_hash() API function, e.g. by passing it wrong parameters. I'd risk to say that this is more likely to be an issue within libimaevm.

Do you see this with other packages than just bash?

Ok. Do you have an idea why the following signature then has the wrong length (26368 instead of 103) in the header?

This size mismatch is also quite suspicious, it almost seems like there's some uninitialized variable in use.

I just tried to reproduce this on a c9s system with ima-inspect on the /usr/bin/bash file from bash-5.1.8-4.el9.x86_64 but the signatures seemed totally fine:
$ ima_inspect /usr/bin/bash
/usr/bin/bash

security.ima
------------
digital signature version 2
digest algorithm: sha256
key-id v2 (gpg compatible): 7c435bb5
signature length: 824 bits
signature data:

3065023100fd10f9 a5f8e6a700c1e119 a7cd4279a11eaa8e 37a9712ff491f3aa 74e60da9fbf8fe6b 02ecaae0bca257a9 8d078e8c63023064 3acb687d6a59e9bd
9a1196278dae017a a7e88338e4eccda1 26a2321df86c0424 9e26953093f1f871 6dae570c6e8a22

security.evm
------------
no such attribute

OK, I can reproduce something with bash-5.1.8-2.el9.x86_64:
$ ima_inspect /usr/bin/bash
/usr/bin/bash

security.ima
------------
digital signature version 2
RuntimeError: premature end of attribute data (sig)

security.evm
------------
no such attribute

I can confirm that. For some reason CI systems use older image with bash-5.1.8-2.el9. After updating to -4 things seems to work well. So hopefully these wrong signatures were present only for older builds.
I am sorry for the noise. Probably we can close this issue.

Good to know, thanks!

I'll close this now. If we see this again in the future, now we know where to search.


WASM build and testing

Summary
We want the game building with WASM so we can play it as a browser game as the Bevy Jam says exposure increases dramatically if it can be played from a browser.
Output

[ ] Agree on game resolution (1280x720 ?)
[ ] Set game resolution using window manager
[ ] Test WASM Build
[ ] Provide test page and documentation for testing in browser

@lepsalex provided wireframe UI in 1024x768. Will move forward with that resolution for now.


Presence not showing up on ubuntu

Stacktrace
[atom-discord ERROR] Error: ENOENT: no such file or directory, stat '/home/syed/.atom/atom-discord/customize.json'
showError @ /home/syed/Desktop/code/repos/atom-discord/src/index.js:68

The error above only occurs on the first start after installing the package.
After refreshing atom, It creates the customize.json file in ~/.atom/atom-discord/ and the error goes away.
But still, the presence doesn't work.
customize.json:
{"projects":{},"filetypes":{}}

Information
Platform: Ubuntu 20.04
Atom: 1.48.0
Package Version: 2.0.2

same


enhancement - generative model sample code / gan zoo

to foster community involvement - some richer sample code beyond MNIST should be tackled.
Generative Adversarial Networks is a hot topic amongst ML - and some sample code using swift should help encourage researchers to pick up swift / a list of GAN projects attached below.
thinking out loud - consider google swift team - could have a budget with bounties that community could vote on and award to spur development ahead.
python reference -
https://github.com/hwalsuklee/tensorflow-generative-model-collections
https://github.com/hindupuravinash/the-gan-zoo
3D-ED-GAN - Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks
3D-GAN - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (github)
3D-IWGAN - Improved Adversarial Systems for 3D Object Generation and Reconstruction (github)
3D-RecGAN - 3D Object Reconstruction from a Single Depth View with Adversarial Learning (github)
ABC-GAN - ABC-GAN: Adaptive Blur and Control for improved training stability of Generative Adversarial Networks (github)
ABC-GAN - GANs for LIFE: Generative Adversarial Networks for Likelihood Free Inference
AC-GAN - Conditional Image Synthesis With Auxiliary Classifier GANs
acGAN - Face Aging With Conditional Generative Adversarial Networks
ACGAN - Coverless Information Hiding Based on Generative adversarial networks
ACtuAL - ACtuAL: Actor-Critic Under Adversarial Learning
AdaGAN - AdaGAN: Boosting Generative Models
AdvGAN - Generating adversarial examples with adversarial networks
AE-GAN - AE-GAN: adversarial eliminating with GAN
AEGAN - Learning Inverse Mapping by Autoencoder based Generative Adversarial Nets
AF-DCGAN - AF-DCGAN: Amplitude Feature Deep Convolutional GAN for Fingerprint Construction in Indoor Localization System
AffGAN - Amortised MAP Inference for Image Super-resolution
AL-CGAN - Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
ALI - Adversarially Learned Inference (github)
AlignGAN - AlignGAN: Learning to Align Cross-Domain Images with Conditional Generative Adversarial Networks
AM-GAN - Activation Maximization Generative Adversarial Nets
AmbientGAN - AmbientGAN: Generative models from lossy measurements (github)
AnoGAN - Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery
APE-GAN - APE-GAN: Adversarial Perturbation Elimination with GAN
ARAE - Adversarially Regularized Autoencoders for Generating Discrete Structures (github)
ARDA - Adversarial Representation Learning for Domain Adaptation
ARIGAN - ARIGAN: Synthetic Arabidopsis Plants using Generative Adversarial Network
ArtGAN - ArtGAN: Artwork Synthesis with Conditional Categorial GANs
ATA-GAN - Attention-Aware Generative Adversarial Networks (ATA-GANs)
Attention-GAN - Attention-GAN for Object Transfiguration in Wild Images
AttGAN - Arbitrary Facial Attribute Editing: Only Change What You Want (github)
AttnGAN - AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (github)
B-DCGAN - B-DCGAN:Evaluation of Binarized DCGAN for FPGA
b-GAN - Generative Adversarial Nets from a Density Ratio Estimation Perspective
BAGAN - BAGAN: Data Augmentation with Balancing GAN
Bayesian GAN - Deep and Hierarchical Implicit Models
Bayesian GAN - Bayesian GAN (github)
BCGAN - Bayesian Conditional Generative Adverserial Networks
BCGAN - Bidirectional Conditional Generative Adversarial networks
BEGAN - BEGAN: Boundary Equilibrium Generative Adversarial Networks
BGAN - Binary Generative Adversarial Networks for Image Retrieval (github)
BicycleGAN - Toward Multimodal Image-to-Image Translation (github)
BiGAN - Adversarial Feature Learning
BranchGAN - Branched Generative Adversarial Networks for Multi-Scale Image Manifold Learning
BS-GAN - Boundary-Seeking Generative Adversarial Networks
C-GAN - Face Aging with Contextual Generative Adversarial Nets
C-RNN-GAN - C-RNN-GAN: Continuous recurrent neural networks with adversarial training (github)
CA-GAN - Composition-aided Sketch-realistic Portrait Generation
CaloGAN - CaloGAN: Simulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networks (github)
CAN - CAN: Creative Adversarial Networks, Generating Art by Learning About Styles and Deviating from Style Norms
CapsuleGAN - CapsuleGAN: Generative Adversarial Capsule Network
CatGAN - Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
CatGAN - CatGAN: Coupled Adversarial Transfer for Domain Generation
CausalGAN - CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training
CC-GAN - Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks (github)
CDcGAN - Simultaneously Color-Depth Super-Resolution with Conditional Generative Adversarial Network
CFG-GAN - Composite Functional Gradient Learning of Generative Adversarial Models
CGAN - Conditional Generative Adversarial Nets
CGAN - Controllable Generative Adversarial Network
Chekhov GAN - An Online Learning Approach to Generative Adversarial Networks
CipherGAN - Unsupervised Cipher Cracking Using Discrete GANs
CM-GAN - CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning
CoAtt-GAN - Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning
CoGAN - Coupled Generative Adversarial Networks
ComboGAN - ComboGAN: Unrestrained Scalability for Image Domain Translation (github)
ConceptGAN - Learning Compositional Visual Concepts with Mutual Consistency
Conditional cycleGAN - Conditional CycleGAN for Attribute Guided Face Image Generation
constrast-GAN - Generative Semantic Manipulation with Contrasting GAN
Context-RNN-GAN - Contextual RNN-GANs for Abstract Reasoning Diagram Generation
CorrGAN - Correlated discrete data generation using adversarial training
Coulomb GAN - Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields
Cover-GAN - Generative Steganography with Kerckhoffs' Principle based on Generative Adversarial Networks
Cramèr GAN - The Cramer Distance as a Solution to Biased Wasserstein Gradients
Cross-GAN - Crossing Generative Adversarial Networks for Cross-View Person Re-identification
crVAE-GAN - Channel-Recurrent Variational Autoencoders
CS-GAN - Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets
CVAE-GAN - CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training
CycleGAN - Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (github)
D-GAN - Differential Generative Adversarial Networks: Synthesizing Non-linear Facial Variations with Limited Number of Training Data
D-WCGAN - I-vector Transformation Using Conditional Generative Adversarial Networks for Short Utterance Speaker Verification
D2GAN - Dual Discriminator Generative Adversarial Nets
D2IA-GAN - Tagging like Humans: Diverse and Distinct Image Annotation
DA-GAN - DA-GAN: Instance-level Image Translation by Deep Attention Generative Adversarial Networks (with Supplementary Materials)
DAGAN - Data Augmentation Generative Adversarial Networks
DAN - Distributional Adversarial Networks
DBLRGAN - Adversarial Spatio-Temporal Learning for Video Deblurring
DCGAN - Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (github)
DeblurGAN - DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks (github)
Defense-GAN - Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models
DeliGAN - DeLiGAN : Generative Adversarial Networks for Diverse and Limited Data (github)
DF-GAN - Learning Disentangling and Fusing Networks for Face Completion Under Structured Occlusions
DiscoGAN - Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
DistanceGAN - One-Sided Unsupervised Domain Mapping
DM-GAN - Dual Motion GAN for Future-Flow Embedded Video Prediction
DNA-GAN - DNA-GAN: Learning Disentangled Representations from Multi-Attribute Images
Domain and Geometry Agnostic CNNs for Left Atrium Segmentation in 3D Ultrasound - https://arxiv.org/abs/1805.00357 (github)
dp-GAN - Differentially Private Releasing via Deep Generative Model
DP-GAN - DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text
DPGAN - Differentially Private Generative Adversarial Network
DR-GAN - Representation Learning by Rotating Your Faces
DRAGAN - How to Train Your DRAGAN (github)
DRPAN - Discriminative Region Proposal Adversarial Networks for High-Quality Image-to-Image Translation
DSP-GAN - Depth Structure Preserving Scene Image Generation
DTN - Unsupervised Cross-Domain Image Generation
DualGAN - DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
Dualing GAN - Dualing GANs
Dynamics Transfer GAN - Dynamics Transfer GAN: Generating Video by Transferring Arbitrary Temporal Dynamics from a Source Video to a Single Target Image
E-GAN - Evolutionary Generative Adversarial Networks
EBGAN - Energy-based Generative Adversarial Network
ecGAN - eCommerceGAN : A Generative Adversarial Network for E-commerce
ED//GAN - Stabilizing Training of Generative Adversarial Networks through Regularization
EGAN - Enhanced Experience Replay Generation for Efficient Reinforcement Learning
ELEGANT - ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributes
EnergyWGAN - Energy-relaxed Wassertein GANs (EnergyWGAN): Towards More Stable and High Resolution Image Generation
ExGAN - Eye In-Painting with Exemplar Generative Adversarial Networks
ExposureGAN - Exposure: A White-Box Photo Post-Processing Framework (github)
ExprGAN - ExprGAN: Facial Expression Editing with Controllable Expression Intensity
f-CLSWGAN - Feature Generating Networks for Zero-Shot Learning
f-GAN - f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
FBGAN - Feedback GAN (FBGAN) for DNA: a Novel Feedback-Loop Architecture for Optimizing Protein Functions
FF-GAN - Towards Large-Pose Face Frontalization in the Wild
Fictitious GAN - Fictitious GAN: Training GANs with Historical Models
FIGAN - Frame Interpolation with Multi-Scale Deep Loss Functions and Generative Adversarial Networks
Fila-GAN - Synthesizing Filamentary Structured Images with GANs
First Order GAN - First Order Generative Adversarial Networks (github)
Fisher GAN - Fisher GAN
Flow-GAN - Flow-GAN: Bridging implicit and prescribed learning in generative models
FSEGAN - Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition
FTGAN - Hierarchical Video Generation from Orthogonal Information: Optical Flow and Texture
FusedGAN - Semi-supervised FusedGAN for Conditional Image Generation
FusionGAN - Learning to Fuse Music Genres with Generative Adversarial Dual Learning
G2-GAN - Geometry Guided Adversarial Facial Expression Synthesis
GAAN - Generative Adversarial Autoencoder Networks
GAGAN - GAGAN: Geometry-Aware Generative Adverserial Networks
GAMN - Generative Adversarial Mapping Networks
GAN - Generative Adversarial Networks (github)
GAN-ATV - A Novel Approach to Artistic Textual Visualization via GAN
GAN-CLS - Generative Adversarial Text to Image Synthesis (github)
GAN-RS - Towards Qualitative Advancement of Underwater Machine Vision with Generative Adversarial Networks
GAN-sep - GANs for Biological Image Synthesis (github)
GAN-VFS - Generative Adversarial Network-based Synthesis of Visible Faces from Polarimetric Thermal Faces
GANCS - Deep Generative Adversarial Networks for Compressed Sensing Automates MRI
GANDI - Guiding the search in continuous state-action spaces by learning an action sampling distribution from off-target samples
GANG - GANGs: Generative Adversarial Network Games
GANosaic - GANosaic: Mosaic Creation with Generative Texture Manifolds
GAP - Context-Aware Generative Adversarial Privacy
GAWWN - Learning What and Where to Draw (github)
GC-GAN - Geometry-Contrastive Generative Adversarial Network for Facial Expression Synthesis
GeneGAN - GeneGAN: Learning Object Transfiguration and Attribute Subspace from Unpaired Data (github)
GeoGAN - Generating Instance Segmentation Annotation by Geometry-guided GAN
Geometric GAN - Geometric GAN
GLCA-GAN - Global and Local Consistent Age Generative Adversarial Networks
GMAN - Generative Multi-Adversarial Networks
GMM-GAN - Towards Understanding the Dynamics of Generative Adversarial Networks
GoGAN - Gang of GANs: Generative Adversarial Networks with Maximum Margin Ranking
GONet - GONet: A Semi-Supervised Deep Learning Approach For Traversability Estimation
GP-GAN - GP-GAN: Towards Realistic High-Resolution Image Blending (github)
GP-GAN - GP-GAN: Gender Preserving GAN for Synthesizing Faces from Landmarks
GPU - A generative adversarial framework for positive-unlabeled classification
GRAN - Generating images with recurrent adversarial networks (github)
Graphical-GAN - Graphical Generative Adversarial Networks
GraspGAN - Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping
HAN - Chinese Typeface Transformation with Hierarchical Adversarial Network
HP-GAN - HP-GAN: Probabilistic 3D human motion prediction via GAN
HR-DCGAN - High-Resolution Deep Convolutional Generative Adversarial Networks
IAN - Neural Photo Editing with Introspective Adversarial Networks (github)
IcGAN - Invertible Conditional GANs for image editing (github)
ID-CGAN - Image De-raining Using a Conditional Generative Adversarial Network
IdCycleGAN - Face Translation between Images and Videos using Identity-aware CycleGAN
IFcVAEGAN - Conditional Autoencoders with Adversarial Information Factorization
iGAN - Generative Visual Manipulation on the Natural Image Manifold (github)
Improved GAN - Improved Techniques for Training GANs (github)
In2I - In2I : Unsupervised Multi-Image-to-Image Translation Using Generative Adversarial Networks
InfoGAN - InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (github)
IRGAN - IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval models
Iterative-GAN - Two Birds with One Stone: Iteratively Learn Facial Attributes with GANs (github)
IVE-GAN - IVE-GAN: Invariant Encoding Generative Adversarial Networks
iVGAN - Towards an Understanding of Our World by GANing Videos in the Wild (github)
IWGAN - On Unifying Deep Generative Models
KBGAN - KBGAN: Adversarial Learning for Knowledge Graph Embeddings
KGAN - KGAN: How to Break The Minimax Game in GAN
l-GAN - Representation Learning and Adversarial Generation of 3D Point Clouds
LAC-GAN - Grounded Language Understanding for Manipulation Instructions Using GAN-Based Classification
LAGAN - Learning Particle Physics by Example: Location-Aware Generative Adversarial Networks for Physics Synthesis
LAPGAN - Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (github)
LB-GAN - Load Balanced GANs for Multi-view Face Image Synthesis
LD-GAN - Linear Discriminant Generative Adversarial Networks
LDAN - Label Denoising Adversarial Network (LDAN) for Inverse Lighting of Face Images
LeakGAN - Long Text Generation via Adversarial Training with Leaked Information
LeGAN - Likelihood Estimation for Generative Adversarial Networks
LGAN - Global versus Localized Generative Adversarial Nets
LR-GAN - LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
LS-GAN - Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities
LSGAN - Least Squares Generative Adversarial Networks
MAD-GAN - Multi-Agent Diverse Generative Adversarial Networks
MAGAN - MAGAN: Margin Adaptation for Generative Adversarial Networks
MAGAN - MAGAN: Aligning Biological Manifolds
MalGAN - Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN
MaliGAN - Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
manifold-WGAN - Manifold-valued Image Generation with Wasserstein Adversarial Networks
MARTA-GAN - Deep Unsupervised Representation Learning for Remote Sensing Images
MaskGAN - MaskGAN: Better Text Generation via Filling in the ______
MC-GAN - Multi-Content GAN for Few-Shot Font Style Transfer (github)
McGAN - McGan: Mean and Covariance Feature Matching GAN
MD-GAN - Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks
MDGAN - Mode Regularized Generative Adversarial Networks
MedGAN - Generating Multi-label Discrete Electronic Health Records using Generative Adversarial Networks
MelanoGAN - MelanoGANs: High Resolution Skin Lesion Synthesis with GANs
memoryGAN - Memorization Precedes Generation: Learning Unsupervised GANs with Memory Networks
MGAN - Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks (github)
MGGAN - Multi-Generator Generative Adversarial Nets
MGGAN - MGGAN: Solving Mode Collapse using Manifold Guided Training
MIL-GAN - Multimodal Storytelling via Generative Adversarial Imitation Learning
MIX+GAN - Generalization and Equilibrium in Generative Adversarial Nets (GANs)
MLGAN - Metric Learning-based Generative Adversarial Network
MMD-GAN - MMD GAN: Towards Deeper Understanding of Moment Matching Network (github)
MMGAN - MMGAN: Manifold Matching Generative Adversarial Network for Generating Images
MoCoGAN - MoCoGAN: Decomposing Motion and Content for Video Generation (github)
ModularGAN - Modular Generative Adversarial Networks
MPM-GAN - Message Passing Multi-Agent GANs
MS-GAN - Temporal Coherency based Criteria for Predicting Video Frames using Deep Multi-stage Generative Adversarial Networks
MTGAN - MTGAN: Speaker Verification through Multitasking Triplet Generative Adversarial Networks
MuseGAN - MuseGAN: Symbolic-domain Music Generation and Accompaniment with Multi-track Sequential Generative Adversarial Networks
MV-BiGAN - Multi-view Generative Adversarial Networks
NAN - Understanding Humans in Crowded Scenes: Deep Nested Adversarial Learning and A New Benchmark for Multi-Human Parsing
NCE-GAN - Dihedral angle prediction using generative adversarial networks
ND-GAN - Novelty Detection with GAN
NetGAN - NetGAN: Generating Graphs via Random Walks
OCAN - One-Class Adversarial Nets for Fraud Detection
OptionGAN - OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning
ORGAN - Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models
ORGAN - 3D Reconstruction of Incomplete Archaeological Objects Using a Generative Adversary Network
OT-GAN - Improving GANs Using Optimal Transport
PacGAN - PacGAN: The power of two samples in generative adversarial networks
PAN - Perceptual Adversarial Networks for Image-to-Image Transformation
PassGAN - PassGAN: A Deep Learning Approach for Password Guessing
Perceptual GAN - Perceptual Generative Adversarial Networks for Small Object Detection
PGAN - Probabilistic Generative Adversarial Networks
PGD-GAN - Solving Linear Inverse Problems Using GAN Priors: An Algorithm with Provable Guarantees
PGGAN - Patch-Based Image Inpainting with Generative Adversarial Networks
Pip-GAN - Pipeline Generative Adversarial Networks for Facial Images Generation with Multiple Attributes
pix2pix - Image-to-Image Translation with Conditional Adversarial Networks (github)
pix2pixHD - High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (github)
PixelGAN - PixelGAN Autoencoders
PN-GAN - Pose-Normalized Image Generation for Person Re-identification
Pose-GAN - The Pose Knows: Video Forecasting by Generating Pose Futures
PPAN - Privacy-Preserving Adversarial Networks
PPGN - Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
PrGAN - 3D Shape Induction from 2D Views of Multiple Objects
ProGanSR - A Fully Progressive Approach to Single-Image Super-Resolution
Progressive GAN - Progressive Growing of GANs for Improved Quality, Stability, and Variation (github)
PS-GAN - Pedestrian-Synthesis-GAN: Generating Pedestrian Data in Real Scene and Beyond
PSGAN - Learning Texture Manifolds with the Periodic Spatial GAN
PS²-GAN - High-Quality Facial Photo-Sketch Synthesis Using Multi-Adversarial Networks
RadialGAN - RadialGAN: Leveraging multiple datasets to improve target-specific predictive models using Generative Adversarial Networks
RAN - RAN4IQA: Restorative Adversarial Nets for No-Reference Image Quality Assessment (github)
RankGAN - Adversarial Ranking for Language Generation
RCGAN - Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs
RefineGAN - Compressed Sensing MRI Reconstruction with Cyclic Loss in Generative Adversarial Networks
RenderGAN - RenderGAN: Generating Realistic Labeled Data
ResGAN - Generative Adversarial Network based on Resnet for Conditional Image Restoration
RNN-WGAN - Language Generation with Recurrent Generative Adversarial Networks without Pre-training (github)
RPGAN - Stabilizing GAN Training with Multiple Random Projections (github)
RTT-GAN - Recurrent Topic-Transition GAN for Visual Paragraph Generation
RWGAN - Relaxed Wasserstein with Applications to GANs
SAD-GAN - SAD-GAN: Synthetic Autonomous Driving using Generative Adversarial Networks
SAGA - Generative Adversarial Learning for Spectrum Sensing
SalGAN - SalGAN: Visual Saliency Prediction with Generative Adversarial Networks (github)
SAR-GAN - Generating High Quality Visible Images from SAR Images Using CNNs
SBADA-GAN - From source to target and back: symmetric bi-directional adaptive GAN
SCH-GAN - SCH-GAN: Semi-supervised Cross-modal Hashing by Generative Adversarial Network
SD-GAN - Semantically Decomposing the Latent Spaces of Generative Adversarial Networks
Sdf-GAN - Sdf-GAN: Semi-supervised Depth Fusion with Multi-scale Adversarial Networks
SEGAN - SEGAN: Speech Enhancement Generative Adversarial Network
SeGAN - SeGAN: Segmenting and Generating the Invisible
SegAN - SegAN: Adversarial Network with Multi-scale L1 Loss for Medical Image Segmentation
SeqGAN - SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient (github)
SG-GAN - Semantic-aware Grad-GAN for Virtual-to-Real Urban Scene Adaption (github)
SGAN - Texture Synthesis with Spatial Generative Adversarial Networks
SGAN - Stacked Generative Adversarial Networks (github)
SGAN - Steganographic Generative Adversarial Networks
SGAN - SGAN: An Alternative Training of Generative Adversarial Networks
sGAN - Generative Adversarial Training for MRA Image Synthesis Using Multi-Contrast MRI
SimGAN - Learning from Simulated and Unsupervised Images through Adversarial Training
SisGAN - Semantic Image Synthesis via Adversarial Learning
SketchGAN - Adversarial Training For Sketch Retrieval
SketchyGAN - SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis
SL-GAN - Semi-Latent GAN: Learning to generate and modify facial images from attributes
SN-GAN - Spectral Normalization for Generative Adversarial Networks (github)
Sobolev GAN - Sobolev GAN
Social GAN - Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks
Softmax GAN - Softmax GAN
Spike-GAN - Synthesizing realistic neural population activity patterns using Generative Adversarial Networks
Splitting GAN - Class-Splitting Generative Adversarial Networks
SRGAN - Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
SRPGAN - SRPGAN: Perceptual Generative Adversarial Network for Single Image Super Resolution
SS-GAN - Semi-supervised Conditional GANs
ss-InfoGAN - Guiding InfoGAN with Semi-Supervision
SSGAN - SSGAN: Secure Steganography Based on Generative Adversarial Networks
SSL-GAN - Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks
ST-CGAN - Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal
ST-GAN - Style Transfer Generative Adversarial Networks: Learning to Play Chess Differently
ST-GAN - ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
StackGAN - StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks (github)
StainGAN: Stain Style Transfer for Digital Histological Images - https://arxiv.org/abs/1804.01601 (github)
StarGAN - StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation (github)
SteinGAN - Learning Deep Energy Models: Contrastive Divergence vs. Amortized MLE
Super-FAN - Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs
SVSGAN - SVSGAN: Singing Voice Separation via Generative Adversarial Network
SWGAN - Solving Approximate Wasserstein GANs to Stationarity
SyncGAN - SyncGAN: Synchronize the Latent Space of Cross-modal Generative Adversarial Networks
S^2GAN - Generative Image Modeling using Style and Structure Adversarial Networks
TAC-GAN - TAC-GAN - Text Conditioned Auxiliary Classifier Generative Adversarial Network (github)
TAN - Outline Colorization through Tandem Adversarial Networks
tempoGAN - tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow
Text2Shape - Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings
textGAN - Generating Text via Adversarial Training
TextureGAN - TextureGAN: Controlling Deep Image Synthesis with Texture Patches
TGAN - Temporal Generative Adversarial Nets
TGAN - Tensorizing Generative Adversarial Nets
TGAN - Tensor-Generative Adversarial Network with Two-dimensional Sparse Coding: Application to Real-time Indoor Localization
tiny-GAN - Analysis of Nonautonomous Adversarial Systems
TP-GAN - Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis
Triple-GAN - Triple Generative Adversarial Nets
tripletGAN - TripletGAN: Training Generative Model with Triplet Loss
TV-GAN - TV-GAN: Generative Adversarial Network Based Thermal to Visible Face Recognition
UGACH - Unsupervised Generative Adversarial Cross-modal Hashing
UGAN - Enhancing Underwater Imagery using Generative Adversarial Networks
Unim2im - Unsupervised Image-to-Image Translation with Generative Adversarial Networks (github)
UNIT - Unsupervised Image-to-image Translation Networks (github)
Unrolled GAN - Unrolled Generative Adversarial Networks (github)
UV-GAN - UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition
VAE-GAN - Autoencoding beyond pixels using a learned similarity metric
VariGAN - Multi-View Image Generation from a Single-View
VAW-GAN - Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks
VEEGAN - VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning (github)
VGAN - Generating Videos with Scene Dynamics (github)
VGAN - Generative Adversarial Networks as Variational Training of Energy Based Models (github)
VGAN - Text Generation Based on Generative Adversarial Nets with Latent Variable
ViGAN - Image Generation and Editing with Variational Info Generative Adversarial Networks
VIGAN - VIGAN: Missing View Imputation with Generative Adversarial Networks
VoiceGAN - Voice Impersonation using Generative Adversarial Networks
VOS-GAN - VOS-GAN: Adversarial Learning of Visual-Temporal Dynamics for Unsupervised Dense Prediction in Videos
VRAL - Variance Regularizing Adversarial Learning
WaterGAN - WaterGAN: Unsupervised Generative Network to Enable Real-time Color Correction of Monocular Underwater Images
WaveGAN - Synthesizing Audio with Generative Adversarial Networks
weGAN - Generative Adversarial Nets for Multiple Text Corpora
WGAN - Wasserstein GAN (github)
WGAN-GP - Improved Training of Wasserstein GANs (github)
WS-GAN - Weakly Supervised Generative Adversarial Networks for 3D Reconstruction
XGAN - XGAN: Unsupervised Image-to-Image Translation for many-to-many Mappings
ZipNet-GAN - ZipNet-GAN: Inferring Fine-grained Mobile Traffic Patterns via a Generative Adversarial Neural Network
α-GAN - Variational Approaches for Auto-Encoding Generative Adversarial Networks (github)
β-GAN - Annealed Generative Adversarial Networks
Δ-GAN - Triangle Generative Adversarial Networks

moving to other repo.


Add support for exporting SuperGLUE task test predictions in leaderboard submission format

This issue tracks progress on adding support for exporting SuperGLUE task test predictions in leaderboard submission format (see "How do I use SuperGLUE" for details of submission format).

More context: the SuperGLUE leaderboard expects task predictions to have a specific format. The details of the task help to determine this format, for example each line in the MultiRC task predictions jsonl file is expected to contain a json object that follows "...the JSONL structure in the data, so that each line represents a single context paragraph; be sure to use the paragraph ID, question ID, and answer ID."  Here's an example line from the SuperGLUE submission sample data:
{
  "idx": 0,
  "passage": {
    "questions": [
      {
        "idx": 0,
        "answers": [
          {
            "idx": 0,
            "label": 0
          },
          {
            "idx": 1,
            "label": 0
          },
          {
            "idx": 2,
            "label": 0
          }
        ]
      },
      {
        "idx": 1,
        "answers": [
          {
            "idx": 3,
            "label": 0
          },
          {
            "idx": 4,
            "label": 0
          },
          {
            "idx": 5,
            "label": 0
          },
          {
            "idx": 6,
            "label": 0
          },
          {
            "idx": 7,
            "label": 0
          }
        ]
      }
    ]
  }
}

For other tasks in SuperGLUE the submission format is simpler — it's just a json object with an idx field (int) and a label field (int or string), like this:
{"idx": 0, "label": 0}

or
{"idx": 0, "label": "entailment"}

or
{"idx": 1, "label": "true"}

And more background:
 jiant2 already has --write_val_preds and --write_test_preds options, and these save preds as arrays using torch.save. To format these preds into valid leaderboard submissions we need to do some more work (especially for MultiRC, which, as explained above has some special requirements).
Current plan:

Add a new command line arg (args.super_glue_format) indicating that any task that has a SuperGLUE submission formatting method should have its predictions formatted for SuperGLUE leaderboard submission.
At this line in runscript, add the check for args.super_glue_format and, for any SuperGLUE tasks in the test_task_list, apply the task's SuperGLUE prediction formatter (defined in step 3...) to translate the predictions and write them to the run dir in the expected format/filetype.
Add a SuperGLUE prediction formatter method to the task definitions for all the SuperGLUE tasks.

@jeswan, @zphang — please let me know if you have any recommendations for better ways to do this, or if you have any concerns about how this would extend to support the other leaderboard tasks.

My recommendation is to put the super-glue-format logic and running into a separate script. It is a specific requirement for a specific benchmark, and shouldn't be incorporated into the main runscript flow.

My recommendation is to put the super-glue-format logic and running into a separate script. It is a specific requirement for a specific benchmark, and shouldn't be incorporated into the main runscript flow.
Adding the prediction formatter to the tasks themselves is fine by me.

Thanks, @zphang — PR https://github.com/jiant-dev/jiant/pull/117 (now visible as a draft) tries to implement this way.


Get time left from timer

I wanted to implement a debounce function using a timer but it seems like we can't get the time left.
Am I missing something or is it not possible with the current API?

I think a pull request implementing that functionality is welcome.
The catch is that the timeout is stored as the event loop's idea of the current time + the timeout, i.e., as absolute time. To derive the remaining time, you have to compute handle->timeout - loop->time and check it's >= 0.

For reference, diagnostic report in Node.js and its predecessor node-report do just that to derive the remaining time but the timeout field (due on older versions of libuv on Windows) on the timer handle isn't public.
https://github.com/nodejs/node/blob/84b35b286769758610173291c6e9fd9eebdaaf1f/src/node_report_utils.cc#L150-L157
https://github.com/nodejs/node-report/blob/b19df9b4f8f3fce58c7fa2408b2bf2060781515e/src/utilities.cc#L468-L490
There's a previously abandoned attempt at adding an API: https://github.com/libuv/libuv/pull/1255


Fix: homepage screen height

Describe the bug
On the homepage of the website, there is an unnecessary scrolling, even when the entire content can fit in without the need of scrollbar to appear.
Expected behavior
Adjust the height of main content to take full height while not exceeding 100vh.
Can you assign me under gssoc'24

Hey @Miran-Firdausi, the UI is redesigned, so this will be fixed. Contribute to other issues good luck.


Ethan: Watch powerfullovespellz.blogspot

Ethan requests the watch of the watch_keyword powerfullovespellz\.blogspot. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

powerfullovespellz\.blogspot has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Machavity in Charcoal HQ


help center: Update "SAML authentication" page to follow help center guidelines

This PR replaces #22680.
Testing:

Manually tested links
Did not test the instructions, so a careful review of that would be great.

Current page: https://zulip.com/help/saml-authentication

Intro:

Instructions:

@mateuszmandera @drrosa would be great to get your reviews on this one!

This is great; posted a couple small comments.

Fixed the one issue and marked this to merge once CI finishes. Thanks for improving this @alya and @mateuszmandera!

Did not test the instructions, so a careful review of that would be great.

@alya Should I go ahead and manually test the instructions using free trial accounts from those 4 providers?

Did not test the instructions, so a careful review of that would be great.

@alya Should I go ahead and manually test the instructions using free trial accounts from those 4 providers?

My vote would be on it not being worth the effort - these instructions (in their original shape) have been used by me and some Zulip users, so they should be correct. This PR kind of just re-phrases them but without actually changing the steps - so it seems unlikely for the instructions to have become incorrect in the process

Not re-testing the instructions at this point works for me. Thanks for checking!


suggestion: mathjax in remark42.

Hi.
I believe that it would be both nice and very easy to get mathjax in remark42, because it requires only to provide a script in any html page produced via remark42. I do not understand the detail of org-export-head code https://github.com/itf/org-export-head that I use for my blog, but at any html export of a new post that contains a formula that can be translated by mathjax, there is the following script in the html exported page, just before :
<!--/*--><![CDATA[/*><!--*/
 function CodeHighlightOn(elem, id)
 {
   var target = document.getElementById(id);
   if(null != target) {
     elem.cacheClassElem = elem.className;
     elem.cacheClassTarget = target.className;
     target.className = "code-highlighted";
     elem.className   = "code-highlighted";
   }
 }
 function CodeHighlightOff(elem, id)
 {
   var target = document.getElementById(id);
   if(elem.cacheClassElem)
     elem.className = elem.cacheClassElem;
   if(elem.cacheClassTarget)
     target.className = elem.cacheClassTarget;
 }
/*]]>*///-->
</script>`
`<script type="text/x-mathjax-config">
    MathJax.Hub.Config({
        displayAlign: "center",
        displayIndent: "0em",
        "HTML-CSS": { scale: 100,
                        linebreaks: { automatic: "false" },
                        webFont: "TeX"
                       },
        SVG: {scale: 100,
              linebreaks: { automatic: "false" },
              font: "TeX"},
        NativeMML: {scale: 100},
        TeX: { equationNumbers: {autoNumber: "AMS"},
               MultLineWidth: "85%",
               TagSide: "right",
               TagIndent: ".8em"
             }
}); </script>`
`<script type="text/javascript"
        src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS_HTML"></script>

And formulas in html page are very clean. This system is already use for comments here:
https://math.stackexchange.com/ , see for example: https://math.stackexchange.com/questions/1705236/show-that-a-theory-is-complete-if-and-only-if-b-or-c-is-provable-implies-eith . if I have correctly undestood, this comment  engine is in  "staticman": https://staticman.net/ and it works with GitHub. In my opinion, it would not be difficult at all to provide this script for all messages generated via remark42, or for only those that required it. I am aware that of course that it is certainly not a "popular request", but it would be an additional feature of remark42 that could be appreciated by some of their users.

Previously, please read "just before </head>"

My vote is against that because of it would be unused by most but the very few sites and there are existing solutions (latex2png, MathJax convert to image, MathML Cloud, Online LaTeX Equation Editor) which allow you to embed formulas as images.
The only case this wouldn't be enough I can think of is some kind of science- or math-related site where users would share formulas frequently, and yet I would think twice about adding such functionality for everyone for the sake of such a scenario.

I am surprised, because  I believe that a script to a cdn mathjx server in
the html of remark42 replies would be enough to povide this function.  If I
am right,  it would take one minute to write this script. People can use
mathjax not . It would cost nothing. ..
Le dim. 22 déc. 2019 23:54, Dmitry Verkhoturov<EMAIL_ADDRESS>a
écrit :

My vote is against that because of it would be unused by most but the very
few sites and there are existing solutions (latex2png
https://latex2png.com/, MathJax convert to image
https://codepen.io/pkra/pen/PZLyQO, MathML Cloud
https://mathmlcloud.org/, Online LaTeX Equation Editor
https://www.codecogs.com/latex/eqneditor.php) which allow you to embed
formulas as images.
The only case this wouldn't be enough I can think of is some kind of
science- or math-related site where users would share formulas frequently,
and yet I would think twice about adding such functionality for everyone
for the sake of such a scenario.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/umputun/remark/issues/478?email_source=notifications&email_token=AB7ENKTU27PUCSULILISK43QZ7V3BA5CNFSM4JMW7642YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHP3RHI#issuecomment-568309917,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB7ENKXV4KTF63WIX374L7LQZ7V3BANCNFSM4JMW764Q
.

I understand that it very trivial to implement and my only concern is increasing complexity of whatever you can put into the comment field and not the software complexity.
I think if you want to add your LaTeX math function in the comment, your expectation today is it's not supported and you need to generate an image with it first. I've never seen a site that supported formulas in the comment except for the one you mentioned, and that says something about the demand for this feature. Based on that I see we are not leaving someone with unfulfilled expectations: users on most of the sites don't expect this from the comment system in the first place.

On the bottom right there is a text "Styling with Markdown supported". It's up to Remark42 maintainer to decide, but I am strongly against having something like "Styling with Markdown and MathJax is supported" unless I'll see some reasoning for that better than "it's cheap to add" (it's not an argument at all) and "would be used on a single site". If there is a feature that will be turned ON on a single installation only, it should not be implemented in the first place in my opinion.

There is a misunderstanding. No user have to decide if he wants use or not use mathjax.  Mathjax works immediately when you write a formula in latex code inside $$ or inside ( ) . If you do not  write such a formula you do not know  that mathjax is supported or not. Now, it was only a suggestion.  Best wishes and merry Christmas.

It seems to me that you do not know how mathjax works for someone that is writing hence a misunderstanding.  The user has not to decide if mathjax must be implemented or not, because  if the mathjax script is in the html page, it will work for any latex code inside $$ or inside antislash and brackets. Now, it was only a suggestion to make Remark42 still more powerful.  If you do not want implement it and if I see that I need it, I will see another solution.  BEST wishes and merry  Christmas.

because if the mathjax script is in the html page, it will work for any latex code inside $$ or inside antislash and brackets

@joseph-vidal-rosset just to clarify - adding a script to the frontend, rendering mathjax (or any other special rendering) is not the solution here, because of remark42 renders everything on the server-side and stores the final html. This final html served directly by the backend, i.e. no special formatting symbols even parsed on the frontend side.

Thanks for this useful clarification.
Merry  Christmas!
Le mar. 24 déc. 2019 18:48, Umputun<EMAIL_ADDRESS>a écrit :

because if the mathjax script is in the html page, it will work for any
latex code inside $$ or inside antislash and brackets
@joseph-vidal-rosset https://github.com/joseph-vidal-rosset just to
clarify - adding a script to the frontend, rendering mathjax (or any other
special rendering) is not the solution here, because of remark42 renders
everything on the server-side and stores the final html. This final html
served directly by the backend, i.e. no special formatting symbols even
parsed on the frontend side.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/umputun/remark/issues/478?email_source=notifications&email_token=AB7ENKQYZVWUMVNJOO3TWN3Q2JDNZA5CNFSM4JMW7642YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHTPMJI#issuecomment-568784421,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB7ENKVUJFUICKIUYZ37UZDQ2JDNZANCNFSM4JMW764Q
.


dataset is incomplete

Hello, author. The dataset you made public is incomplete. Could you please share the remaining datasets with me? Thank you.
The following files are missing:
train distant.json
meta
-rel2id.json
-rel info.json

You can find the requested datasets from this link.


Add support of ES6 modules and --experimental-modules flag

See appDir (require.main.filename) socketcluster/index.js#L72

A more detailed description of the issue at hand:
In Node v8.6.1 it is possible to start using ES6 modules in NodeJS. Because of the lack of a dynamic loader that works (the proposed import() loader is available in V8 through --harmony_dynamic_import but not yet implemented on the NodeJS side) it causes all kinds of problems with SC versions up to and including 8.0.2.
NodeJS currently requires the entry point to be an .mjs file (or ES6 Module file) in order to switch to ES6 module mode. It is possible to use the ES6 import statement to load CommonJS modules, but their environment is a bit different from a non ES6 Module entry point.
One of the key differences is the lack of require.main entry which SC uses to determine the root path of the SC project.
Additionally, there are some problems regarding options.environment which doesn't seem to get passed on properly.
It is possible to rewrite the boilerplate server.js and worker.js in ES6 module form and rename them to .mjs. However SC uses require to load the worker.js file and this is not allowed for ES6 modules.
This means that currently it is not possible to use code written using ES6 Module form to be used directly in SC (unless transpiled).

Set of rudimentary changes to socketcluster to allow workers to run as ES6 modules.
https://gist.github.com/mauritslamers/1e5e24207e389dd00373b57b1f0ce359

@mauritslamers Nice looks good! I'll test it out.
I did something similar for the broker controller but I used the extends keyword and allowed the developer override the run() method with their own https://github.com/SocketCluster/sc-broker/blob/class-based-controllers/test/stubs/broker-controller-stub.mjs#L8-L9
I see you've used a SCWorker.setup(...) approach https://gist.github.com/mauritslamers/1e5e24207e389dd00373b57b1f0ce359#file-worker-mjs-L19-L20 -
Can you think of any problems with the extends approach? I like it a bit more because it reminds me of the Runnable interface which Java uses for running custom code inside threads (which is essentially what we're doing here except using processes instead of threads). http://www.java2novice.com/java_thread_examples/implementing_runnable/

@mauritslamers I created a new feature branch for this change on this repo called class-based-controllers https://github.com/SocketCluster/socketcluster/tree/class-based-controllers. If you want your contribution to show up on the GitHub repo, you can do a pull request against that branch. I'll have to do a bit of tweaking later to finalise this feature before we can merge it into master.

There are a few issues with the extends approach.

it pretends to be class based, which in essence it is not. The class and extends keywords are syntactical sugar over the prototypal inheritance
extends creates a new constructor + prototype. It doesn't create an instance
As described in the comments in the gist: To keep the option of creating a new SCWorker instance you need a reference to the class somehow. When you extend SCWorker using extends you get back a reference to a new constructor + prototype, which you still need to feed to the incoming worker incoming message handler.

it pretends to be class based, which in essence it is not. The class and extends keywords are syntactical sugar over the prototypal inheritance

True, but maybe this syntactic sugar doesn't hurt in this case? I see an increasing number of projects using classes now.

extends creates a new constructor + prototype. It doesn't create an instance

In that case we can leave the responsibility of instantiating the SCWorker (or SCBroker) to the developer; then the developer can even delay the instantiation if that is what they want and the rest of SC would just wait for it (example for the broker):
https://github.com/SocketCluster/sc-broker/blob/class-based-controllers/test/stubs/broker-controller-stub.mjs#L77
It does mean that the developer will need to add one line of extra boilerplate logic (which is a negative) but on the plus side I'm thinking that maybe the extra flexibility is worth it.
We can make the underlying SCWorker class a singleton if necessary and throw an error if the developer tries to instantiate it twice (or something like that).
I understand your point though that maybe instantiation doesn't feel right at a conceptual level. I'll think about this more.

As described in the comments in the gist: To keep the option of creating a new SCWorker instance you need a reference to the class somehow...

Ok. I'll look through your code and see if there are some small adjustments that can be made to solve this problem (to see if it's even possible). I was able to get it to work with the broker controller but it might be different in this case.

In a way I don't see the point of adding the extra boilerplate code of instantiation when you can achieve the same by having the developer call a function which does both extending and instantiation by means of an object literal. So, in a way you could even merge these two approaches and just have something like:
new SCWorker({
  run() {
    // attach handlers here.
    this.on('masterMessage', (data, respond) => {
    // etc
    })
  }

@mauritslamers I think that functionally it's perfect.
But it's a big change and it should affect all SC-based logic in the future for the rest of eternity. It's not just about functionality and correctness; there is also a convention (aesthetic) aspect to it.
Developers are attached to conventions; I think that's why they introduced classes to JavaScript (ECMAScript people had to give up eventually); developers just couldn't adopt to the prototype approach fully.
So in terms of functionality, your current approach is a huge improvement (it does everything we need) - but in terms of following existing programming conventions I think we should evaluate it further. Are there any other frameworks using a similar pattern?
I will consult with some existing SC users to see what they think.

I've raised a proposal for people to vote on https://github.com/SocketCluster/socketcluster/issues/333


Should view/compiled/kint.js be minified?

To be more precise, why serve a minified version and not the unminified /view/base.js?

Well there are times when you need to see the source code of the page, and I'd rather the css/js would reside in one line and not clutter up the remainder of the page.
I don't see any real reason not to do that. If you do, share your thoughts, I will consider them :)

Sorry for delay.
I needed to change something (can't remember what) from the js. I understand that unminified javascript would clutter the document, but doesn't Kint do that pretty fine on its own if a big object is being debugged? :)
I for one haven't had to look at page source even once while there is Kint output on page. Someone else might though, so how about making the unminified js reside in the same folder and adding an option for deciding which one to use?

an option for deciding which one to use?

There actually used to be a setting for that. Then, after several years of never using it once I removed it in an attempt to minimize feature creep.
This change is too arbitrary to become a priority (of which are many - mainly updating the friggin project page :( ), but you're free to try to convince me otherwise :) If there's an actual real-world usecase for this, I will implement it, but currently I see none...

I don't completely agree, but I see your point, and it's more like a matter of preference than of doing things wrong or right :)

I don't see how you got that from what I said. It's precisely a matter of doing changes for objective reasons and not because of baseless opinions.
If you have a real-world reason for the change (any change) you are requesting, I will consider it and if it's a valid concern that has a viable solution, I will definitely implement it, take a look at the discussion in any of the 100+ issues I closed if you don't believe me.


MOUSE UTILITIES/FIND MY MOUSE - Possibility to have the 'SPOTLIGHT COLOR' transparent (NO COLOR)

Description of the new feature / enhancement
In MOUSE UTILITIES/FIND MY MOUSE/SPOTLIGHT COLOR to have a choice with No Color (Transparent/Clear).
Scenario when this would be used?
I do share my screen a lot and when I use the feature ENABLE FIND MY MOUSE to show something specific on my screen, I do have a lot of comments that it is difficult to see, because it is blurry. My spotlight color is white.
Supporting information
No response

/dup https://github.com/microsoft/PowerToys/issues/15512#ref-issue-1233975062


missing boost_python3 when cmake.. in bulid folder

Hi,
I am trying to install this in a Ubuntu system，but i meet the following errors. Any help will be much appreciated. Thank you.

This external version of the NumPy/Boost.Python bindings is no longer being maintained, now that all of its functionality is available directly from Boost.Python.  If you've got Boost 1.71 (as it seems above), you don't need this package at all.


[SPARK-8887][SQL]Explicit define which data types can be used as dynamic partition columns

This PR enforce dynamic partition column data type requirements by adding analysis rules.
JIRA: https://issues.apache.org/jira/browse/SPARK-8887

Merged build triggered.

Merged build started.

I mistakenly delete the branch on github, sorry about this.

Test build #40872 has started for   PR 8201 at commit d926a61.

@rxin @liancheng

This is exactly the same as #8132, right?

yes, it's the same, I mistakenly deleted them both on github, and reopened a new PR here

OK. This LGTM, but I'd like to ask @rxin for a final review.

Test build #40872 has finished for   PR 8201 at commit d926a61.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Merging in master.


[Bug] - yield keyword not formatted

Describe the bug
When using this theme with PyCharm, the yield keyword is not formatted as a secondary keyword.
Jetbrains Screenshot

VSCode Screenshot

Additional context/Logs

Addressed in https://github.com/dinbtechit/vscode-theme/pull/196

Resolved in V1.10.11


[TF 2.0 API Docs] tf.audio.encode_wav

API Doc update for https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/audio/encode_wav
Correct links? No
Clear Description? No
Usage Example?
Parameters Defined? Yes, not verified against current code
Returns defined? Yes, not verified against current code
Raises listed and defined? No
Visuals, if applicable? No
I will submit a PR, please assign this bug to me.
Part of Issue #28237 & Issue #28236

@missaugustina Can you please let us know if this is the duplicate of #28237, if so please let us know so that we will close this issue as the same issue is being referenced in #28237

I thought I was supposed to file one issue per api call?

See also Issue #28237 & Issue #28236

I plan to submit a PR for this issue

These details are generated through .pbtxt file, you can see the same details here.
For tutorials and usage examples on tf.audio, you can refer this link, Thanks.


reduces image size to 1 mb when greater

This PR reduces the image size to 1 mb, having size greater than 1 mb. Resolution is 975 X 975

Not required as of now.


build script output flags not passed to rustdoc

Seems to be the same issue as #1469
I have a simple build script that just outputs the search path:
fn main() {
    println!("cargo:rustc-link-search=native=/home/lee/opt/lib");
}

This seems to work correctly for building and running normal tests (both inline and in tests/ folder) but the -L flag doesn't get passed to rustdoc.
Example of normal test output vs rustdoc output:
Running `rustc tests/version.rs --crate-name version --crate-type bin -g --test -C metadata=0ac5f00c90e58674 -C extra-filename=-0ac5f00c90e58674 --out-dir /home/lee/rust/doctest/target/debug --emit=dep-info,link -L dependency=/home/lee/rust/doctest/target/debug -L dependency=/home/lee/rust/doctest/target/debug/deps --extern doctest=/home/lee/rust/doctest/target/debug/libdoctest.rlib -L native=/home/lee/opt/lib -L native=/home/lee/opt/lib`

Running `rustdoc --test /home/lee/rust/doctest/src/lib.rs --crate-name doctest -L dependency=/home/lee/rust/doctest/target/debug/deps --extern doctest=/home/lee/rust/doctest/target/debug/libdoctest.rlib`

(Also note the search path is added twice to the regular test but I suppose that's a separate issue)

Is there a failure mode of this you ran into? Generating documentation shouldn't require linking a library so I don't believe it requires native objects to be available.

It failed to link when compiling the doc tests. I will try to set up a minimal test case.

Ok I think it's only a problem when linking to a shared library. Minimal test project available here.
Just run build_hello.sh (requires cmake) and then run cargo test.

I believe this was fixed in #1530

It looks like this is still a problem in cargo 0.5.0-nightly (833b947 2015-09-13) on Windows, when depending on rust-openssl. It says that -lssl32 and -leay32 can't be found. Here is the log from my latest attempt to enable doc-tests on AppVeyor.

Can you try using the nightly channel to make sure you're running the most up-to-date Cargo? I'm not sure if this fix was in that Cargo or not.

Sure, I can give it a try tomorrow. I just thought that it would have propagated to the official builds by now, since it was merged in April.

@gkoz sugested in the forums that my problem could be this. It happens in Travis in stable, beta and nightly chanels.

Ah ok, sounds like this is probably still an issue!

Just ran into this trying to get Diesel running on my Windows box. Can confirm it's still an issue.

It seems this issue was never really fixed, and this has been causing projects to avoid doctests for packages that use FFI to shared libraries simply because it requires testers to set the environment variables manually. I can't tell if it's now intentional that cargo:rustc-link-search=native=* not apply to doctests.
I know one strain of advice is that native libraries built in OUT_DIR should usually be static, though this offloads complexity when that library may have transitive dependencies linked using RPATH.