create dashboard admin user hangs and ceph -s has  PGs pending on creation with slow ops.

Bug Report
What happened:
Once in a while, the installation hangs on create dashboard admin user and takes 40min+ to complete that particular task (ceph-dashboard-no-grafana is a duplicate of ceph-dashboard with grafana server installation removed) :
TASK [ceph-dashboard-no-grafana : config the current dashboard backend] *************************************************
Friday 14 May 2021  23:40:56 +0000 (0:00:00.098)       0:21:21.884 ************
ok: [ceph01 -> <IP_ADDRESS>]

TASK [ceph-dashboard-no-grafana : disable mgr dashboard module (restart)] ***********************************************
Friday 14 May 2021  23:40:57 +0000 (0:00:00.989)       0:21:22.873 ************
changed: [ceph01 -> <IP_ADDRESS>]

TASK [ceph-dashboard-no-grafana : enable mgr dashboard module (restart)] ************************************************
Friday 14 May 2021  23:40:59 +0000 (0:00:01.786)       0:21:24.660 ************
changed: [ceph01 -> <IP_ADDRESS>]

TASK [ceph-dashboard-no-grafana : create dashboard admin user] **********************************************************
Friday 14 May 2021  23:41:01 +0000 (0:00:02.019)       0:21:26.679 ************

ceph -s
  cluster:
    id:     <fsid>
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim
            1 MDSs report slow metadata IOs
            25484 PGs pending on creation
            Reduced data availability: 50 pgs inactive
            20 slow ops, oldest one blocked for 1609 sec, daemons [osd.36,osd.37,osd.46] have slow ops.

  services:
    mon: 3 daemons, quorum ceph-node-2,ceph-node-0,ceph-node-1 (age 38m)
    mgr: ceph-node-1(active, since 21m), standbys: ceph-node-2, ceph-node-0
    mds: 1/1 daemons up, 2 standby
    osd: 66 osds: 66 up (since 27m), 66 in (since 28m)

  data:
    volumes: 1/1 healthy
    pools:   13 pools, 2164 pgs
    objects: 20 objects, 2.3 KiB
    usage:   4.0 GiB used, 6.4 TiB / 6.4 TiB avail
    pgs:     0.832% pgs unknown
             1.571% pgs not active
             2112 active+clean
             32   creating+activating
             18   unknown
             2    peering

What you expected to happen:
ceph mgr and rados-gw commands should not hang for over 40min each.
How to reproduce it (minimal and precise):
install ceph and once in a while, it will hang on that task.  Roughly once every three installs that happens to me. Not sure if it is specific to the env or is an installation issue. Running 3 VMs on top of openstack with lvm mounts and installing ceph on those vms and volumes.

Share your group_vars files, inventory and full ceph-ansibe log
# grep -rEv '^ *#|^\s*$|^\s*dummy:' . --include="*.yml"
./clients.yml:---
./mdss.yml:---
./mgrs.yml:---
./mons.yml:---
./osds.yml:---
./osds.yml:devices:
./osds.yml:  - /dev/vdd
./osds.yml:  - /dev/vde
./osds.yml:  - /dev/vdf
./osds.yml:  - /dev/vdg
./osds.yml:  - /dev/vdh
./osds.yml:  - /dev/vdi
./osds.yml:  - /dev/vdj
./osds.yml:  - /dev/vdk
./osds.yml:  - /dev/vdl
./osds.yml:  - /dev/vdm
./osds.yml:  - /dev/vdn
./osds.yml:  - /dev/vdo
./osds.yml:  - /dev/vdp
./osds.yml:  - /dev/vdq
./osds.yml:  - /dev/vdr
./osds.yml:  - /dev/vds
./osds.yml:  - /dev/vdt
./osds.yml:  - /dev/vdu
./osds.yml:  - /dev/vdv
./osds.yml:  - /dev/vdw
./osds.yml:  - /dev/vdx
./osds.yml:  - /dev/vdy
./osds.yml:bluestore_wal_devices:
./osds.yml: - /dev/vdb
./osds.yml: - /dev/vdc
./osds.yml:osd_auto_discovery: false
./rgws.yml:---
./rgws.yml:rgw_create_pools:
./rgws.yml:  "{{ rgw_zone }}.rgw.buckets.data":
./rgws.yml:    pg_num: 64
./rgws.yml:    type: replicated
./rgws.yml:    size: 3
./rgws.yml:  "{{ rgw_zone }}.rgw.buckets.index":
./rgws.yml:    pg_num: 16
./rgws.yml:    size: 3
./rgws.yml:    type: replicated
./rgws.yml:  "{{ rgw_zone }}.rgw.meta":
./rgws.yml:    pg_num: 8
./rgws.yml:    size: 3
./rgws.yml:    type: replicated
./rgws.yml:  "{{ rgw_zone }}.rgw.log":
./rgws.yml:    pg_num: 8
./rgws.yml:    size: 3
./rgws.yml:    type: replicated
./rgws.yml:  "{{ rgw_zone }}.rgw.control":
./rgws.yml:    pg_num: 8
./rgws.yml:    size: 3
./rgws.yml:    type: replicated
./zone.yml:rgw_zone: us-west-1
./zone.yml:rgw_zonemaster: true
./zone.yml:rgw_zonesecondary: false
./zone.yml:rgw_zonegroup: us-west
./zone.yml:rgw_zonegroupmaster: true
./all/all.yml:---
./all/all.yml:ceph_release_num:
./all/all.yml:  octopus: 15
./all/all.yml:ntp_service_enabled: false
./all/all.yml:ceph_origin: repository
./all/all.yml:ceph_repository: community
./all/all.yml:ceph_stable_release: octopus
./all/all.yml:monitor_interface: ens3
./all/all.yml:journal_size: 5120 # OSD journal size in MB
./all/all.yml:public_network: <IP_ADDRESS>/24
./all/all.yml:radosgw_interface: ens3
./all/all.yml:containerized_deployment: true
./all/all.yml:openstack_config: true
./all/all.yml:openstack_glance_pool:
./all/all.yml: name: "images"
./all/all.yml: application: "rbd"
./all/all.yml: pg_autoscale_mode: true
./all/all.yml:openstack_cinder_pool:
./all/all.yml: name: "volumes"
./all/all.yml: application: "rbd"
./all/all.yml: pg_autoscale_mode: true
./all/all.yml:openstack_nova_pool:
./all/all.yml: name: "vms"
./all/all.yml: application: "rbd"
./all/all.yml: pg_autoscale_mode: true
./all/all.yml:openstack_cinder_backup_pool:
./all/all.yml: name: "backups"
./all/all.yml: application: "rbd"
./all/all.yml: pg_autoscale_mode: true
./all/all.yml:openstack_pools:
./all/all.yml: - "{{ openstack_glance_pool }}"
./all/all.yml: - "{{ openstack_cinder_pool }}"
./all/all.yml: - "{{ openstack_nova_pool }}"
./all/all.yml: - "{{ openstack_cinder_backup_pool }}"
./all/all.yml:openstack_keys:
./all/all.yml: - { name: client.glance, caps: { mon: "profile rbd", osd: "profile rbd pool={{ openstack_cinder_pool.name }}, profile rbd pool={{ openstack_glance_pool.name }}"}, mode: "0600" }
./all/all.yml: - { name: client.cinder, caps: { mon: "profile rbd", osd: "profile rbd pool={{ openstack_cinder_pool.name }}, profile rbd pool={{ openstack_nova_pool.name }}, profile rbd pool={{ openstack_glance_pool.name }}"}, mode: "0600" }
./all/all.yml: - { name: client.cinder-backup, caps: { mon: "profile rbd", osd: "profile rbd pool={{ openstack_cinder_backup_pool.name }}"}, mode: "0600" }
./all/all.yml: - { name: client.openstack, caps: { mon: "profile rbd", osd: "profile rbd pool={{ openstack_glance_pool.name }}, profile rbd pool={{ openstack_nova_pool.name }}, profile rbd pool={{ openstack_cinder_pool.name }}, profile rbd pool={{ openstack_cinder_backup_pool.name }}"}, mode: "0600" }
./all/all.yml:dashboard_enabled: True
./all/all.yml:dashboard_protocol: http
./all/all.yml:dashboard_admin_user: admin
./all/all.yml:dashboard_admin_password: password
./all/all.yml:grafana_admin_user: admin
./all/all.yml:grafana_admin_password: password

Environment:

OS (e.g. from /etc/os-release):NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Kernel (e.g. uname -a): Linux ceph-node-0 5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Docker version if applicable (e.g. docker version): Docker version 20.10.2, build 20.10.2-0ubuntu1~20.04.2
Ansible version (e.g. ansible-playbook --version):ansible-playbook 2.9.6
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/john/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Aug 24 2020, 17:57:11) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]

ceph-ansible version (e.g. git head or tag or stable branch): commit e6d8b058ba92fecdc78ee55b0dd8ce12c5120df0
Ceph version (e.g. ceph -v): ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)

@karasjoh000 that's probably because of this :
20 slow ops, oldest one blocked for 1609 sec, daemons [osd.36,osd.37,osd.46] have slow ops.

could you check the output of :
ceph pg dump
ceph osd dump

I encountered the same error.
group_vars/all.yml
ceph_origin: repository
ceph_repository: community
ceph_stable_release: pacific
public_network: "<IP_ADDRESS>/24"
radosgw_interface: eth0
monitor_interface: eth0
dashboard_enabled: true
containerized_deployment: false
devices:
  - '/dev/vdb'
  - '/dev/vdc'
  - '/dev/vdd'
  - '/dev/vde'
monitoring_group_name: monitoring
dashboard_admin_password: xxx
grafana_admin_password: xxx

/etc/os-release
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"

Full log: https://textbin.net/raw/1owjr4ovud
The solution was to do sudo ceph osd pool set (pool name) size 2 for each pool, doing this again each time a new pool was created. It seems ceph did not want to acknowledge the write until the required replication factor was decreased.
However, my ceph -s now looks like this:
sudo ceph -s
  cluster:
    id:     127318bb-031f-42c5-acc3-bc43e37ce230
    health: HEALTH_WARN
            mon is allowing insecure global_id reclaim
            Degraded data redundancy: 192/384 objects degraded (50.000%), 46 pgs degraded, 161 pgs undersized

  services:
    mon: 1 daemons, quorum ceph-mon0 (age 59m)
    mgr: ceph-mgr0(active, since 20m)
    osd: 4 osds: 4 up (since 56m), 4 in (since 56m)
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    pools:   6 pools, 161 pgs
    objects: 192 objects, 6.9 KiB
    usage:   28 MiB used, 400 GiB / 400 GiB avail
    pgs:     192/384 objects degraded (50.000%)
             115 active+undersized
             46  active+undersized+degraded

  io:
    client:   1.2 KiB/s rd, 1 op/s rd, 0 op/s wr

Since I have 4 OSDs, shouldn't the writes be able to go through at the default size of 3? Is this a bug or am I misunderstanding something?

could you share the output of ceph osd dump ?

epoch 77
fsid 127318bb-031f-42c5-acc3-bc43e37ce230
created 2021-06-03T20:19:45.629002+0000
modified 2021-06-04T00:03:12.171511+0000
flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
crush_version 9
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client luminous
min_compat_client jewel
require_osd_release pacific
stretch_mode_enabled false
pool 1 'device_health_metrics' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 77 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth
pool 2 '.rgw.root' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 50 flags hashpspool stripe_width 0 application rgw
pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 67 flags hashpspool stripe_width 0 application rbd
pool 4 'default.rgw.log' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 43 flags hashpspool stripe_width 0 application rgw
pool 5 'default.rgw.control' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 51 flags hashpspool stripe_width 0 application rgw
pool 6 'default.rgw.meta' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 pg_num_target 8 pgp_num_target 8 autoscale_mode on last_change 60 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw
max_osd 4
osd.0 up   in  weight 1 up_from 18 up_thru 67 down_at 0 last_clean_interval [0,0) [v2:<IP_ADDRESS>:6800/17411,v1:<IP_ADDRESS>:6801/17411] [v2:<IP_ADDRESS>:6802/17411,v1:<IP_ADDRESS>:6803/17411] exists,up ba392249-5421-4de0-971a-2233b17da392
osd.1 up   in  weight 1 up_from 18 up_thru 67 down_at 0 last_clean_interval [0,0) [v2:<IP_ADDRESS>:6808/18825,v1:<IP_ADDRESS>:6809/18825] [v2:<IP_ADDRESS>:6810/18825,v1:<IP_ADDRESS>:6811/18825] exists,up e476f828-cd73-40b7-a926-393c93a43558
osd.2 up   in  weight 1 up_from 18 up_thru 67 down_at 0 last_clean_interval [0,0) [v2:<IP_ADDRESS>:6816/20248,v1:<IP_ADDRESS>:6817/20248] [v2:<IP_ADDRESS>:6818/20248,v1:<IP_ADDRESS>:6819/20248] exists,up 1d7a1700-91f9-4822-9546-a54a26f908de
osd.3 up   in  weight 1 up_from 18 up_thru 68 down_at 0 last_clean_interval [0,0) [v2:<IP_ADDRESS>:6824/21663,v1:<IP_ADDRESS>:6825/21663] [v2:<IP_ADDRESS>:6826/21663,v1:<IP_ADDRESS>:6827/21663] exists,up 7f125387-6bae-4a4c-a188-511b0f0ba082
blocklist <IP_ADDRESS>:0/678357123 expires 2021-06-04T21:53:06.125600+0000
blocklist <IP_ADDRESS>:0/2528750619 expires 2021-06-04T21:53:06.125600+0000
blocklist <IP_ADDRESS>:0/3826862751 expires 2021-06-04T21:52:34.408413+0000
blocklist <IP_ADDRESS>:0/558637544 expires 2021-06-04T20:58:55.166815+0000
blocklist <IP_ADDRESS>:0/3912413307 expires 2021-06-04T21:52:34.408413+0000
blocklist <IP_ADDRESS>:0/137647918 expires 2021-06-04T20:21:26.797963+0000
blocklist <IP_ADDRESS>:6800/15427 expires 2021-06-04T20:21:26.797963+0000
blocklist <IP_ADDRESS>:6801/15427 expires 2021-06-04T20:21:26.797963+0000
blocklist <IP_ADDRESS>:0/128005271 expires 2021-06-04T20:21:26.797963+0000
blocklist <IP_ADDRESS>:0/908547403 expires 2021-06-04T20:28:16.266869+0000
blocklist <IP_ADDRESS>:0/2663080188 expires 2021-06-04T20:58:55.166815+0000
blocklist <IP_ADDRESS>:0/807399896 expires 2021-06-04T20:28:16.266869+0000

@HashFail does ceph-ansible branch correspond with the ceph_stable_release: pacific. See https://docs.ceph.com/projects/ceph-ansible/en/latest/index.html#releases. I do not have this error anymore and most likely that was the reason for it in my case.

@karasjoh000 yes, the commit was the head of stable-6.0 at the time.
I think I might have figured out what is behind this: as you can see from my ceph osd dump, all my OSDs are on the same host. Adding a second host seems to fix the issue.
Perhaps this is the expected behavior? Maybe when the pool size is greater than 2, ceph wants at least one of the OSDs to be on a different host?

@karasjoh000 yes, the commit was the head of stable-6.0 at the time.
I think I might have figured out what is behind this: as you can see from my ceph osd dump, all my OSDs are on the same host. Adding a second host seems to fix the issue.
Perhaps this is the expected behavior? Maybe when the pool size is greater than 2, ceph wants at least one of the OSDs to be on a different host?

yes, this is expected behavior, your pool are all size 2 and you only have 1 osd node. Hence the 161 pgs undersized


md-tabs which are not active doesn't respond to animation-triggers

Bug, feature request, or proposal:
md-tabs should show the correct animation state when navigated to.
What is the current behavior?
The tab which is not active initially does not show the correct animation state
What are the steps to reproduce?
Set an animation trigger inside a tab which is not active, the element will not show styles from the right state.
Providing a Plunker (or similar) is the best way to get the team to see your issue.
Plunker: http://plnkr.co/edit/QMN5tUzN1JWjmQWQu48H
What is the use-case or motivation for changing an existing behavior?
Better use of animations in tabs that are not active from the start.
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
"@angular/cli": "^1.2.0",
"@angular/material": "2.0.0-beta.8",
"@angular/core": "^4.3.0-beta.1",
"typescript": "~2.3.4",
Chrome
Is there anything else we should know?

#5269 is related to this.

Ah yes, sorry for not replying, this can be closed in favor of #5269 .


Fixed in #64


Using insecure Urls despite forceAdminURL being set

This comment considers the near-identical logic which appears in these three places:
core/server/api/authentication.js:49:                var baseUrl = config.forceAdminSSL ? (config.urlSSL || config.url) : config.url,
core/server/api/users.js:48:        var baseUrl = config.forceAdminSSL ? (config.urlSSL || config.url) : config.url;
core/server/api/mail.js:88:                siteUrl: config.forceAdminSSL ? (config.urlSSL || config.url) : config.url

In all cases, if forceAdminSSL is set but urlSSL is not, an insecure Url will be used, when the configuration option clearly requested to force SSL.
Why not update the validation of config.forceAdminSSL to require that  config.urlSSL also be set if config.forceAdminSSL is set?  Then all three locations above could be simplified to definitely  use config.urlSSL  when config.forceAdminSSL is set, instead of returning an insecure Url when the config requested that secure one be forced.

But if forceAdminSSL is set, a redirect will be issued in the event that a request comes in on a non-https link.  The "forcing" of SSL isn't necessarily by way of providing an SSL URL, it's in forcing the connection itself to be SSL.
I believe the intended use for urlSSL is if the secure URL differs from the non-secure URL, e.g., http://my-ghost-blog.com/ vs. https://secure.my-ghost-blog.com/.  If the only difference is in the protocol, a standard redirect will suffice and urlSSL isn't needed.
I'm going to close this but if anyone has more to add feel free to do so and I'm happy to re-open.


Change Sprite dynamically

In the old docs there was this function:
// could change sprite for anim if you don't use spritesheet
obj.changeSprite("froggy_left");

But it doesn't seem to work as of now. What is the best workaround?

Now it's obj.use(sprite("froggy_left")) to replace the sprite component.

Tysm! I was building a day-night-cycle and creating and hiding multiple elements was a pain!


PR title won't auto fill from last commit

Unlike #4701, I want GHPR auto fill  PR title from last commit. When there is more than 1 commit ahead of destination branch it will auto fill title from last commit. But it won't auto fill when there is only one commit to add in PR!

Extension version: 0.66.1
VSCode Version: 1.79.2
OS: Manjaro

Steps to Reproduce:

Click on create new PR
Change destination branch which is 1 commit behind from source branch

I'm having trouble reproducing this issue. Is there a step that I'm missing?

Checkout the default branch of my repo.
Checkout the n-1 commit from that branch
git checkout -b main2 , git push
git checkout -b alexr00/mergeIntoMain2
Make a change and commit
"Create Pull Request"
Change the into branch to main2

Looks like it works now. I close this issue and reopen it when it happened again. Thanks.

Hi again. Looks like there is a situation where the plugin fills the PR title, otherwise it doesn't.
But, I can't understand what is it!
Can you tell me how it works?


Wait till GrpCurl job is ready

removed debug resource which was added accidentally a long time ago

Test results:

TestExposeGrpcWitPassthroughGateway https://master-jenkins-csb-servicemesh.apps.ocp-c1.prod.psi.redhat.com/job/maistra/job/maistra-test-tool/3074/
TestExposeGrpcWithHttpsGateway https://master-jenkins-csb-servicemesh.apps.ocp-c1.prod.psi.redhat.com/job/maistra/job/maistra-test-tool/3075/

Looks good to me. Thanks


自定义图标后点击图标提示 runtime-core.esm-bundler.js:38 [Vue warn]: Component emitted event "handleCornerClick" but it is neither declared in the emits option nor as an "onHandleCornerClick" prop.

自定义图标后点击图标提示 runtime-core.esm-bundler.js:38 [Vue warn]: Component emitted event "handleCornerClick" but it is neither declared in the emits option nor as an "onHandleCornerClick" prop.

这种写法不规范，建议修改下

1.3.7 修复了 onHandleCornerClick


Bootstrap

Coverage decreased (-0.03%) to 83.657% when pulling 53e362441c98afb841053f021f187b8378bb0018 on tany:bootstrap into 9d716fc081de5dbe649f58266fd105ce67808e4a on shirasagi:master.


HTML5: this.rtenv.callMain is not a function when using latest-upstream backend

Added needed changes for normal compiling with emscripten 1.38.41 and later
Bugsquad edit: Fixes #31297.

Thanks for fixing https://github.com/godotengine/godot/issues/31297 :+1:

Thanks!

Cherry-picked for 3.1.2.

Cherry-picked for 3.0.7.


PHP 5.5 seems to be required for develop branch

I was able to install the dev branch on Laravel 5.0, but after enabling the service provider and running composer-update or dump-autoload I was getting an unexpected Class exception on line 56 of "vendor/mitchellvanw/laravel-doctrine/src/LaravelDoctrineServiceProvider.php". Turned out I needed to update my installed php version from 5.4 to 5.5 to get the error to go away. I noticed that in the Dec 4, 2014 commit (f2f8643e8c371d885b179b5227a69421ff4914ff) the composer.json minimum php version requirements were changed from 5.5 to 5.4, so maybe it needs to be changed back to 5.5 in the dev branch?

Yeah seems we just need to update composer. Easy done - thanks :)


Improved error handling

We currently have a FailedTransactionError, which should be subclassed into different error types. Also an enum with different error reasons should be provided by the CashScript SDK.

Added this in https://github.com/Bitcoin-com/cashscript/commit/6cf1b8da90cc71c73817554d549d2ce82f427036. Added FailedSigCheckError, FailedTimeCheckError and FailedRequireError. Also exported a Reason enum with all possible Script failure reason strings. In the future additional strings that are unrelated to Script failures can be added.


Support for cfunction + closures on Apple M1 chip

Hi everyone,
it seems that the @batch macro in combination with closures does not on the arm64-apple-darwin21.2.0 platform (Apple silicon). Any chance this could be fixed?
Here is a MWE:
using Polyester

function test1()
    a = 5
    function test2()
        return a
    end
    @batch for _ = 1:100
        test2()
    end
    return nothing
end

test1()

which produces this error message:
ERROR: cfunction: closures are not supported on this platform
Stacktrace:

batch_closure at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

macro expansion at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

#_batch_no_reserve#5 at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

(::Polyester.var"#_batch_no_reserve##kw")(::NamedTuple{(:threadlocal,), Tuple{Val{false}}}, ::typeof(Polyester._batch_no_reserve), ::var"#9#11"{var"#test2#10"{Int64}}, ::Tuple{UInt8}, ::Tuple{UInt32}, ::Tuple{UInt8}, ::UInt64, ::UInt64, ::UInt64, ::Static.StaticInt{1}, ::Static.StaticInt{1}, ::Polyester.NoLoop, ::Polyester.CombineIndices) at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

#batch#7 at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

batch at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

macro expansion at [~/.julia/packages/Polyester/4Q0NM/src/closure.jl]()

test1() at [./Untitled-1]()

top-level scope at [Untitled-1]()

ERROR: cfunction: closures are not supported on this platform
Stacktrace:

batch_closure at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

macro expansion at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

#_batch_no_reserve#5 at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

(::Polyester.var"#_batch_no_reserve##kw")(::NamedTuple{(:threadlocal,), Tuple{Val{false}}}, ::typeof(Polyester._batch_no_reserve), ::var"#9#11"{var"#test2#10"{Int64}}, ::Tuple{UInt8}, ::Tuple{UInt32}, ::Tuple{UInt8}, ::UInt64, ::UInt64, ::UInt64, ::Static.StaticInt{1}, ::Static.StaticInt{1}, ::Polyester.NoLoop, ::Polyester.CombineIndices) at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

#batch#7 at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

batch at [~/.julia/packages/Polyester/4Q0NM/src/batch.jl]()

macro expansion at [~/.julia/packages/Polyester/4Q0NM/src/closure.jl]()

test1() at [./Untitled-1]()

top-level scope at [Untitled-1]()

running on this system/Julia version:
julia> versioninfo()
Julia Version 1.8.0-beta1
Commit 7b711ce699 (2022-02-23 15:09 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.2.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 4 on 4 virtual cores
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 4

Thanks a bunch for looking into this!

I'm getting also this error on the first run in Julia 1.11, and if I re-rerun the same line a second time, it seems to work :/.  Does anyone has a workaround?

Interesting.
    @ Thunderbolt.OS ~/.julia/dev/Thunderbolt/src/solver/operator_splitting/solver.jl:36
 [20] construct_inner_cache
    @ ~/.julia/dev/Thunderbolt/src/solver/operator_splitting/solver.jl:30 [inlined]
 [21] construct_inner_cache(f::GenericSplitFunction{…}, alg::Thunderbolt.ReactionTangentController{…}; u0::Vector{…}, kwargs::@Kwargs{…})

    @ Thunderbolt.OS ~/.julia/dev/Thunderbolt/src/solver/operator_splitting/integrator.jl:75
 [26] __init
    @ ~/.julia/dev/Thunderbolt/src/solver/operator_splitting/integrator.jl:46 [inlined]
 [27] init_call(_prob::OperatorSplittingProblem{…}, args::Thunderbolt.ReactionTangentController{…}; merge_callbacks::Bool, kwargshandle::Nothing, kwargs::@Kwargs{…})
     Use `show(err)` to see complete types.

However, if I run the example twice it does not error :D


problem with .Net6.0.1

UABEA call me download .Net

but my PC have .Net 6.0.1

The dialog window does not make it super clear which version of the .NET 5 downloads to pick: console or desktop?

It turns out that the runtime for desktop apps is sufficient.


Feature/3 tokenomics related updates

Update Profile contract into Profile & KYC Tokens
Adding stake wallet and pending tokens for verifiers #3
Update the testings and deployment script

Keeping this on hold till the audit is completed, lets test internally and release as part of tranche 2.


Justified Issues Score

I have Justified some issue, but they still show the same score. This confuse me, becouse I though that after I justify the problem, it is acknowledge as something normal and no longer higher my score.
I would like to clean up my account with either fixing issues or justfying it, so and the end of the day I have a score or "zero". Then next day I can check what are new issues. Is there a way to achieve it ?
Thanks.

This was fixed in PR #825 with a very simple change to dart/lib/component/item_table_component/item_table_component.html
https://github.com/Netflix/security_monkey/pull/825/files#diff-66e6c0d6b02b84c8228f639fc3ec49d8
I suspect you're on an old codebase.  Can you confirm?

Hello,
Thanks for contact. Can you tell me how to check security monkey version
installed? I was using instruction from latest documentation:
http://securitymonkey.readthedocs.io/en/latest/quickstart.html#setup-on-aws-or-gcp
I copied code from developer branch (default):
sudo git clone --depth 1 --branch develop
https://github.com/Netflix/security_monkey.git
Which branch shall I use in order to have stable version?
Thanks for your time and helping me out :)
Michal
2017-11-04 6:29 GMT+01:00 Patrick Kelley<EMAIL_ADDRESS>
This was fixed in PR #825
https://github.com/Netflix/security_monkey/pull/825 with a very simple
change to dart/lib/component/item_table_component/item_table_
component.html
https://github.com/Netflix/security_monkey/pull/825/files#diff-
66e6c0d6b02b84c8228f639fc3ec49d8
I suspect you're on an old codebase. Can you confirm?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Netflix/security_monkey/issues/854#issuecomment-341873514,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AdDTEzYSPKiDwSOW69RV-gFlCo4rydenks5sy_Y-gaJpZM4QHcky
.

Hello,
I just verified the code changes in #825 and I have it applied already.
What I see is that I have justified some issues on 27.10.2017 in security
groups (400 points) and since then it still show up in Dashboard. So I have
securitygrups score 540 (140 new issues and 400 from justified issues).
Attaching screenshot from one justified issue.
Thanks:
Michal
2017-11-06 9:17 GMT+01:00 Michal Grzelak<EMAIL_ADDRESS>
Hello,
Thanks for contact. Can you tell me how to check security monkey version
installed? I was using instruction from latest documentation:
http://securitymonkey.readthedocs.io/en/latest/
quickstart.html#setup-on-aws-or-gcp
I copied code from developer branch (default):
sudo git clone --depth 1 --branch develop https://github.com/Netflix/security_monkey.git
Which branch shall I use in order to have stable version?
Thanks for your time and helping me out :)
Michal
2017-11-04 6:29 GMT+01:00 Patrick Kelley<EMAIL_ADDRESS>
This was fixed in PR #825
https://github.com/Netflix/security_monkey/pull/825 with a very simple
change to dart/lib/component/item_table_component/item_table_component
.html
https://github.com/Netflix/security_monkey/pull/825/files#
diff-66e6c0d6b02b84c8228f639fc3ec49d8
I suspect you're on an old codebase. Can you confirm?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Netflix/security_monkey/issues/854#issuecomment-341873514,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AdDTEzYSPKiDwSOW69RV-gFlCo4rydenks5sy_Y-gaJpZM4QHcky
.

Ohh, So you're seeing this on the Dashboard and not on the default Item Search page?

I see it on both dashboard and when i do search.
06.11.2017 19:25 "Patrick Kelley"<EMAIL_ADDRESS>napisał(a):

Ohh, So you're seeing this on the Dashboard and not on the default Item
Search page?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Netflix/security_monkey/issues/854#issuecomment-342239821,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AdDTEzxw8g1pORgFvqWgMeHtm_3e-yFHks5sz08MgaJpZM4QHcky
.

Are you compiling the dart yourself or downloading the static.tar.gz file from the last release?
Have you tried to open the Chrome Inspector and then right-click on the refresh button and go to "Empty Cache and Hard Reload" ?

I follow the instruction from latest doc i think dart is compiled there.
I used a few browser,  private modę, i even see justifued score when
exporting result to Excel.
06.11.2017 19:36 "Patrick Kelley"<EMAIL_ADDRESS>napisał(a):

Are you compiling the dart yourself or downloading the static.tar.gz file
from the last release?
Have you tried to open the Chrome Inspector and then right-click on the
refresh button and go to "Empty Cache and Hard Reload" ?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Netflix/security_monkey/issues/854#issuecomment-342243091,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AdDTEw3B5pvfT1tpiQJhq9OXj3NRDu75ks5sz1G8gaJpZM4QHcky
.

I also am seeing no change in score on the dashboard.  I pulled the develop branch and compiled the DART code as of yesterday (11/8/2017)

Ok. So you also see this problem? Do you know how to fix IT?
Thanks
Michal
09.11.2017 15:54 "Eric Kelson"<EMAIL_ADDRESS>napisał(a):
I also am seeing no change in score on the dashboard. I pulled the develop
branch and compiled the DART code as of yesterday (11/8/2017)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Netflix/security_monkey/issues/854#issuecomment-343178685,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AdDTE0olhQlk9zrMarYNiVhWrMhYEl5Mks5s0xI3gaJpZM4QHcky
.

The audit runs from the scheduler at hour 10 (for whatever timezone it is set on) daily, you are not seeing it change over days?

No. It didn't change for last week or more. I also trud to run schelduer
from command line with no efect.
I am really interested in sorting it out, as customer i support in aws
likes the Security Monkey and want to have it depliyed in production.
I can also share my config or live share desktop if that can help somehow.
Thank you.
Michal
09.11.2017 17:14 "Michael Stair"<EMAIL_ADDRESS>napisał(a):
The audit runs from the scheduler at hour 10 (for whatever timezone it is
set on) daily, you are not seeing it change over days?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Netflix/security_monkey/issues/854#issuecomment-343205356,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AdDTE1AN2KpEng_9UT_GBkWAUIZmWFQPks5s0yTkgaJpZM4QHcky
.

Oh sorry, this is justify vs "fix", let me check.

I built from the latest develop code and UI and confirmed I see same. I am looking into where justified issues get reflected in scoring.

Hello,
Were you able to find solution for this problem?
Thanks:
Michał
2017-11-10 19:43 GMT+01:00 Michael Stair<EMAIL_ADDRESS>
I built from the latest develop code and UI and confirmed I see same. I am
looking into where justified issues get reflected in scoring.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Netflix/security_monkey/issues/854#issuecomment-343553766,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AdDTEy81qMa67B4J4vgYGV_d0zHBAaprks5s1JlggaJpZM4QHcky
.

Hi mstair - any luck on finding where this is tabulating incorrectly?

Hi @mstair
I'm seeing an issue similar to this both with scores still showing as though the issue was not fixed or justified on the search results section. Compiling dart and using the codebase as recent as https://github.com/Netflix/security_monkey/pull/1091
I think this may be related to inconsistencies in the API responses between /api/1/items/itemNumberHERE and /api/1/items?searchparams
First example with justified issues:

Auditor finds issue with Item

then the issue is justified

Search for the justified issue by ARN, name, account etc.

the search triggers an API like this:
GET /api/1/items?accounts=EXAMPLE&accounttypes=&active=True&arns=&count=25&enabledonly=true&names=&page=1&regions=&technologies=iamrole.

results: of the API call like this:
{"count": 25, "items": [{"account": "EXAMPLE", "justification": null, "account_type": "AWS", "name": "EXAMPLEROLE", "technology": "iamrole", "issue": "Sensitive Permissions", "region": "universal", "score": 10, "notes": "Service [s3] Category: [Permissions] Resources: [\"*\"]", "item_links": [], "item_id": 33877, "justified": false, "fixed": false, "justified_date": "2018-04-20 20:03:19.462511", "id": 41874} ....
Note Justified shows false but a justified_date is present

Click on the particular Item within the search results. the detail page does the following:
API call to GET /api/1/items/33877

{"justification": "Required  to function", "issue": "Sensitive Permissions", "notes": "Service [iam] Category: [Permissions] Resources: [\"*\"]", "score": 10, "item_links": [{"id": 33877, "name": "REDEACTED"}], "item_id": 33877, "justified_user": "USER", "justified": true, "fixed": false, "justified_date": "2018-07-12 18:43:11.761536", "id": 93368}
Note the Justified: true
second example (fixed issues)

Auditor finds issue with Item
item issue is fixed but item is still present (ie not deleted)
Search results show old unjustified or unfixed scores.
GET /api/1/items?accounts=&accounttypes=&arns=ARNREDACTED&count=25&names=&page=1&regions=&technologies=
example:

{"count": 1, "items": [{"account": "EXAMPLE", "account_type": "AWS", "name": "ARNREDACTED", "region": "us-east-1", "unjustified_issue_score": 2, "num_issues": 1, "active": true, "issue_score": 2, "first_seen": "2018-05-11 16:28:16.715374", "technology": "lambda", "id": 40744, "last_seen": "2018-08-08 00:55:51.097552"}], "total": 1, "page": 1, "auth": {"authenticated": true, "user": "USER", "roles": [{"name": "Admin"}, {"name": "Justify"}, {"name": "Comment"}, {"name": "View"}]}}

clicking on item shows accurate score.
example:
GET /api/1/items/40744
returns

{"item": {"account": "EXAMPLE", "account_type": "AWS", "name": "REDACTED (ARNREDACTED)", "region": "us-east-1", "technology": "lambda", "id": 40744}, "revisions": [{"active": true, "date_created": "2018-08-08 00:55:51.097552", "date_last_ephemeral_change": null, "id": 176747, "item_id": 40744}, {"active": true, "date_created": "2018-05-11 16:28:16.715374", "date_last_ephemeral_change": null, "id": 59089, "item_id": 40744}], "auth": {"authenticated": true, "user": "USER", "roles": [{"name": "Admin"}, {"name": "Justify"}, {"name": "Comment"}, {"name": "View"}]}, "issues": [], "comments": []}
Note "issues": []
I haven't had time to research further but I think it is related to the ItemList class. https://github.com/Netflix/security_monkey/blob/develop/security_monkey/views/item.py#L143


eks:describe-cluster failures

Running collect using master (e611d2a05b462c33abb570bed2b1aae491b49fe0) gets this error in every region:
* Getting eks:describe-cluster info
  Making call for account-data/./aws-bard-development/eu-north-1/eks-describe-cluster.json
Exception: Parameter validation failed:
Missing required parameter in input: "name"

Looks like I forgot to do a list-clusters then describe-cluster with the values from that here: https://github.com/duo-labs/cloudmapper/blob/e611d2a05b462c33abb570bed2b1aae491b49fe0/collect_commands.yaml#L351
Currently, CloudMapper does not do anything with the info collected there. So you can comment out those lines in the collect_commands.yaml file as an interim fix.

Oh, that's a really nice mechanism for this — it made a patch trivial.

Affected me too.
Why this PR closed without merge to master branch?
I apply this patch, my collect process made 0 errors.

Hmm, I think what happened is 14 days ago I renamed the master branch to main in light of current societal issues being raised, and deleted the master branch.   I think in doing so it somehow resulted in closing the PR.  I don't think I can revive the PR, so I'll create a new PR myself unless @acdha wants to ensure he gets in the commit history by recreating a PR.

@0xdabbad00 I cherry-picked #742 for this in case you haven't already done so

Merged. Thank you @acdha !

That was impressively fast – thanks!


Custom registry does not seem supported

Describe your use case
I'd like to integrate cargo-semver-check workflow into CI at work where we use a custom crate registry to (internally) publish our crates.
Unfortunately, cargo-semver-check fails to run on a crate that is published on the custom registry:
❯ cargo semver-checks check-release --verbose
    Updating index
     Parsing mycrate v0.1.0 (current)
 Documenting mycrate v0.1.0 (~/mycrate.git)
    Finished dev [unoptimized + debuginfo] target(s) in 0.96s
Error: mycrate not found in registry

I suspect this is because it is not aware of the crate registry when trying to download the crate.
Describe the solution you'd like
cargo-semver-check should fetch the published package from the crate registry it has been published previously.
In my case, this information is located in the repo's .cargo/config, but I think that info can also be located elsewhere.
Alternatives, if applicable
No response
Additional Context
No response

Thanks for the detailed report!
I think we both want to add custom registry support, and in the meantime also provide a better, more actionable error message than the current one.
While the registry-based previous version lookup won't work with a custom registry, you should be able to use one of the other ways to tell cargo-semver-checks about the previous version to compare against. Excerpt from cargo semver-checks check-release --help:
      --baseline-rev <REV>
          Git revision to lookup for a baseline

      --baseline-root <MANIFEST_ROOT>
          Directory containing baseline crate source

      --baseline-rustdoc <JSON_PATH>
          The rustdoc json file to use as a semver baseline

I think all of those should be unaffected by the custom registry setup, and should able to unblock your current use case today. Please let us know if that works!

I'm adding an entry to the FAQ about using the other baseline generation options for custom registries until this issue is resolved: #165

This issue was "closed as completed", does this mean custom registries are now supported ? Otherwise, I would suggest leaving this issue open, as an indication the project could accept PRs to add support


Add Accord Project Tag to Cicero generator

The Cicero template generator creates a README.md for generated templates. Once we have an approved/agreed upon "Powered by Accord Project" badge, it would be nice if the Cicero generator automatically includes it in the README.md
See: https://github.com/accordproject/techdocs/issues/81

hello.can i take up this issue?
thanks!!

hello.can i take up this issue?
thanks!!

This issue only makes sense after https://github.com/accordproject/techdocs/issues/81 has been resolved.

Fixed in #535


File.binary? mistakes binary pfx files as text

As is shown in the image below of a snippet of a pfx file, this file is clearly binary but the File.binary? method says it's false/text.

I see the line ((s.size - s.grep(" ".."~").size) / s.size.to_f) > 0.30 and now understand it to mean that 30% of the bytes are "binary".
30% seems a little high for this check, at the very least maybe add the percentage as a second optional parameter to File.binary? so that the percentage may be customised if desired?

@ElvenSpellmaker Sure, I can do that.

@ElvenSpellmaker Best I can do for now: https://github.com/djberg96/ptools/pull/29

Excellent, thanks! =)


make TAG_OR_NIGHTLY include -<hash> if on tag

Also includes a few other improvements

implements janus version -dir flag to enable grabbing version outside of cwd
includes usage information about %B (experimental hybrid version number)
removes old unused brew Formula from repo; it was a byproduct of goreleaser defaults from initial setup and hasn't been in use since v0.1.2

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Throttling policies/rate limit not working across tenants

Description:
If one tenant publishes an API and another tenant subscribe it with a tier that is not present in its own policies, every call to the API returns an error (forbidden). If a tier is created in the subscriber tenant with the same name, the API call works fine.
Also, the rate limit applied is the one in the subscriber tier, not the one defined by the publisher.
Suggested Labels:
Type/bug
Suggested Assignees:
Affected Product Version:
2.2.0 update 4
OS, DB, other environment details and versions:
AWS Linux, Postgres 9.6
Steps to reproduce:

Create a tenant A
Create a new subscription throttling policy, e.g. "Tier 1"
Publish an API available to all tenants, with "Tier 1" enabled
Create a tenant B
Subscribe to it using "Tier 1"
Try to call the API

// rate limit problem

Create "Tier 1" in Tenant B with another rate limit than the one defined by tenant A for Tier 1
Redo the subscription to the API using this tier
The rate limit enforced is the one defined by tenant B, not by tenant A (the publisher of the API)

Related Issues:
n/a

I have the same issue here

Same here too.

here too.

Root cause Analyze.
The underlying query does not return any data from the subscription policy table. As a result, API returns an error 900908 (forbidden). In the query, There is an equality check for policy subscriber tenant id and subscriber tenant id which fail in this scenario.

Developer testing record.
Before

### After

Rate test
CrossThrottling.mov.zip


HistoryBuffer Add pop_oldest

Motivation
There currently is no interface for removing elements from the HistoryBuffer.
Added

HistoryBuffer.pop_oldest()
HistoryBuffer.filled() getter to replace filled member variable
Unit test for HistoryBuffer.pop_oldest()

Changed

HistoryBuffer.len() to be a getter rather than computed on call.
HistoryBuffer.write()
HistoryBuffer.recent()
HistoryBuffer.oldest_ordered()
OldestOrdered.next()

Warning
HistoryBuffer.as_slice() no longer has the same meaning. The slice returned may contain unused values.

what's the use case for this? It seems to me there's a lot of overlap with Deque here. IMO if you need push/pop you should use Deque, and HistoryBuffer should be used only when you want to do "statistics" on the last N values, not as a queue you pop things from.


After converting 4 wallets, the wallet is stuck on initializing

Repro steps

Create 4 wallets on revine, each with 1 TFT
Lock all 4
Start stellar wallet

Expected Result
All converted and wallet main screen is shown
Actual Result
After restarting the app, wallets were converted and shown correctly, but I was stuck in initializing for over 3 minutes before I Just killed the app.

System Info

When trying to convert accounts, all calls are serialized on the conversion service. This resulting in slow calls and long load time https://github.com/threefoldfoundation/tft-stellar/issues/122


No such file or folder for /usr/local/bin/download_snapshot.sh

Running .setup.sh ran into this error:
Downloading near chain snapshot
Fetching... this can take some time...
[FATAL tini (7)] exec /usr/local/bin/download_snapshot.sh failed: No such file or directory

Please make sure to use the latest version of the docker images.


registration key is not scoped to an account.  host uuid also are not

Closed in favor of rancher/rancher#3848


Laravel can’t handle the pagination with the union query

Laravel Version: 5.5.13
PHP Version: 7.0.23
Database Driver MySql & Version: 5.7

Description:
right now there is a problem with pagination when you use the $query->union() function with a subsequent $query->paginate(). You will get an error saying SQLSTATE[21000]: Cardinality violation: 1222 The used SELECT statements have a different number of columns
Laravel can’t handle the pagination with the union query, as the first SELECT that you have will be altered to be count(*) AS aggregate but the second one will not, resulting in a different number of columns and thus throwing an error.
Steps To Reproduce:
$a = Order::where(['a' => 2])->whereIn('status', [3, 5]);
$b = Order::where(['b' => 1])->whereIn('status', [3, 5]);
$result =  $a->unionAll($b)->->paginate(15);

This is a known limitation and we're open to solutions for the issue, check https://github.com/laravel/framework/issues/2968#issuecomment-33741816

@themsaid : I lack coding skills to fix it, after lots of research I found a package which seems to solve this issue & I confirm it's working on Laravel 5.5:
https://github.com/krossroad/laravel-unionpaginator
I found one more script but haven't tested yet:
https://github.com/SMATAR/laravel-union-paginator
I hope this bug will be solved soon, probably in next release ;)

Use this one instead until bug is fixed.
It's worked for now.
https://itsolutionstuff.com/post/how-to-add-pagination-with-union-in-laravel-4-and-laravel-5

@Porrapat : The solution you suggested has a big drawback. It will first fetch all the records from the table and then perform the union+pagination. This will crash or will get considerably slow if a table has 50k+ records. Check this library, it solves the issue exactly as needed:
https://github.com/krossroad/laravel-unionpaginator

@akhileshdarjee Thank you, but, unfortunately, my project using Laravel 5.2. So, I am not sure it work with my current code with Laravel 5.2 yet.

@akhileshdarjee The second thing is my query is build from query builder but your suggest library use for Eloquent only. So, how about solution that can work with Laravel 5.2 or less and use query builder.

I found this one. Maybe, someone have to test it.
https://github.com/kaizer666/LaravelUnionPaginator

Tested!
This one solve my problems.
https://github.com/kaizer666/LaravelUnionPaginator

simply use new instance of paginaton and pass your variable to that

I got a solution!
In class:
$todos = (new Todo())
            ->select('todos.files', 'u.name', 'todos.created_at')
            ->join('users AS u', 'u.id', '=', 'todos.user_id')
            ->whereNotNull('files');

        $comments = (new Comment())
            ->select('comments.files', 'u.name', 'comments.created_at')
            ->join('users AS u', 'u.id', '=', 'comments.user_id')
            ->whereNotNull('files')
            ->union($todos)
            ->get()
            ->toArray();

        $currentPage = $request->get('page')?:1;
        $perPage = 1;
        $currentItems = array_slice($comments, $perPage * ($currentPage - 1), $perPage);
        $files = new LengthAwarePaginator($currentItems, count($comments), $perPage, $currentPage, ['path' => 'files']);

        return view('views', ['files' => $files]);

On blade:
{{$files}}
@foreach($files as $file)
      {{$file['name']}}
@endforeach

I got a solution!
In class:
        $todos = (new Todo())
            ->select('todos.files', 'u.name', 'todos.created_at')
            ->join('users AS u', 'u.id', '=', 'todos.user_id')
            ->whereNotNull('files');

        $comments = (new Comment())
            ->select('comments.files', 'u.name', 'comments.created_at')
            ->join('users AS u', 'u.id', '=', 'comments.user_id')
            ->whereNotNull('files')
            ->union($todos)
            ->get()
            ->toArray();

        $currentPage = $request->get('page')?:1;
        $perPage = 1;
        $currentItems = array_slice($comments, $perPage * ($currentPage - 1), $perPage);
        $files = new LengthAwarePaginator($currentItems, count($comments), $perPage, $currentPage, ['path' => 'files']);

        return view('views', ['files' => $files]);

On blade:
{{$files}}
@foreach($files as $file)
        {{$file['name']}}
@endforeach

I got a solution!
In class:
        $a = Order::where(['a' => 2])->whereIn('status', [3, 5]);
        $b = Order::where(['b' => 1])->whereIn('status', [3, 5]);
        $union = $a->union($b)->get()->toArray();

        $currentPage = $request->get('page')?:1;
        $perPage = 1;
        $currentItems = array_slice($union, $perPage * ($currentPage - 1), $perPage);
        $result = new LengthAwarePaginator($currentItems, count($union), $perPage, $currentPage, ['path' => 'files']);

        return view('views', ['result' => $result]);

On blade:
{{$result}}
{{$result->links('something_here_or_delete')}}


AMBARI-24572 Fixing the cluster name alignment

What changes were proposed in this pull request?
Alignment of the cluster name with other elements in the top nav when cluster name is slightly longer.

How was this patch tested?
Manually tested.

@atkach @aBabiichuk Please review

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3873/
Test PASSed.

@aBabiichuk what additional changes would be required? and what are the other special characters that you mentioned?

@sjanardhanhw I can't remember exactly. But generally speaking, just decreasing the left margin might not address the issue for all possible cases (since non-monotype font is used for cluster name, and different char combination would have different widths). Also, negative margin might potentially cause intersection with breadcrumbs block.
Actually it would be better to look at arrangement of children of <div class="container main-container">, since the issue is caused by insufficient width for cluster name element.

Thanks for the pointer @aBabiichuk I've updated the PR with change in the appropriate place. Please review

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/4502/
Test FAILed.
Test FAILured.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/4503/
Test PASSed.


Wrong version of VCLib Redist listed

In the manually update section, the following is noted:

You may need to install the Desktop Bridge VC++ v14 Redistributable Package and the associated Microsoft.VCLibs.140.00.UWPDesktop package. This should only be necessary on older builds of Windows if you get an error about missing framework packages.

After installing the the bridge package on a fresh VM, the appx bundle that is in C:\Program Files (x86)\Microsoft SDKs\Windows Kits\10\ExtensionSDKs\Microsoft.VCLibs.Desktop\14.0\Appx\Retail\x64 is 14,.0.24217. The minimum required version appears to be 14.0.27810.0

Hi, just want to note this also happened to me today. If I install the VCLibs package dowloaded from here, it installs a version that is too old for the latest winget version (https://github.com/microsoft/winget-cli/releases/tag/v1.1.12653, from yesterday).
If I instead try to install https://github.com/microsoft/winget-cli/releases/tag/v1.0.11692, from June 24, then it works. So maybe the issue is just that the VCLibs packages available at https://docs.microsoft.com/troubleshoot/cpp/c-runtime-packages-desktop-bridge#how-to-install-and-update-desktop-framework-packages take a while to be updated to the latest version ?

Thanks. I am following up with Visual Studio.

Greetings.  The VCLibs packages has been updated.  Thanks for reporting.


Get quality of image ?

I use this lib to transform and convert to webp if possible, but on low quality jpeg it output high quality webp, sometime heavier.
Is it possible to add a getQuality() method on jpeg/webp, and the ability to save to webp from jpeg with the same quality (in order to have always a lighter file) ?

I would love to integrate this but according to my research, the coding quality is sometimes but not always stored in the image. At least for Jpeg.
Unfortunately, there are also further hurdles with GD, which unfortunately does not offer any functions for reading out the quality. Imagick does have identifyImage() and getCompressionQuality(), but these do not always deliver the desired result.


Can't play in Safari browser

When play in Safari, the player controller will always display "Loading". From the Network panel, we can see the flv file is continue downloading, but no video is displayed.

Minimum required Safari version is 10.0. The MSE implementation on Safari 9.x is complete broken.
HTTP FLV live stream is not supported on Safari because of the lost of stream IO.

OK. Got it. Seems I face the problem because I'm using live stream.
On Nov 2, 2016 7:56 PM, "xqq"<EMAIL_ADDRESS>wrote:

Minimum required Safari version is 10.0. The MSE implementation on Safari
9.x is complete broken.
HTTP FLV live stream is not supported on Safari because of the lost of
stream IO.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Bilibili/flv.js/issues/22#issuecomment-258054948, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAB8EddOC9BGO22REfWaP9wO7tTt4bf0ks5q6U1OgaJpZM4Kn5gD
.

You can test flv live stream under Safari Technology Preview which supports fetch + stream API.
https://developer.apple.com/safari/technology-preview/
But It cannot be considered completely stable for now.


Collection.modify uses incorrect iteration method

In the line
https://github.com/dfahlander/Dexie.js/blob/4d44e73c88e828cc6b8521ce93a5adaf9ede7f38/src/classes/collection/collection.ts#L499
for…in is used to iterate over an array, which is not how you're supposed to iterate over object keys in JavaScript.
I'm using a framework (ember.js) that injects extra keys into the Object prototype. Using the correct Object.getOwnPropertyNames(failures), this isn't an issue, since it's not part of the failures object itself. However, with the code shown above, it iterates over the index “_super” and inserts it into the totalFailures array, even though there are actually no failures.

here you can see how that looks in Chrome's devtools. Note how failures is empty and pos is _super.

Digging deeper, the underlying issue seems to be that failures is an array at runtime, but TypeScript is told that it's an object containing number indices here:
https://github.com/dfahlander/Dexie.js/blob/4d44e73c88e828cc6b8521ce93a5adaf9ede7f38/src/public/types/dbcore.d.ts#L36
This causes TypeScript to generate the wrong JavaScript output (it just uses the same code verbatim instead of iterating over the indices instead).

Thanks! I suppose the immediate fix would be to change the code in collection.ts to do a for-of Object.keys() instead. Also, the failures object is instantiated as an array in dbcore-indexeddb.ts while it really should be an object representing map between numbers and errors.

I have implemented a fix here: https://github.com/anlumo/Dexie.js/tree/issue-947-collection.modify
I don't know whether it's clean, but it works so far.


relabelled Mendelian to hereditary disease

Addresses #5539

@nicolevasilevsky
I did not create new terms for the child terms (e.g. monogenic/Mendelian disease) as I think we will need to review which terms should be classified under these child terms in the same release. We could look into this for April.

sounds good!


Need clarification about the dispose() method

After looking at the code to understand a bug mentioned in another issue I found some strange things about the operation of the dispose() method.
I would like to understand its precise objective :

Should this function reset the various properties to a default value so that the various methods ( like setAutoCollectExceptions ) can be recalled to redefine the property values?
Should this function stop all "watches" ( track request, track dependency calls, ... ) in order to allow a new instantiation of each property ( _console, _exceptions, _performance, ... ) without edge effects / duplicates?

I ask this because currently the description of the function suggests that it only resets the properties to default values when in fact it prepares for a new instantiation (it sets defaultClient to null and therefore it is necessary to call setup() for the monitoring to work again except that in this function we create a new instance for each property ( _console, _exceptions, _performance, ... ))

Should stop all automatic tracking and remove all telemetry processors that were added, setup need to be called again after calling dispose, not sure what you mean by "watches" we patch a lot of code and dispose is not actually removing these patches, we should not generate telemetry though(#946), about the actual internal implementation, there is a bunch of static classes so dispose doesn't mean we are actually cleaning up everything from memory

not sure what you mean by "watches"
I've called "watches" : tracking and telemetry processor ( I didn't know how to name both at the same time 😂 )

In fact, the current operation allowed me to notice some problems more or less related to the operation of the "dispose()" method. For example, for the automatic collection of HTTP requests you overwrite the "http(s).createServer" prototype. The problem is that if we call the .setup method after calling http(s).createServer, we won't take advantage of the automatic collection. So I was wondering why you didn't overwrite the emit prototype of Server (Server.prototype.emit) to take advantage of the collection even if .setup is called after http(s).createServer...
Maybe I should create a separate ticket

Yes you are right, all monkey patching including the one we do for native http modules will not work if application insights is not the first package to be loaded, we added some messages in readme mentioning this, is pretty common for customers to reach to us because there is no telemetry even if they configure the SDK correctly, unfortunately there is no way to work around this unless the library we want to generate telemetry for provides some kind of API to hook into.

Yes I understand but I was wondering if trying to override the emit function to intercept when there was an emit('request', ...) call could work but apparently not...
The problem with the current management (I think it can be patched but not sure) is that if you call setup() then create the server (http.createServer()) and finally call .dispose() followed by .setup() the http requests are not traced anymore. This is partly because the old instance of _serverRequests was overwritten (when calling .dispose() followed by .setup()) but the new one is waiting for the call to a http(s).createServer to register a new listener. Why not keep the old one and implement it in such a way that it works for several instances of 'AutoCollectHttpRequests'?
Currently, the old listener is still running ( created with the first .setup().setAutoCollectRequests(true) ) but he will not track requests because the .dispose() set isEnabled to false on his instance

Oh I see what you are saying, ideally we use similar approach as we do for other libraries like bunyan or mongodb, where a publisher patch the events and send the event to all subscribers, in this case the specific client is being patched instead, so if you create the server in between the other client is gone, I will consider that quite the edge case though, we are currently working on updating the SDK to use OpenTelemetry internally, instrumentators actually patch and unpatch the code when disabled, it is good because patched code is actually removed but the issue you mention will remain.

Ok I see.
On the other hand, is it possible to propose for this version a patch so that trackRequests are not stopped if you do a dispose and a setup after calling createServer? For example, this code would allow to continue to record the requests even if a dispose() and a setup() are called after the createServer ( the library still must be instanciated the first time before the createServer ) :
import http = require("http");
import https = require("https");
import url = require("url");

import Contracts = require("../Declarations/Contracts");
import TelemetryClient = require("../Library/TelemetryClient");
import Logging = require("../Library/Logging");
import Util = require("../Library/Util");
import RequestResponseHeaders = require("../Library/RequestResponseHeaders");
import HttpRequestParser = require("./HttpRequestParser");
import { CorrelationContextManager, CorrelationContext, PrivateCustomProperties } from "./CorrelationContextManager";
import AutoCollectPerformance = require("./Performance");

class AutoCollectHttpRequests {

    public static INSTANCE: AutoCollectHttpRequests;
    public static READY: boolean = false;

    private static alreadyAutoCollectedFlag = "_appInsightsAutoCollected";

    private _client: TelemetryClient;
    private _isEnabled: boolean;
    private _isInitialized: boolean;
    private _isAutoCorrelating: boolean;

    constructor(client: TelemetryClient) {
        if (!!AutoCollectHttpRequests.INSTANCE) {
            throw new Error("Server request tracking should be configured from the applicationInsights object");
        }

        AutoCollectHttpRequests.INSTANCE = this;
        this._client = client;
    }

    public enable(isEnabled: boolean) {
        this._isEnabled = isEnabled;

        // Autocorrelation requires automatic monitoring of incoming server requests
        // Disabling autocollection but enabling autocorrelation will still enable
        // request monitoring but will not produce request events
        if ((this._isAutoCorrelating || this._isEnabled || AutoCollectPerformance.isEnabled()) && !this._isInitialized) {
            this.useAutoCorrelation(this._isAutoCorrelating);
            this._initialize();
        }
    }

    public useAutoCorrelation(isEnabled: boolean, forceClsHooked?: boolean) {
        if (isEnabled && !this._isAutoCorrelating) {
            CorrelationContextManager.enable(forceClsHooked);
        } else if (!isEnabled && this._isAutoCorrelating) {
            CorrelationContextManager.disable();
        }
        this._isAutoCorrelating = isEnabled;
    }

    public isInitialized() {
        return this._isInitialized;
    }

    public isAutoCorrelating() {
        return this._isAutoCorrelating;
    }

    private _generateCorrelationContext(requestParser: HttpRequestParser): CorrelationContext {
        if (!this._isAutoCorrelating) {
            return;
        }

        return CorrelationContextManager.generateContextObject(
            requestParser.getOperationId(this._client.context.tags),
            requestParser.getRequestId(),
            requestParser.getOperationName(this._client.context.tags),
            requestParser.getCorrelationContextHeader(),
            requestParser.getTraceparent(),
            requestParser.getTracestate()
        );
    }

    private registerRequest(request: http.ServerRequest, response: http.ServerResponse, onRequest: Function) {
        // Set up correlation context
        const requestParser = new HttpRequestParser(request);
        const correlationContext = this._generateCorrelationContext(requestParser);

        // Note: Check for if correlation is enabled happens within this method.
        // If not enabled, function will directly call the callback.
        CorrelationContextManager.runWithContext(correlationContext, () => {
            if (this._isEnabled) {
                // Mark as auto collected
                (<any>request)[AutoCollectHttpRequests.alreadyAutoCollectedFlag] = true;

                // Auto collect request
                AutoCollectHttpRequests.trackRequest(this._client, { request: request, response: response }, requestParser);
            }

            if (typeof onRequest === "function") {
                onRequest(request, response);
            }
        });
    }

    private _initialize() {
        this._isInitialized = true;

        if (AutoCollectHttpRequests.READY) {
            return;
        }

        const wrapOnRequestHandler: Function = (onRequest?: Function) => {
            if (!onRequest) {
                return undefined;
            }
            if (typeof onRequest !== "function") {
                throw new Error("onRequest handler must be a function");
            }
            return (request: http.ServerRequest, response: http.ServerResponse) => {
                CorrelationContextManager.wrapEmitter(request);
                CorrelationContextManager.wrapEmitter(response);
                const shouldCollect: boolean = request && !(<any>request)[AutoCollectHttpRequests.alreadyAutoCollectedFlag];

                if (request && shouldCollect) {
                    AutoCollectHttpRequests.INSTANCE?.registerRequest(request, response, onRequest)
                } else {
                    if (typeof onRequest === "function") {
                        onRequest(request, response);
                    }
                }
            }
        }

        // The `http.createServer` function will instantiate a new http.Server object.
        // Inside the Server's constructor, it is using addListener to register the
        // onRequest handler. So there are two ways to inject the wrapped onRequest handler:
        // 1) Overwrite Server.prototype.addListener (and .on()) globally and not patching
        //    the http.createServer call. Or
        // 2) Overwrite the http.createServer method and add a patched addListener to the
        //    fresh server instance. This seems more stable for possible future changes as
        //    it also covers the case where the Server might not use addListener to manage
        //    the callback internally.
        //    And also as long as the constructor uses addListener to add the handle, it is
        //    ok to patch the addListener after construction only. Because if we would patch
        //    the prototype one and the createServer method, we would wrap the handler twice
        //    in case of the constructor call.
        const wrapServerEventHandler: Function = (server: (http.Server | https.Server)) => {
            const originalAddListener = server.addListener.bind(server);
            server.addListener = (eventType: string, eventHandler: Function) => {
                switch (eventType) {
                    case "request":
                    case "checkContinue":
                        return originalAddListener(eventType, wrapOnRequestHandler(eventHandler));
                    default:
                        return originalAddListener(eventType, eventHandler);
                }
            };
            // on is an alias to addListener only
            server.on = server.addListener;
        }

        const originalHttpServer: any = http.createServer;

        // options parameter was added in Node.js v9.6.0, v8.12.0
        // function createServer(requestListener?: RequestListener): Server;
        // function createServer(options: ServerOptions, requestListener?: RequestListener): Server;
        http.createServer = (param1?: Object, param2?: Function) => {
            // todo: get a pointer to the server so the IP address can be read from server.address
            if (param2 && typeof param2 === "function") {
                const server: http.Server = originalHttpServer(param1, wrapOnRequestHandler(param2));
                wrapServerEventHandler(server);
                return server;
            }
            else {
                const server: http.Server = originalHttpServer(wrapOnRequestHandler(param1));
                wrapServerEventHandler(server);
                return server;
            }
        }

        const originalHttpsServer = https.createServer;
        https.createServer = (options: https.ServerOptions, onRequest?: Function) => {
            const server: https.Server = originalHttpsServer(options, wrapOnRequestHandler(onRequest));
            wrapServerEventHandler(server);
            return server;
        }

        AutoCollectHttpRequests.READY = true;
    }

    /**
     * Tracks a request synchronously (doesn't wait for response 'finish' event)
     */
    public static trackRequestSync(client: TelemetryClient, telemetry: Contracts.NodeHttpRequestTelemetry) {
        if (!telemetry.request || !telemetry.response || !client) {
            Logging.info("AutoCollectHttpRequests.trackRequestSync was called with invalid parameters: ", !telemetry.request, !telemetry.response, !client);
            return;
        }

        AutoCollectHttpRequests.addResponseCorrelationIdHeader(client, telemetry.response);

        // store data about the request
        var correlationContext = CorrelationContextManager.getCurrentContext();
        var requestParser = new HttpRequestParser(telemetry.request, (correlationContext && correlationContext.operation.parentId));

        // Overwrite correlation context with request parser results
        if (correlationContext) {
            correlationContext.operation.id = requestParser.getOperationId(client.context.tags) || correlationContext.operation.id;
            correlationContext.operation.name = requestParser.getOperationName(client.context.tags) || correlationContext.operation.name;
            correlationContext.operation.parentId = requestParser.getRequestId() || correlationContext.operation.parentId;
            (<PrivateCustomProperties>correlationContext.customProperties).addHeaderData(requestParser.getCorrelationContextHeader());
        }

        AutoCollectHttpRequests.endRequest(client, requestParser, telemetry, telemetry.duration, telemetry.error);
    }

    /**
     * Tracks a request by listening to the response 'finish' event
     */
    public static trackRequest(client: TelemetryClient, telemetry: Contracts.NodeHttpRequestTelemetry, _requestParser?: HttpRequestParser) {
        if (!telemetry.request || !telemetry.response || !client) {
            Logging.info("AutoCollectHttpRequests.trackRequest was called with invalid parameters: ", !telemetry.request, !telemetry.response, !client);
            return;
        }

        // store data about the request
        var correlationContext = CorrelationContextManager.getCurrentContext();
        var requestParser = _requestParser || new HttpRequestParser(telemetry.request, correlationContext && correlationContext.operation.parentId);

        if (Util.canIncludeCorrelationHeader(client, requestParser.getUrl())) {
            AutoCollectHttpRequests.addResponseCorrelationIdHeader(client, telemetry.response);
        }

        // Overwrite correlation context with request parser results (if not an automatic track. we've already precalculated the correlation context in that case)
        if (correlationContext && !_requestParser) {
            correlationContext.operation.id = requestParser.getOperationId(client.context.tags) || correlationContext.operation.id;
            correlationContext.operation.name = requestParser.getOperationName(client.context.tags) || correlationContext.operation.name;
            correlationContext.operation.parentId = requestParser.getOperationParentId(client.context.tags) || correlationContext.operation.parentId;
            (<PrivateCustomProperties>correlationContext.customProperties).addHeaderData(requestParser.getCorrelationContextHeader());
        }

        // response listeners
        if (telemetry.response.once) {
            telemetry.response.once("finish", () => {
                AutoCollectHttpRequests.endRequest(client, requestParser, telemetry, null, null);
            });
        }

        // track a failed request if an error is emitted
        if (telemetry.request.on) {
            telemetry.request.on("error", (error: any) => {
                AutoCollectHttpRequests.endRequest(client, requestParser, telemetry, null, error);
            });
        }

        // track an aborted request if an aborted event is emitted
        if (telemetry.request.on) {
            telemetry.request.on("aborted", () => {
                const errorMessage = "The request has been aborted and the network socket has closed.";
                    AutoCollectHttpRequests.endRequest(client, requestParser, telemetry, null, errorMessage);
            });
        }
    }

    /**
     * Add the target correlationId to the response headers, if not already provided.
     */
    private static addResponseCorrelationIdHeader(client: TelemetryClient, response: http.ServerResponse) {
        if (client.config && client.config.correlationId &&
            response.getHeader && response.setHeader && !(<any>response).headersSent) {
            const correlationHeader = <any>response.getHeader(RequestResponseHeaders.requestContextHeader);
            Util.safeIncludeCorrelationHeader(client, response, correlationHeader);
        }
    }

    private static endRequest(client: TelemetryClient, requestParser: HttpRequestParser, telemetry: Contracts.NodeHttpRequestTelemetry, ellapsedMilliseconds?: number, error?: any) {
        if (error) {
            requestParser.onError(error, ellapsedMilliseconds);
        } else {
            requestParser.onResponse(telemetry.response, ellapsedMilliseconds);
        }

        var requestTelemetry = requestParser.getRequestTelemetry(telemetry);

        requestTelemetry.tagOverrides = requestParser.getRequestTags(client.context.tags);
        if (telemetry.tagOverrides) {
            for (let key in telemetry.tagOverrides) {
                requestTelemetry.tagOverrides[key] = telemetry.tagOverrides[key];
            }
        }

        const legacyRootId = requestParser.getLegacyRootId();
        if (legacyRootId) {
            requestTelemetry.properties["ai_legacyRootId"] = legacyRootId;
        }

        requestTelemetry.contextObjects = requestTelemetry.contextObjects || {};
        requestTelemetry.contextObjects["http.ServerRequest"] = telemetry.request;
        requestTelemetry.contextObjects["http.ServerResponse"] = telemetry.response;

        client.trackRequest(requestTelemetry);
    }

    public dispose() {
        AutoCollectHttpRequests.INSTANCE = null;
        this.enable(false);
        this._isInitialized = false;
        CorrelationContextManager.disable();
        this._isAutoCorrelating = false;
    }
}

export = AutoCollectHttpRequests;

@Apokalypt  Sorry is hard to see what the differences are here, feel free to submit a PR

@Apokalypt  Sorry is hard to see what the differences are here, feel free to submit a PR

I will make one, no problem 👍


Update Composer dependencies (2023-04-08-00-11)

Visual regression test passed!
View the visual regression test report


Serialize trigger in LogReport

Some triggers including IntervalTrigger have serialize method. LogReport should call this method in serialization.

OK. LGTM.


Game crashes when I open

Whenever I press the delete key the console begins to load and then my among us crashes

Version of the game, version of the client ?

Version of the game, version of the client ?

v2020.10.22s and AUCC 0.0.3-beta.
I am having this issue as well.

the latest versions of the game are not supported yet.
You need 2020.9.22s or older to use the client
Sorry for the inconvenience

the latest versions of the game are not supported yet.
You need 2020.9.22s or older to use the client
Sorry for the inconvenience

Yup, I know, that's why I didn't open an issue. I just saw this issue was opened and responded with the version that it happens with.


[BUG] Openseach stores the fields when I set the "store: false" parameter

Describe the bug
Using the example in https://opensearch.org/docs/latest/quickstart/#create-an-index-and-field-mappings-using-sample-data, I've modified the mapping of the customer_first_name field with the parameter store: false but when I query the index, the field customer_first_name  is still returned in the _source.
To Reproduce
Steps to reproduce the behavior:
1 - Follow the instructions in https://opensearch.org/docs/latest/quickstart/#create-an-index-and-field-mappings-using-sample-data
2 - Modify the customer_first_name field in mapping file https://raw.githubusercontent.com/opensearch-project/documentation-website/2.4/assets/examples/ecommerce-field_mappings.json like so:
...
      "customer_first_name" : {
        "type": "text",
        "store": false,    <--- added this
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256,
            "store": false <--- and this just to make sure
          }
        }
      },
...

3 - Add the bulk data as shown in the tutorial
4 - Perform the query using the customer_first_name as shown in the tutorial.
Expected behavior
We should be able to perform the query, but the field customer_first_name should not be part of the _source.
Plugins
The default on the docker-compose of the tutorial, but I've disabled the SSL plugins on all nodes and dashboard.
Host/Environment (please complete the following information):

OS: Windows 10 - WSL2 Ubuntu 20.04 - Docker
Windows Version 10.0.19044
Docker Version 20.10.18
Opensearch Version 2.4.1

That would be a bug. Want to try implementing a (failing) unit test for it?

Sorry I would like to, but I have no knowledge of java. The tests that I did were purely REST calls.

Scratch what I said. Where did you find the documentation for "store"?

I didn't see the docs. I know the configuration from Elasticsearch, so I assumed it should work the same here, also since Opensearch API doesn't support other "random" parameters, I again assumed that this was a supported parameter.
Skimming through the Opensearch Docs I can't find the parameter, but in the Elasticsearch docs there is this https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-store.html.

We forked at 7.10, so the relevant equivalent docs would be https://www.elastic.co/guide/en/elasticsearch/reference/7.10/mapping-store.html. There's definitely code that seems to evaluate store options, but I'm not familiar with the feature and the syntax you're using may have been added in subsequent versions of Elasticsearch.
Next steps would be to dive into the code and see what's up, document current behavior properly, and open feature requests for anything that's not supported. I think this is domain of @macohen, but please don't let me stop from doing any of this @filipetavares, and thank you for your time here!

Sorry, I think I missed something. If the version docs that you sent me does contain the store flag and the documentation states that, by default, the fields are not stored unless we specify that we want to store them, we should not see any field in _source unless we set store: true, but again the tutorial doesn't specify the store parameter and every field is stored, or at least is returned in the _source.
My main question is, is there anyway to only index fields without storing them?

@filipetavares all fields by default are indexed without being stored, but the _source field is a bit different.
The _source field is special. _source gives you a single field with the entire document before it is indexed, but OpenSearch only stores it. It's not analyzed or indexed. To pull any specific data out of that field, I believe you need to pull the json and parse it yourself. There are ways to disable it if you're concerned about disk space, although it's not recommended (https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html).
If you take the string "The quick brown fox jumped over the Lazy Dog. Hmm!" and you created a field for it called "thing" with "store": true, OpenSearch would store the entire string in the "thing" field. If you leave store=false, and used the standard analyzer (https://opensearch.org/docs/latest/opensearch/query-dsl/text-analyzers/) at index time, the analyzer would lowercase all the words, remove the punctuation, remove stop words (like "the" if you specify), and indexes the document so you could search with "lazy" and find this document in the returned set.
I hope that helps clarify things a bit. Can you confirm if this explanation helps or if I'm misunderstanding the issue?

@filipetavares I'm closing this. Feel free to re-open or create a new issue if this didn't address your question. I hope we were able to help.


How can I use multiple GPUs' video memory?

I try to run multimer for a 3200aa homotetramer which has four of peptide chain with 2 * A100 40G. It consumes all video memory od one of the A100 and about 72GB memory. At the same time, the other A100 doesn't seem to be working and its video memory usage is close to zero. It tooks about 10 hours to complete the first model.
And I meet the same problem when run multimer in the same 3200aa homotetrame whether in 4 * V100 32G or 4 * P40.

I've observed the same, adding GPUs for a job with memory requirements that surpass the available memory in one GPU, doesn't reduce the computing time. Unified Memory uses the RAM from one GPU plus system's RAM, but not form other GPUs.
In this web page:
https://elearning.bits.vib.be/courses/alphafold/lessons/alphafold-on-the-hpc/topic/computational-limits/
they state that :
"AlphaFold supports multi-GPU usage, which means that when using two GPUs, the GPU memory can be almost doubled. Like this, longer sequences can be processed on a cluster with limited individual GPU memory. Note that you should increase the number of allocated CPU cores when allocating multiple GPUs."
but I haven't been able to reproduce it.

I think you are seeing this maybe? I noticed that the bottleneck seems to be relaxation and not the unrelaxed prediction:
When two GPUs are used, both identical model / mem size, with gpu relaxation enabled, each running its own folding with respective gpu ids - while the unrelaxed prediction seems to run smoothly without interference, when one of the gpus came off to do a very long relaxation (e.g. large protein size), it seem to always make the other gpu wait for it in the relaxation phase, until it finishes and then both will be released back to run unrelaxed gpu prediction - thus creating a bottleneck by the longer relaxation. Note that the “stuck” only seem to happen once the shorter task GPU goes into relaxation, because if it is still in unrelaxed prediction, it will stay on to finish that without problem. So there it seems like they both have independent unrelaxed prediction but converge on some bottleneck in the relaxation part, more pronounced if one of them is doing larger protein; small proteins seem unnoticeable.
Best,
David

Although this does not address the potential bottleneck issue mentioned above, personally I set these environment variables to allow a single AlphaFold prediction to pool the memory across multiple available GPUs.
TF_FORCE_UNIFIED_MEMORY=1
XLA_PYTHON_CLIENT_MEM_FRACTION=4.0
Although, normally those variables are assigned in run_docker.py here, so they shouldn't have to be set manually when using Docker. You can potentially increase the value of XLA_PYTHON_CLIENT_MEM_FRACTION since that may be limiting the memory pool: #149

Hi @epenning, the bottleneck problem is very interesting. Setting the TF_FORCE_UNIFIED_MEMORY=0 or even using the ENABLE_GPU_RELAX=False still gives the same bottle neck, so it is not necessarily GPU or GPU memory related it seems. It seems to be when the system enters a stage of single thread processing, that there is some resource queuing there. Are you seeing the same issue yourself? Best, David.

@davidyanglee I actually haven't observed the issue you described yet myself, but only because lately I haven't been running any large proteins in parallel with relaxation enabled, so I haven't encountered the circumstances which might produce that bottleneck.

@epenning hello, I have set TF_FORCE_UNIFIED_MEMORY=1 and XLA_PYTHON_CLIENT_MEM_FRACTION=4.0, but the same thing happens

It actually turns out that I have the same problem too, even with TF_FORCE_UNIFIED_MEMORY=1 and increased XLA_PYTHON_CLIENT_MEM_FRACTION value. It seems like the higher XLA_PYTHON_CLIENT_MEM_FRACTION let AlphaFold use more of a single GPU's memory which made me think I had solved the problem, but it's still not using the memory from the other GPUs. If the sequence is too long it crashes with an out of memory error.
Wed Jun  8 15:36:27 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.86       Driver Version: 470.86       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro RTX 5000     On   | 00000000:02:00.0 Off |                  Off |
|  0%   48C    P2    75W / 230W |  16122MiB / 16125MiB |    100%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Quadro RTX 5000     On   | 00000000:03:00.0 Off |                  Off |
|  0%   32C    P8    10W / 230W |    120MiB / 16125MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Quadro RTX 5000     On   | 00000000:82:00.0 Off |                  Off |
|  0%   32C    P8     2W / 230W |    120MiB / 16125MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Quadro RTX 5000     On   | 00000000:83:00.0 Off |                  Off |
|  0%   31C    P8    10W / 230W |    120MiB / 16125MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

@epenning , thanks for verifying. On my machine, I also get the same problem no matter if I set TF_FORCE_UNIFIED_MEMORY=0 or 1, or create separate venv, or use 2 different containers. Intriguingly, the hanging problem for me only occurs during the long relaxation step and only when I use gpu relax. In fact, during the long relaxation step, if I start another task with a different gpu specified, it will hang at the no tpu found (normal) step, until the long relaxation exists. The problem mostly resolves (no interference with the shorter task) when the relaxation for the long task is set to use CPU only (ok to leave the shorter task with GPU relax). However, that is not really a solution as CPU relaxation takes longer, and even longer with lengthy proteins. So there seems to be a bottle neck / limiting resource during a long gpu relaxation step.

I tried to understand what's happening with the multiple GPU memory issue, since I have a large complex I'm trying to predict which is taking much longer than expected. I'm not very knowledgeable in this area though, and did not find any solution. In case it's useful to the AlphaFold team or anyone else dealing with this issue, here's what I did find:

CUDA Unified Memory manages a common memory space including all of the GPU and system memory.
Because AlphaFold can only utilize one GPU, once that GPU's memory is filled, the process expands to use system memory. So effectively, using a high value of XLA_PYTHON_CLIENT_MEM_FRACTION just increases the amount of system memory available for use.
When that GPU tries to use data stored in system memory, it will not be as fast as if it was stored in the GPU memory. It seems like when the GPU tries to access data stored in system memory, that data gets migrated back to the GPU to be accessed? Maybe this could cause a significant slowdown in AlphaFold prediction time.
If migrating data from one GPU to another is faster than migrating from CPU to GPU, then I would think prediction time could be improved if the memory in the other GPUs was used instead of system memory.
Several sources indicate that it is not necessarily trivial to get Unified Memory to preferentially use the other GPUs instead of the system memory: Nvidia Forums, Tensorflow
Tensorflow is probably responsible for managing this behavior, so it's unclear whether AlphaFold can even fix this issue without significant effort.

So my takeaway so far is that this might be expected behavior? I'd love to be corrected though, so I'll wait and see if anyone from AlphaFold has a better response!

I tried to understand what's happening with the multiple GPU memory issue, since I have a large complex I'm trying to predict which is taking much longer than expected. I'm not very knowledgeable in this area though, and did not find any solution. In case it's useful to the AlphaFold team or anyone else dealing with this issue, here's what I did find:

CUDA Unified Memory manages a common memory space including all of the GPU and system memory.
Because AlphaFold can only utilize one GPU, once that GPU's memory is filled, the process expands to use system memory. So effectively, using a high value of XLA_PYTHON_CLIENT_MEM_FRACTION just increases the amount of system memory available for use.
When that GPU tries to use data stored in system memory, it will not be as fast as if it was stored in the GPU memory. It seems like when the GPU tries to access data stored in system memory, that data gets migrated back to the GPU to be accessed? Maybe this could cause a significant slowdown in AlphaFold prediction time.
If migrating data from one GPU to another is faster than migrating from CPU to GPU, then I would think prediction time could be improved if the memory in the other GPUs was used instead of system memory.
Several sources indicate that it is not necessarily trivial to get Unified Memory to preferentially use the other GPUs instead of the system memory: Nvidia Forums, Tensorflow
Tensorflow is probably responsible for managing this behavior, so it's unclear whether AlphaFold can even fix this issue without significant effort.

So my takeaway so far is that this might be expected behavior? I'd love to be corrected though, so I'll wait and see if anyone from AlphaFold has a better response!

Hi, I also met this problem, have you solved it? Can you tell me how to deal with it?

Hi, I am not an expert here. For me at least the hangup generally occurs during the relaxation part. So I just perform the unrelaxed and the relaxed part separately. And I run into little problem of hang up and delay this way. David.

Hi, I am not an expert here. For me at least the hangup generally occurs during the relaxation part. So I just perform the unrelaxed and the relaxed part separately. And I run into little problem of hang up and delay this way. David.

Thanks for your reply, I solved this problem by change the XLA_PYTHON_CLIENT_MEM_FRACTION number to 2.

Hi @xlminfei, the provided implementation of AlphaFold can not be split across multiple GPUs (model parallelism). Note most of the time taken to get the first prediction will be MSA search. See a similar issue - https://github.com/deepmind/alphafold/issues/673. Note that the relax stage can also take a long time, and will take even longer when not running on gpu. @abiadak, please note that https://elearning.bits.vib.be/courses/alphafold/lessons/alphafold-on-the-hpc/topic/computational-limits/ is not official documentation and therefore not necessarily correct.


Support TFRecord(Tensorflow) as a DataSet

Description
The TFRecord format in Tensorflow is a simple format for storing a sequence of binary records. A tutorial is here.
Can we support TFRecord in Kedro?
Context
We deal with a lot of images. So Kedro supporting this format would be really useful(and make it easier to sell it our data scientists :) )
If there is an alternative, please let me know. Thanks

Thanks for raising this @RAbraham! TFRecord looks like it would be a great addition to Kedro. We'd recommend creating a custom dataset implementation and creating a PR for Kedro.
At this point, we only support a TensorflowDataset so this could make a lovely addition. I'll mark this ticket with the Request: Help Wanted tag to indicate that we're looking for contributions.


Icons for Map

Added icons for the map in several flavors, changed default icons being drawn to some of the custom ones. Need to add logic to pick correct icon for circumstance.
Added next-images to be able to load pngs.

Added logic, making seperate PR that includes icons and logic, and is on a properly named branch.


Long, single-line pfm_title can cause pfm_platform to not appear

It appears in situations where a pfm_title is long, but doesn't go over one line, the pfm_platform won't appear.  However, if it is longer than two lines it will display.

Example for first highlighted preference in above image:
<dict>
	<key>pfm_default</key>
	<true/>
	<key>pfm_description</key>
	<string></string>
	<key>pfm_description_reference</key>
	<string>Optional. If false, documents in managed apps and accounts only open in other managed apps and accounts. Default is true. Availability: Available only in iOS 7.0 and later.</string>
	<key>pfm_ios_min</key>
	<string>7.0</string>
	<key>pfm_name</key>
	<string>allowOpenFromManagedToUnmanaged</string>
	<key>pfm_platforms</key>
	<array>
		<string>iOS</string>
	</array>
	<key>pfm_title</key>
	<string>Allow documents from managed sources in unmanaged destinations</string>
	<key>pfm_type</key>
	<string>boolean</string>
</dict>

Appears to be sliding through and not wrapping:

Resolved in latest 0.2.5 beta build

Fixed in 0.2.5 release


refactor: Drop unused total local variables

total local variables are set, modified but never consumed.

ACK 11a1e25c8107fd17099e3cfcc9896d2477c614b8


Add test coverage to validate the functionality of the get_cluster function

Issue link

https://issues.redhat.com/browse/RHOAIENG-16104
What changes have been made

Added the test scenarios for get_cluster function to fetch cluster details and job submission
Verification steps

Testing
Run the tests mnist_raycluster_sdk_oauth_test and mnist_raycluster_sdk_kind_test.py by cloning this branch and verify newly added tests scenarios are validating correctly
poetry run pytest -v -s ./tests/e2e/mnist_raycluster_sdk_oauth_test.py
Checks

[ ] I've made sure the tests are passing.
Testing Strategy

[ ] Unit tests
[ ] Manual tests
[ ] Testing is not required for this change

Going to test this when my cluster is up and after the above is resolved I would be happy to merge :+1:


Experimental setting

What about training from ImageNet-pretrained weights rather than COCO-pretrained weights? Will the performance degrade significantly?

Yes, the method is built on a well-trained detector and then choose the optimal operator in terms of the complexity-precision trade-off. It is not trivial to make it converge on the imagenet pretrained models.

Oh, I see. Thanks for your reply and great work!