Discussion: Python performance

(close this if it seems too open-ended or inappropriate for a GH issue)
I wanted to post one of my attempts in analyzing custodian performance in order to stir some conversation about what could be improved performance-wise.
After generating flame graphs for different policy runs, it became evident that there's not really a whole lot of room for improvement in the code. What is really noticeable though is the 77% idle time for the policy run that I've specified below.
I know that many people run custodian under c7n-org which already parallelizes runs across accounts but I wanted to open this up for discussion for the times we have to run a policy against only one account.
The SVG of the flame graph (couldn't embed SVG filetype):
https://gist.github.com/alfredgamulo/d9aa50da2b8051665b74b55ba8a5c48f
Policy and output for reference:
policies:
  - name: ec2-unencrypted
    resource: ec2
    filters:
      - type: ebs
        key: Encrypted
        value: false

(cloud-custodian) ubuntu@ip-xxx-xxx-xxx-xxx:~$ ~/pyflame/src/pyflame -o ec2.perf -t custodian run --dryrun -v  policy.yml -s /tmp/.
2018-04-04 15:43:21,025: custodian.cli:DEBUG using default region:us-east-1 from boto
2018-04-04 15:43:21,521: custodian.commands:DEBUG Loaded file policy.yml. Contains 1 policies
2018-04-04 15:43:21,693: custodian.output:DEBUG Storing output with <DirectoryOutput to dir:/tmp/./ec2-unencrypted>
2018-04-04 15:43:21,693: custodian.policy:DEBUG Running policy ec2-unencrypted resource: ec2 region:us-east-1 c7n:<IP_ADDRESS>
2018-04-04 15:44:05,705: custodian.cache:DEBUG Using cache file /home/ubuntu/.cache/cloud-custodian.cache
2018-04-04 15:44:09,789: custodian.resources.ebs:WARNING event ids not resolved: ['vol-xxxxxxxxxxxxxxxxx'...] error:An error occurred (InvalidVolume.NotFound) when calling the DescribeVolumes operation: The volume 'vol-xxxxxxxxxxxxxxxxx' does not exist.
2018-04-04 15:44:10,101: custodian.resources.ebs:WARNING event ids not resolved: ['vol-xxxxxxxxxxxxxxxxx'...] error:An error occurred (InvalidVolume.NotFound) when calling the DescribeVolumes operation: The volume 'vol-xxxxxxxxxxxxxxxxx' does not exist.
2018-04-04 15:44:32,250: custodian.resources.ebs:WARNING event ids not resolved: ['vol-xxxxxxxxxxxxxxxxx'...] error:An error occurred (InvalidVolume.NotFound) when calling the DescribeVolumes operation: The volume 'vol-xxxxxxxxxxxxxxxxx' does not exist.
2018-04-04 15:44:32,586: custodian.resources.ebs:WARNING event ids not resolved: ['vol-xxxxxxxxxxxxxxxxx'...] error:An error occurred (InvalidVolume.NotFound) when calling the DescribeVolumes operation: The volume 'vol-xxxxxxxxxxxxxxxxx' does not exist.
2018-04-04 15:44:37,427: custodian.resources.ec2:DEBUG Filtered from 4896 to 15 ec2
2018-04-04 15:44:37,428: custodian.policy:INFO policy: ec2-unencrypted resource:ec2 region:us-east-1 count:15 time:75.73
2018-04-04 15:44:37,428: custodian.output:DEBUG metric:ResourceCount Count:15 policy:ec2-unencrypted restype:ec2 scope:policy
2018-04-04 15:44:37,433: custodian.policy:DEBUG dryrun: skipping actions
(cloud-custodian) ubuntu@ip-xxx-xxx-xxx-xxx:~$ FlameGraph/flamegraph.pl < ec2.perf > ec2.svg

One option building off this would be setting up a custom mapping that accounts for potential API collisions in cases like this (EC2/ASG, ALB/ELB, etc.) and reference that when spinning up new policy execution threads. This would require more overhead when adding new resource types though, since they would need compared against that list as well, to possibly be included.

per the profile, it should be apparent the biggest performance gains can be had by minimizing api calls. As aws apis evolve, there are always opportunities for that. As two examples from that last week that could dramatically reduce api calls during policy execution.

https://aws.amazon.com/about-aws/whats-new/2018/04/amazon-ebs-adds-support-for-tagging-ebs-snapshots-upon-creation-and-resource-level-permissions/

https://aws.amazon.com/about-aws/whats-new/2018/03/resource-groups-tagging-api-now-supports-13-additional-aws-services

We run C7N across hundreds of accounts and pretty much every region - so performance matters. It's been a while since I did our proper performance work, but I thought I'd add some notes on my findings here as well:

For us, cross-region calls (we run our code from ap-southeast-2) meant certain services were quite expensive / the latency adds up - particularly in the case of the SSL initialisation (our profiling at the time showed the cost of creating the SSL connection was more than noticeable, and keep alive was important)
You'd need pretty strong cross-type dependency management for per-resource threading - Lots of filters are still pretty ad hoc (e.g. making API calls directly) - and so that could be messier.
There are lots of cases where the high variability of resources an account can really, really slow it down - e.g. lack of caching in a given filter for an API call, etc - can really add up - but most of those are on a as-you-come-across it basis (and the way they're done originally is much clearer than the optimised version - if that makes sense).

@kapilt's note around minimising API calls is a huge one - even parallelising them in threads etc, you're still going to have a more significant improvement using a batch endpoint (as they're introduced) vs. a more smartly scheduling how HTTP stuff is done - especially when the nature of boto etc  means there are going to be edge cases and missed items on our side. Starting with auditing cached resources etc (there is still a few places I'm aware of that aren't caching API calls) - and revisiting some of the older improvements - would be a good place to start.
I'll see about getting some more production profiles from C7N over the next week - we run it primarily in a read-heavy (e.g. no actions, all filters) manner, but thats quite interesting to see - and we typically do thousands of runs of C7N (each with often more than a hundred policies per run) a day - across a wide variety of environments (e.g. thousands of EBS snapshots vs none. Lots of distributed s3 buckets vs. large numbers in one location) - patterns that would be interesting to analyse from a performance edge case perspective.
Pardon the kind of ranty / rambly reply :)


BW Error when running 601 Node run

When running a 601 Node Swift/T run analyzing 1200 samples (with two tasks running in parallel on each node), a failure occurred. There was a truncated alignment file, SRR1293800, that was detected.

Swift/T 601 Node BW run investigation:
An error in a samtools view -c command killed a task and then the swift/t run itself.
Job ID: 7403567.bw
The file truly is truncated; SRR1293800 is the ID.
The file sizes for all of the aligned bams and when they finished are below:
-rw------- 1 jacobrh ILL_baib 0 Aug 29 14:23 SRR1247973.noDedups.bam
-rw------- 1 jacobrh ILL_baib 0 Aug 29 14:23 SRR1248960.noDedups.bam
-rw------- 1 jacobrh ILL_baib 0 Aug 29 14:23 SRR1259003.noDedups.bam
-rw------- 1 jacobrh ILL_baib 0 Aug 29 14:23 SRR1283156.noDedups.bam
-rw------- 1 jacobrh ILL_baib 0 Aug 29 14:23 SRR1287757.noDedups.bam
-rw------- 1 jacobrh ILL_baib 0 Aug 29 14:23 SRR1291478.noDedups.bam
-rw------- 1 jacobrh ILL_baib 0 Aug 29 14:23 SRR1302319.noDedups.bam
-rw------- 1 jacobrh ILL_baib 0 Aug 29 14:23 SRR1422593.noDedups.bam
-rw------- 1 jacobrh ILL_baib 0 Aug 29 14:23 SRR1463272.noDedups.bam
-rw------- 1 jacobrh ILL_baib 0 Aug 29 14:23 SRR1533547.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.1G Aug 29 14:23 SRR1501853.noDedups.bam
-rw------- 1 jacobrh ILL_baib 128M Aug 29 14:23 SRR1280055.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.2G Aug 29 14:23 SRR1186145.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.2G Aug 29 14:23 SRR1260656.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.2G Aug 29 14:23 SRR1267708.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.2G Aug 29 14:23 SRR1284028.noDedups.bam
-rw------- 1 jacobrh ILL_baib 12G Aug 29 14:23 SRR1293800.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.2G Aug 29 14:23 SRR1422619.noDedups.bam
-rw------- 1 jacobrh ILL_baib 12M Aug 29 14:23 SRR1276009.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.3G Aug 29 14:23 SRR1186172.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.3G Aug 29 14:23 SRR1287816.noDedups.bam
-rw------- 1 jacobrh ILL_baib 14G Aug 29 14:19 SRR1275961.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.4G Aug 29 14:23 SRR1210815.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.4G Aug 29 14:23 SRR1222904.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.4G Aug 29 14:23 SRR1280559.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.4G Aug 29 14:23 SRR1313052.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.4G Aug 29 14:23 SRR1427543.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.4G Aug 29 14:23 SRR1533495.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.4G Aug 29 14:23 SRR1533543.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.5G Aug 29 14:23 SRR1287779.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.5G Aug 29 14:23 SRR1303897.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.5G Aug 29 14:23 SRR1423668.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.6G Aug 29 14:23 SRR1248980.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.6G Aug 29 14:23 SRR1250622.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.6G Aug 29 14:23 SRR1289675.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.6G Aug 29 14:23 SRR1304080.noDedups.bam
-rw------- 1 jacobrh ILL_baib 172M Aug 29 14:23 SRR1275553.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.7G Aug 29 14:23 SRR1280045.noDedups.bam
-rw------- 1 jacobrh ILL_baib 188M Aug 29 14:23 SRR1533472.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.8G Aug 29 14:23 SRR1279231.noDedups.bam
-rw------- 1 jacobrh ILL_baib 1.9G Aug 29 14:23 SRR1531508.noDedups.bam
-rw------- 1 jacobrh ILL_baib 203M Aug 29 14:23 SRR1533537.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.1G Aug 29 14:23 SRR1257620.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.1G Aug 29 14:23 SRR1501879.noDedups.bam
-rw------- 1 jacobrh ILL_baib 235M Aug 29 14:23 SRR1263110.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.3G Aug 29 14:23 SRR1303852.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.3G Aug 29 14:23 SRR1533476.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.3G Aug 29 14:23 SRR1533511.noDedups.bam
-rw------- 1 jacobrh ILL_baib 243M Aug 29 14:23 SRR1279479.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.4G Aug 29 14:23 SRR1232209.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.4G Aug 29 14:23 SRR1248235.noDedups.bam
-rw------- 1 jacobrh ILL_baib 251M Aug 29 14:23 SRR1253922.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.5G Aug 29 14:23 SRR1262881.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.5G Aug 29 14:23 SRR1501718.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.6G Aug 29 14:23 SRR1198054.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.6G Aug 29 14:23 SRR1247993.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.6G Aug 29 14:23 SRR1254226.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.6G Aug 29 14:23 SRR1275881.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.6G Aug 29 14:23 SRR1303858.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.6G Aug 29 14:23 SRR1533549.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.6G Aug 29 14:23 SRR1533573.noDedups.bam
-rw------- 1 jacobrh ILL_baib 279M Aug 29 14:23 SRR1302250.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.7G Aug 29 14:23 SRR1198018.noDedups.bam
-rw------- 1 jacobrh ILL_baib 287M Aug 29 14:23 SRR1239432.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.8G Aug 29 14:23 SRR1222847.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.8G Aug 29 14:23 SRR1275825.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.9G Aug 29 14:23 SRR1264786.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.9G Aug 29 14:23 SRR1267637.noDedups.bam
-rw------- 1 jacobrh ILL_baib 2.9G Aug 29 14:23 SRR1291474.noDedups.bam
-rw------- 1 jacobrh ILL_baib 307M Aug 29 14:23 SRR1279557.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.0G Aug 29 14:23 SRR1239395.noDedups.bam
-rw------- 1 jacobrh ILL_baib 323M Aug 29 14:23 SRR1463284.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.2G Aug 29 14:23 SRR1262876.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.3G Aug 29 14:23 SRR1267348.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.3G Aug 29 14:23 SRR1280571.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.3G Aug 29 14:23 SRR1286152.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.5G Aug 29 14:23 SRR1276134.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.6G Aug 29 14:23 SRR1248029.noDedups.bam
-rw------- 1 jacobrh ILL_baib 379M Aug 29 14:23 SRR1335748.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.7G Aug 29 14:23 SRR1271936.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.7G Aug 29 14:23 SRR1302544.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.8G Aug 29 14:23 SRR1222759.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.8G Aug 29 14:23 SRR1275659.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.8G Aug 29 14:23 SRR1279058.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.9G Aug 29 14:23 SRR1268849.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.9G Aug 29 14:23 SRR1269049.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.9G Aug 29 14:23 SRR1279696.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.9G Aug 29 14:23 SRR1284026.noDedups.bam
-rw------- 1 jacobrh ILL_baib 3.9G Aug 29 14:23 SRR1336764.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.0G Aug 29 14:23 SRR1267877.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.1G Aug 29 14:23 SRR1463308.noDedups.bam
-rw------- 1 jacobrh ILL_baib 426M Aug 29 14:23 SRR1289679.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.2G Aug 29 14:23 SRR1284149.noDedups.bam
-rw------- 1 jacobrh ILL_baib 438M Aug 29 14:23 SRR1263135.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.3G Aug 29 14:23 SRR1210921.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.3G Aug 29 14:23 SRR1275815.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.4G Aug 29 14:23 SRR1290405.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.4G Aug 29 14:23 SRR1501889.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.4G Aug 29 14:23 SRR1533496.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.5G Aug 29 14:23 SRR1262925.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.5G Aug 29 14:23 SRR1285060.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.5G Aug 29 14:23 SRR1531509.noDedups.bam
-rw------- 1 jacobrh ILL_baib 470M Aug 29 14:23 SRR1247942.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.7G Aug 29 14:23 SRR1260711.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.7G Aug 29 14:23 SRR1291375.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.8G Aug 29 14:23 SRR1262869.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.8G Aug 29 14:23 SRR1284014.noDedups.bam
-rw------- 1 jacobrh ILL_baib 4.8G Aug 29 14:23 SRR1291446.noDedups.bam
-rw------- 1 jacobrh ILL_baib 5.0G Aug 29 14:23 SRR1287763.noDedups.bam
-rw------- 1 jacobrh ILL_baib 514M Aug 29 14:23 SRR1291566.noDedups.bam
-rw------- 1 jacobrh ILL_baib 5.1G Aug 29 14:23 SRR1248137.noDedups.bam
-rw------- 1 jacobrh ILL_baib 5.1G Aug 29 14:23 SRR1284992.noDedups.bam
-rw------- 1 jacobrh ILL_baib 5.2G Aug 29 14:23 SRR1303935.noDedups.bam
-rw------- 1 jacobrh ILL_baib 5.5G Aug 29 14:23 SRR1250702.noDedups.bam
-rw------- 1 jacobrh ILL_baib 566M Aug 29 14:23 SRR1198179.noDedups.bam
-rw------- 1 jacobrh ILL_baib 605M Aug 29 14:23 SRR1222883.noDedups.bam
-rw------- 1 jacobrh ILL_baib 605M Aug 29 14:23 SRR1279148.noDedups.bam
-rw------- 1 jacobrh ILL_baib 6.0G Aug 29 14:23 SRR1302532.noDedups.bam
-rw------- 1 jacobrh ILL_baib 613M Aug 29 14:23 SRR1260627.noDedups.bam
-rw------- 1 jacobrh ILL_baib 6.1G Aug 29 14:23 SRR1280310.noDedups.bam
-rw------- 1 jacobrh ILL_baib 6.2G Aug 29 14:23 SRR1290401.noDedups.bam
-rw------- 1 jacobrh ILL_baib 637M Aug 29 14:23 SRR1232136.noDedups.bam
-rw------- 1 jacobrh ILL_baib 637M Aug 29 14:23 SRR1248972.noDedups.bam
-rw------- 1 jacobrh ILL_baib 6.3G Aug 29 14:23 SRR1533539.noDedups.bam
-rw------- 1 jacobrh ILL_baib 6.5G Aug 29 14:23 SRR1275745.noDedups.bam
-rw------- 1 jacobrh ILL_baib 661M Aug 29 14:23 SRR1271929.noDedups.bam
-rw------- 1 jacobrh ILL_baib 669M Aug 29 14:23 SRR1222749.noDedups.bam
-rw------- 1 jacobrh ILL_baib 697M Aug 29 14:23 SRR1254284.noDedups.bam
-rw------- 1 jacobrh ILL_baib 72M Aug 29 14:23 SRR1279133.noDedups.bam
-rw------- 1 jacobrh ILL_baib 7.3G Aug 29 14:23 SRR1291597.noDedups.bam
-rw------- 1 jacobrh ILL_baib 7.3G Aug 29 14:23 SRR1295897.noDedups.bam
-rw------- 1 jacobrh ILL_baib 745M Aug 29 14:23 SRR1222826.noDedups.bam
-rw------- 1 jacobrh ILL_baib 8.1G Aug 29 14:23 SRR1267732.noDedups.bam
-rw------- 1 jacobrh ILL_baib 876M Aug 29 14:23 SRR1291456.noDedups.bam
-rw------- 1 jacobrh ILL_baib 888M Aug 29 14:23 SRR1210916.noDedups.bam
-rw------- 1 jacobrh ILL_baib 888M Aug 29 14:23 SRR1291538.noDedups.bam
-rw------- 1 jacobrh ILL_baib 8.8G Aug 29 14:23 SRR1279909.noDedups.bam
-rw------- 1 jacobrh ILL_baib 9.0G Aug 29 14:23 SRR1279623.noDedups.bam
-rw------- 1 jacobrh ILL_baib 928M Aug 29 14:23 SRR1291492.noDedups.bam
-rw------- 1 jacobrh ILL_baib 96M Aug 29 14:23 SRR1289721.noDedups.bam
-rw------- 1 jacobrh ILL_baib 9.7G Aug 29 14:23 SRR1210934.noDedups.bam

It appears that the job was killed at 14:23. A tracejob on this ID confirms this
Starttime:  08/29/2017 14:02:35
Endtime:   08/29/2017 14:24:40
Only one alignment appears to have been completed:
-rw------- 1 jacobrh ILL_baib 14G Aug 29 14:19 SRR1275961.noDedups.bam
This file is not truncated
Another point to note is that although we requested 601 nodes when analyzing 1200 samples with 2 "programs" per node (Verified that is was written correctly in the runfile),
only around 140 alignment files are present in the output folder

Maybe I am not using enough TURBINE_SERVERS to coordinate the execution of so many tasks (the default is 1 I believe).
0.000 WORK TYPES: WORK
0.000 WORKERS: 1201 RANKS: 0 - 1200
0.000 SERVERS: 1 RANKS: 1201 - 1201
0.000 WORK WORKERS: 1201 RANKS: 0 - 1200
0.035 global: u:variables=<-1>
0.037 function:swift:constants

After running the pipeline with 610 nodes and specifying 20 ADLB_SERVERS, I found that it now failed during a call to mkdir when making the timinglog directory in the tmp location. So I added the following line to the VariantCalling.swift source file:
mkdir(strcat(variables["TMPDIR"], "/timinglogs")) =>
I retried running the pipeline and now a mkdir error occurs, but now it happens here:
CAUGHT ERROR:^M
app execution failed on: nid24730 ^M
 shell: Command failed with exit code: 139^M
    invoked from within^M
"c::sync_exec $stdin_src  $stdout_dst $stderr_dst $cmd {*}$args " ^M
 command: mkdir^M
    while executing^M
"error $msg"^M
    (procedure "turbine_error" line 8)^M
    invoked from within^M
"turbine_error "app execution failed" on: [ c_utils::hostname ]  "\n $msg" "\n command: $cmd""^M
    (procedure "app_error" line 13)^M
    invoked from within^M
"app_error $tries $options $cmd {*}$args"^M
    (procedure "turbine::exec_external" line 32)^M
    invoked from within^M
"turbine::exec_external "mkdir" [ dict create ] "-p" ${v:AlignDir}"^M
    (procedure "mkdir-app-leaf1-3" line 4)^M
    invoked from within^M
"mkdir-app-leaf1-3 /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1366248/align/"^M
Turbine worker task error in: mkdir-app-leaf1-3 /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1366248/align/^M

To me this suggests that the error occurs when making the directory at /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1366248/align/
After manually checking, this directory already exists. Swift's mkdir function is supposed to be implemented with the -p flag, which will not through an error if the directory already exists.

Can you remind me which installation this is?  I can fix it.

I am using the installation you installed on Blue Waters.
By typing the following:
~wozniak/Public/sfw/login/swift-t/stc/bin/swift-t -v
The following output is returned:
STC: Swift-Turbine Compiler 0.7.3
	 for Turbine: 1.0.0
Using Java VM:    /opt/java/jdk1.8.0_51/bin/java
Using Turbine in: /u/sciteam/wozniak/Public/sfw/login/swift-t/turbine

Turbine 1.0.0
 in: /u/sciteam/wozniak/Public/sfw/login/swift-t/turbine
 installed from: /u/sciteam/wozniak/proj/pub/login/swift-t/turbine/code
 using MPI: /mnt/a/u/sciteam/wozniak/Public/login/mpich-3.1/lib
 using Tcl: /usr/bin/tclsh8.5
 using Python:  python2.7

I appreciate your help on this! Thank you

I just checked and this installation is using a Swift/T recent enough to have this app definition:
app (void v) mkdir(string dirname)
{
"mkdir" "-p" dirname;
}
It is visible here: ~wozniak/Public/sfw/login/swift-t/turbine/export/unix.swift
Can you try doing a short run with environment variable TURBINE_LOG=1
That will show the command before it is executed, and may reveal the issue.
We could also make this do mkdir -vp for debugging.
I can do a call if you want to do screen sharing.
Note that there is a compute installation too.  The difference is the MPI library used by Swift/T.

I have been running the tests with the TURBINE_LOG=1, so I should already have this info.
I've copied the preceding lines below, but since this is a run with 1200 samples being processed, there is a good chance that these commands do not correspond directly with the sample that failed (different IDs, etc), but the commands should be the same
   1.998 store: <29084>="/scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1451156/logs/SRR1451156_Alignment.log"
   1.859 exec: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_tmp/timinglogs/ {}
   1.906 shell: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1380081/variant/
   2.151 rule: <30348> => alignReads-async_op-eq_string 30049 30348 {file 29878 is_mapped 1} {file 13314 is_mapped 1} {22884} {10147} {10763} 29721 {file 30122 is_mapped 1} {-1}
   2.993 allocated u:threads=<39890> t:2=<38767> t:5=<40929> t:7=<39910> t:11=<39552>
   2.993 allocated t:13=<40021> t:29=<38994>
   2.993 creating reference: <-1>["CORES_PER_NODE"] <- <*38767> (string)
   2.153 shell command duration: 0.295
   1.877 allocated u:threads=<29078> t:2=<29229> t:5=<29510> t:7=<28653> t:11=<29082>
   1.877 allocated t:13=<29286> t:29=<29102>
   1.859 shell: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_tmp/timinglogs/
   1.998 rule: <29084> <29261> <28095> <28115> <11951> <30592> => bwa_mem-app-leaf1 20086.0 20086.1 13694.0 29084 29261 28095 28115 0 1 SRR1451156 {SRR1451156  ALL     BWAMEM start<PHONE_NUMBER>
} {file 30652 is_mapped 1} {file 13694 is_mapped 1} 11951 30592 {file 29806 is_mapped 1}
   2.268 function: alignReads
   1.876 store: <7312>=void
   1.851 shell command duration: 0.296
   2.074 shell: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1345122/realign/
   1.751 shell: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1428317/realign/
   2.137 rule: <29567> => alignReads-async_op-strcat-5 29567 {5031} {file 30068 is_mapped 1}
   1.783 shell command duration: 0.467
   1.877 shell command duration: 0.329
   2.291 allocated u:threads=<31679> t:2=<31902> t:5=<32271> t:7=<31908> t:11=<31699>
   2.291 allocated t:13=<31538> t:29=<31922>
   1.770 rule: <28301> => alignReads-async_op-eq_string 28304 28301 {file 29577 is_mapped 1} {file 7920 is_mapped 1} {9761} {6219} {6468} 29537 {file 30020 is_mapped 1} {-1}
   1.170 store: <27829>="@RG\tID:SRR1396467\tLB:\tPL:ILLUMINA\tPU:SRR1396467\tSM:SRR1396467\tCN:"
   2.161 rule: <30193> => alignReads-async_op-strcat-4 30296 30193 {file 30047 is_mapped 1} {10062}
   2.161 rule: <30441> => alignReads-async_op-strcat-5 30441 {10062} {file 29495 is_mapped 1}
   2.140 exec: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1279946/variant/ {}
   1.569 shell command duration: 0.338
   1.903 exec: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1450209/align/ {}
   2.106 allocated u:threads=<29607> t:2=<30369> t:5=<29744> t:7=<30342> t:11=<30108>
   2.106 allocated t:13=<30159> t:29=<31240>
   1.789 exec: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1378648/align/ {}
   1.789 shell: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1378648/align/
   2.078 shell: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1412258/realign/
   1.899 creating reference: <-1>["CORES_PER_NODE"] <- <*30097> (string)
   1.864 creating reference: <-1>["BWAEXE"] <- <*28513> (string)
   1.864 creating reference: <-1>["BWAINDEX"] <- <*29817> (string)
   2.271 creating reference: <-1>["ALIGNERTOOL"] <- <*31136> (string)
   2.271 rule: <31136> => alignReads-async_op-eq_string 32297 31136 {file 31330 is_mapped 1} {file 22170 is_mapped 1} {22961} {18211} {18212} 32845 {file 32252 is_mapped 1} {-1}
   1.179 shell: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1276015/logs/
   1.818 shell command duration: 0.696
   2.185 exec: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1291456/variant/ {}
   1.159 allocated u:mkdirSignal=<26313> t:96:1=<26818> u:threads=<27010> t:140:1=<27792> t:143:1=<26294>
   1.159 allocated t:145:1=<26333> t:147:1=<27108> t:149:1=<26353>
   1.159 array_kv_build: <27108> 1 elems, write_decr 1
   2.186 shell: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1291456/variant/
   1.819 exec: mkdir -p /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1259071/variant/ {}
CAUGHT ERROR:^M
app execution failed on: nid16561 ^M
 shell: Command failed with exit code: 139^M
    invoked from within^M
"c::sync_exec $stdin_src  $stdout_dst $stderr_dst $cmd {*}$args " ^M
 command: mkdir^M
    while executing^M
"error $msg"^M
    (procedure "turbine_error" line 8)^M
    invoked from within^M
"turbine_error "app execution failed" on: [ c_utils::hostname ]  "\n $msg" "\n command: $cmd""^M
    (procedure "app_error" line 13)^M
    invoked from within^M
"app_error $tries $options $cmd {*}$args"^M
    (procedure "turbine::exec_external" line 32)^M
    invoked from within^M
"turbine::exec_external "mkdir" [ dict create ] "-p" ${v:LogDir}"^M
    (procedure "mkdir-app-leaf1-2" line 4)^M
    invoked from within^M
"mkdir-app-leaf1-2 /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1276015/align/ /scratch/sciteam/jacobrh/purge_exempt/..."^M
Turbine worker task error in: mkdir-app-leaf1-2 /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1276015/align/ /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1276015/logs/ /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1276015/realign/ /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_output/SRR1276015/variant/ /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/610Nodes/610Node_tmp/timinglogs/ 29165^M
    invoked from within^M
"c::worker_loop $WORK_TYPE($mode) $keyword_args"^M
    (procedure "standard_worker" line 27)^M
    invoked from within^M
"standard_worker $rules $startup_cmd "^M
    (procedure "enter_mode_unchecked" line 5)^M
    invoked from within^M
"enter_mode_unchecked $rules $startup_cmd"^M
    (procedure "enter_mode" line 10)^M
    invoked from within^M
"enter_mode $rules $startup_cmd "^M

Thanks for the log.
I now realize that exit code 139 is signal 11, which means mkdir crashed.  Something could be going wrong in the FS.
Could you try our app retry feature: http://swift-lang.github.io/swift-t/guide.html#_app_retry

I set TURBINE_APP_RETRIES=6 and am rerunning the 1200 sample test. I'll let you know how it goes.
Thanks for your help!

So I ran the test again and got past the mkdir stages, but it now failed at a different spot. It was a app call failure, which I was experiencing earlier.
Basically, after the first major program produces output, the "checkBam" function determines whether or not that output file was empty or not. But in order to do that, we are using a tool called Samtools.
The call to Samtools returns an error stating that the file it is looking at is truncated, which has usually suggested to us that swift/t is not waiting for the output to be done writing to before trying to process it.
Here is (what I think to be) the relevant section of code:
alignedbam, tmpsamtoolsLog = samtools_view_logged(vars["SAMTOOLSEXE"], alignedsam, threads, ["-u"], sampleName);

	// Verify alignment was successful
	if ( checkBam(vars, alignedbam) ) {
				/*****
				DO STUFF
				******/

and for reference here is the definition of the checkBam function:
(boolean success) checkBam(string vars[string], file bamFile) {
	// Swift/T would try to use samtools_view2 even before bamFile is read. Added explicit wait command
	wait (bamFile) {
		int alignNum = samtools_view2(vars["SAMTOOLSEXE"], filename(bamFile));
		success = alignNum > 0;
	}
}

Is it reasonable for us to assume that Swift/T will wait for alignedbam to be fully written to before trying to use the checkBam function on it? Of course the alignedbam file was mapped to the filesystem in an earlier section of code, but I figured that is not important for the dataflow style of execution.

Another question,
In this run we are trying to analyze 1200 samples in parallel on 610 nodes with 2 processes per node (20 processes are set to be ADLB_SERVERS).
However, I've noticed that during these test runs, only around 100 or so output files are being produced (100 samples are being worked on) by the time the workflow fails.
I am assuming that the reason we are not seeing the maximum level of parallelism is because managing all of this work requires even more than 20 ADBL_SERVERS.
My first question is has anyone you guys have worked with analyzed anything at this scale before?
And secondly, is there some rough guideline that we can use to determine how many ADLB_SERVERS we should use for a given batch size?
Maybe 1 SERVER for every 50 WORKER processes or something like that?

I think 1 server overall should be fine for your workflow- we may want to try to trace execution to see what is running when.

Update: things have moved on quite a bit since this ticket was created:
I successfully ran a test using novosort for marking duplicates on 101 nodes (with 1 ADLB_SERVER). It did not work when using Picard to mark duplicates, for reasons I never figured out
When I had the successful run on 101 nodes, I had a 1-to-1 correspondence between # of samples and # of nodes.
I tried scaling out to 601 nodes with 1200 samples (2 samples per node) marking duplicates with novosort and it failed at the HaplotypeCaller stage. The error message for the failing sample complained about not having enough memory (java heap was set to 16g, which really should be plenty)
Tried again using only 600 samples, and reran the Swift qsub creating script. It still failed and gave the same error message.
I will try again after increasing the heap size to 32g (which should obviously be way more than enough)

Good to hear- I have more time next week so I can discuss this further or try to run it, etc.

So, we have tried a lot of things to get the pipeline to run with 600 nodes (and we even tried only 300 nodes) and could not get it to work. The details of what options we tried and what went wrong can be found in the test/BWtests folder in the repo.
For now, we are just going to put this problem on hold

I should be able to run this over the next few days.  About what fraction of the tasks fail?

That is hard to say. But whenever one fails, it kills that workflow. And even after repeated attempts, none of the runs have made it past the last stage (Haplotype Caller). And since different samples fail on different runs, we think the problem is not simply the presence of a faulty sample.

I have made a few changes to make it runnable by others but I am not sure how long this is going to take.  Currently, it is trying to make a directory in your scratch space.  Can you fix this up so others can run it?

If the main error you are getting is  about Java -Xmx  , then this is definitely Java exceeding the user-specified memory limit.  I would try to gather the cases that cause this to happen and run them on some other system to analyze how much RAM they require.

This is already done. We are running with the same java parameters as we always have with the Bash workflow on BW. Additionally, these same parameters have worked on other, non-Cray systems. The problem only arises when we scale up to lots of nodes. Smaller tests with fewer samples and nodes run fine on BW.

Hey @jmjwozniak,
So our team have been testing all of their stuff in my scratch space, as we were running out of space in our project directory. The data for the 600 samples is also stored in my /scratch space.
Although I copied those sh scripts into the Swift-T directory where you were editing them, they were originally from this location. /scratch/sciteam/jacobrh/purge_exempt/Swift_testing/601Nodes
And I just realized that, do to the nature of this data, you probably don't have the permissions necessary to test this 600 node run anyway.
@lsmainzer would it be possible to get him added to the ILL_baib group? Otherwise, there is no location we could move everything where he would have the permissions to try and test this run.
Thanks,
Jacob

Hi, Jacob, Justin!
Justin is already member of baib, but only Jacob and I are approved by the UIUC IRB to touch the ADSP data. There are two ways to proceed:

Justin makes edits and Jacob runs the workflow on the ADSP data
we create 600 copies of a small synthetic dataset ad Justin tests on those.

Let me know which you would prefer.
-Luda

on reflection, we really do not have a lot of node-hours in that allocation, and should be careful debugging at this point
I will go ahead and create the 600 copies of small data and the respective sample information sheet; Justin can debug with that, and Jacob can do the last run with the ADSP data when it has all been proven to work


Software bill of materials (SBOM) (docs)

Related PR: https://github.com/kubescape/workflows/tree/feat/add-sbom-creation


Update to TeX Live 2020

It was released quite some time ago.

Progress is being made! I have now moved the bundle-creation setup to this new repo with a substantially cleaner infrastructure. I have a draft TL2020 bundle and it is passing some basic tests. I'm currently going through the issue backlogs and seeing which ones I can address before I upload a beta.


Domain is not allowed

Hello,
I did add domain to my domains and i tested SMTP and IMAP, i even added users in whitelist... But still says domain is no allowed..
.
Please advise...

Do you have any add=on/extensions enabled?

Hello, I do not have anything enabled

Do you have a default domain set under the login tab in the admin interface?


Add support to patch resources in KubeOps function

Is your feature request related to a problem? Please describe.
Currently, KubeOps function supports create and delete operations. It would be useful to add support to patch resources through KubeOps function. Sample use cases are - e.g the places where we want to add labels or annotations as a part of backup operations
Describe the solution you'd like
Sample BP specs:
- func: KubeOps
  name: patchResource
  args:
    operation: patch
    objectReference:
      apiVersion: ""
      group: ""
      kind: ""
      name: ""
      namespace: ""
      resource: ""
    patch: |                              # yaml/json patch
      spec:
        replicas: 3

Describe alternatives you've considered
As of now, this can be achieved by running shell commands (kubectl patch) with kubeops function.

I'd love to revive this issue.
That's a feature that I would love to have in multiple cases I encountered.
Most recently for adding secrets to existing service accounts.
@pavannd1


IPAddress created using args host and prefix_length are unable to be retrieved later using address arg

Environment

Nautobot version (Docker tag too if applicable): 1.6.4
Python version: 3.8.18
Database platform, version: PSQL container
Middleware(s):

Steps to Reproduce

Ensure IPAddress does not exist
Create IPAddress via ORM with IPAddress.objects.create(host=<address>, prefix_length=32, status=<status>)
Try to retrieve the address with IPAddress.objects.get(address=<address>/32)

Expected Behavior
I expected the IPAddress to be retrieved

Observed Behavior
Exception nautobot.ipam.models.IPAddress.DoesNotExist was raised
>>> active_status = Status.objects.get(name="Active")
>>> IPAddress.objects.get(address="<IP_ADDRESS>/32")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/nautobot/ipam/querysets.py", line 434, in get
    return super().get(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
nautobot.ipam.models.IPAddress.DoesNotExist: IPAddress matching query does not exist.
>>> IPAddress.objects.get(host="<IP_ADDRESS>", prefix_length=32)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/nautobot/ipam/querysets.py", line 434, in get
    return super().get(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
nautobot.ipam.models.IPAddress.DoesNotExist: IPAddress matching query does not exist.

>>> ip_with_host = IPAddress.objects.create(host="<IP_ADDRESS>", prefix_length=32, status=active_status)
>>> ip_with_host.validated_save()

>>> IPAddress.objects.get(host="<IP_ADDRESS>", prefix_length=32)
<IPAddress: <IP_ADDRESS>/32>

>>> IPAddress.objects.get(address="<IP_ADDRESS>/32")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/nautobot/ipam/querysets.py", line 434, in get
    return super().get(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
nautobot.ipam.models.IPAddress.DoesNotExist: IPAddress matching query does not exist.
>>> IPAddress.objects.get(address="<IP_ADDRESS>")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/nautobot/ipam/querysets.py", line 434, in get
    return super().get(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
nautobot.ipam.models.IPAddress.DoesNotExist: IPAddress matching query does not exist.```

Fixed in ltm-1.6 by #4770; the VarbinaryField fix from that PR needs to be added to develop as well before closing this.

#4804 is the corresponding PR to develop.


How to restrict processing based on Kubernetes cluster?

We are running delayed jobs in a kubernetes environment. Our setup is that we have one main cluster and then a few aliased clusters to allow each cluster to share a database. This works almost flawlessly and prevents people from having to remember different credentials and data on different clusters.
Recently, I have to test some work that I have done that involves background jobs. Unfortunately, I run into problems that the wrong workers are picking off jobs, which leads to confusion, errors, and misunderstanding.
I asked a question over on StackOverflow: https://stackoverflow.com/questions/64650185/how-to-constrain-delayed-job-processing-based-on-kubernetes-cluster.
This is kind of what I had in mind for a solution https://github.com/collectiveidea/delayed_job_active_record/compare/master...wundertax:master.
I am not sure if I have to go through all of that, or is there a better way? Perhaps using the lifecycle events?
Any help, pointers, and suggestions are deeply appreciated.

After some more investigations and learnings I ended up with a simple monkey patch:
require 'delayed/backend/active_record'

module Delayed
  module Backend
    module ActiveRecord
      class Configuration
        attr_accessor :cluster
      end

      # A job object that is persisted to the database.
      # Contains the work object as a YAML field.
      class Job < ::ActiveRecord::Base
        READY_SQL = <<~SQL.squish.freeze
          ((cluster = ? AND run_at <= ? AND (locked_at IS NULL OR locked_at < ?)) OR locked_by = ?) AND failed_at IS NULL
        SQL

        before_save :set_cluster

        def self.ready_to_run(worker_name, max_run_time)
          where(READY_SQL, cluster, db_time_now, db_time_now - max_run_time, worker_name)
        end

        # When a worker is exiting, make sure we don't have any locked jobs.
        def self.clear_locks!(worker_name)
          where(cluster: cluster, locked_by: worker_name)
            .update_all(locked_by: nil, locked_at: nil) # rubocop:disable Rails/SkipsModelValidations
        end

        def self.cluster
          Delayed::Backend::ActiveRecord.configuration.cluster
        end

        def set_cluster
          self.cluster ||= self.class.cluster
        end
      end
    end
  end
end

Delayed::Backend::ActiveRecord.configuration.cluster = ENV['CLUSTER'] if ENV['CLUSTER']

After having added the cluster column everything works fine as far as I can tell, any suggestions for improvements? Did I forget anything?


Allow to disable upmerge for a specific branch

Effectively disabling upmerge for a brach will continue with other branches

Q
A

Bug fix?
no

New feature?
yes

BC breaks?
no

Deprecations?
no

Tickets

License
MIT

Effectively disabling upmerge for a brach and continues with a next possible version.
Use-case: Say 3.[1-3] is disabled but you still want to upmerge 2.4 (LTS) into 3.4.

Build failure is unrelated, I'm not using the exact same version of php-cs-fixer as the one used for CI.
And building a new image (surprise) fails 😑


Help Examples

I remember seeing a command somewhere on HackerNews as an alternative to man that shows you common examples of how the command is used. I can't find it though and was hoping it would be on here.

Ah, this is it: https://tldr.sh/


Fix/ignore with several work dirs

Description
This PR fixes ignore files not being respected in different work directories.
Checklist

[X] Tests added and all succeed

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


é character showing up as ? in text files

When I open a text file containing the é special character it shows up as a ? with a white diamond around it.

@danlfarnell This is most likely a file with a different encoding than UTF8. Please use the switcher at the bottom right to open the file with a different encoding or use the "files.encoding" setting to change the default from UTF8 to something else.

Awesome thanks.  Dos format works like a charm :)
On Feb 16, 2016 11:43 AM, "Alexandru Dima"<EMAIL_ADDRESS>wrote:

@danlfarnell https://github.com/danlfarnell This is most likely a file
with a different encoding than UTF8. Please use the switcher at the bottom
right to open the file with a different encoding or use the
"files.encoding" setting to change the default from UTF8 to something
else.
—
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/vscode/issues/3053#issuecomment-184761513.

Closing


fix(runtime-core): correct type inference for PascalCase emits

fix vuejs/language-tools#4269

@KazariEX can you resolve the conflicts since we've pointed main to the 3.5 beta now?

/ecosystem-ci run


Sound not playing

Hi  ,  I am using your  docker-compose file   with  a Topping E30  DAC ,  LMS and Squeezelite in containers.  However even if LMS sees the Sqeezelite player as functional , sound is not played in the speakers.   Any ideas why this is happening ?

Hello, I have been away on holiday for a few days, I will come back to you as soon as I can.

Can you provide your docker run command or docker compose file, along with the output of the command:
aplay -L
Thank you

Same here.
Linux raspberrypi 5.10.63-v7l+ #1496 SMP Wed Dec 1 15:58:56 GMT 2021 armv7l
on https://hub.docker.com/r/giof71/squeezelite
docker run -d --rm --network host --device /dev/snd --net host giof71/squeezelite:stable
returns error: double host
So running by:
docker run --privileged -it -i --entrypoint=/bin/bash -t giof71/squeezelite:stable
works and sounds to hdmi.
I would like to use docker-compose and headphone jack.
By docker-compose, run and connect but any sound.
  squeezelite:
    image: giof71/squeezelite:stable
    container_name: squeezelite
    devices:
      - /dev/snd:/dev/snd
    environment:
      - SQUEEZELITE_NAME=home-pi
      #- SQUEEZELITE_AUDIO_DEVICE=hw:CARD=Headphones,DEV=0
      #- SQUEEZELITE_AUDIO_DEVICE=default:CARD=Headphones
      - SQUEEZELITE_AUDIO_DEVICE=default:CARD=b1
      - SQUEEZELITE_SERVER_PORT=<IP_ADDRESS>:3483
      - SQUEEZELITE_RATES="44100,48000,88200,96000,176400,192000,352800,384000"
      - SQUEEZELITE_UPSAMPLING="v::4:28:95:105:45"
      - STARTUP_DELAY_SEC=1
    restart: unless-stopped

I login to
docker run --privileged -it -i --entrypoint=/bin/bash -t giof71/squeezelite:stable
(I use this but I'm not sure it is correct), then
aplay -L
returns:
bash: aplay: command not found
I also try:
# /usr/bin/squeezelite -l
Output devices:
  null                           - Discard all samples (playback) or generate zero samples (capture)
  default:CARD=b1                - bcm2835 HDMI 1, bcm2835 HDMI 1 - Default Audio Device
  sysdefault:CARD=b1             - bcm2835 HDMI 1, bcm2835 HDMI 1 - Default Audio Device
  dmix:CARD=b1,DEV=0             - bcm2835 HDMI 1, bcm2835 HDMI 1 - Direct sample mixing device
  dsnoop:CARD=b1,DEV=0           - bcm2835 HDMI 1, bcm2835 HDMI 1 - Direct sample snooping device
  hw:CARD=b1,DEV=0               - bcm2835 HDMI 1, bcm2835 HDMI 1 - Direct hardware device without any conversions
  plughw:CARD=b1,DEV=0           - bcm2835 HDMI 1, bcm2835 HDMI 1 - Hardware device with all software conversions
  default:CARD=Headphones        - bcm2835 Headphones, bcm2835 Headphones - Default Audio Device
  sysdefault:CARD=Headphones     - bcm2835 Headphones, bcm2835 Headphones - Default Audio Device
  dmix:CARD=Headphones,DEV=0     - bcm2835 Headphones, bcm2835 Headphones - Direct sample mixing device
  dsnoop:CARD=Headphones,DEV=0   - bcm2835 Headphones, bcm2835 Headphones - Direct sample snooping device
  hw:CARD=Headphones,DEV=0       - bcm2835 Headphones, bcm2835 Headphones - Direct hardware device without any conversions
  plughw:CARD=Headphones,DEV=0   - bcm2835 Headphones, bcm2835 Headphones - Hardware device with all software conversions

Finally, aplay -L run in the system terminal:

default
    Playback/recording through the PulseAudio sound server
null
    Discard all samples (playback) or generate zero samples (capture)
jack
    JACK Audio Connection Kit
pulse
    PulseAudio Sound Server
sysdefault:CARD=b1
    bcm2835 HDMI 1, bcm2835 HDMI 1
    Default Audio Device
dmix:CARD=b1,DEV=0
    bcm2835 HDMI 1, bcm2835 HDMI 1
    Direct sample mixing device
dsnoop:CARD=b1,DEV=0
    bcm2835 HDMI 1, bcm2835 HDMI 1
    Direct sample snooping device
hw:CARD=b1,DEV=0
    bcm2835 HDMI 1, bcm2835 HDMI 1
    Direct hardware device without any conversions
plughw:CARD=b1,DEV=0
    bcm2835 HDMI 1, bcm2835 HDMI 1
    Hardware device with all software conversions
usbstream:CARD=b1
    bcm2835 HDMI 1
    USB Stream Output
sysdefault:CARD=Headphones
    bcm2835 Headphones, bcm2835 Headphones
    Default Audio Device
dmix:CARD=Headphones,DEV=0
    bcm2835 Headphones, bcm2835 Headphones
    Direct sample mixing device
dsnoop:CARD=Headphones,DEV=0
    bcm2835 Headphones, bcm2835 Headphones
    Direct sample snooping device
hw:CARD=Headphones,DEV=0
    bcm2835 Headphones, bcm2835 Headphones
    Direct hardware device without any conversions
plughw:CARD=Headphones,DEV=0
    bcm2835 Headphones, bcm2835 Headphones
    Hardware device with all software conversions
usbstream:CARD=Headphones
    bcm2835 Headphones
    USB Stream Output

As you can see above I try three SQUEEZELITE_AUDIO_DEVICE, but any sounds.
Sorry for long post.
I hope can this help to resolve.
Thanks

Just a quick reply, the doc is probably wrong about host networking
Either remove --network host or --net host
Sorry about that, I will try to fix as soon as possible.
I will have a look at the rest of your post(s) and come back hopefully with some answers

Documentation fixed (host networking)

Hello, are you sure your devices support those very high rates?
First you might want to try without SQUEEZELITE_RATES and SQUEEZELITE_UPSAMPLING variables.
Let me know if this helps.

Hello, looking at the specs of your dac, it seems it should work at those rates. Can you try anyway with a simple configuration and see if you can get sound?
Then we can setup upsampling if you wish to do so.
Cheers

@vespadj are you using the raspberry pi onboard audio card?

@vespadj have a look at the README.md file, I added your docker-compose.

Hello @vespadj, can we close this issue?

Hello @joinstuff , can @GioF71 close this issue?
Yes by my side.

I forgot to ask the op, sorry

Closing. @joinstuff feel free to reopen or create a new issue if you need further information.


Added copyright stmt and follow us section

@notsoocool
I have added copyright statement and add more social icons in this pull request itself #138
Kindly check it out

@AswaniBolisetti this file have commit of scroll progress bar, i will not reopen it

@notsoocool
It won't be a problem if you merge the scroll-progress PR first
Then you can merge this enhance-footer branch PR

@notsoocool It won't be a problem if you merge the scroll-progress PR first Then you can merge this enhance-footer branch PR

@AswaniBolisetti I am not going to entertain this any further, if you want points for your PR just follow the steps which i am guiding you for. GssoC is basically to learn Open source contribution, its not just for the sake of raising PR and gaining some points.


Inline-block whitespace issue for parents with nested children

Feels wrong filing an issue, but thanks for a great project! Feels good to declutter after some time on bootstrap.
Yesterday I ran into the issue of four parent elements with 25% width .w-25-l overflowing to the next line. After much debugging -- checking margins, paddings, borders, anything that might be adding just enough pixels to be making the parents larger than true 25% -- I found an old css issue with display: inline-block on a parent rendering the new line characters between the child divs as whitespace.
My solution was to style the parent font-size: 0;, which did not affect any child text at all apply flex class to parent. As simple as that. The whitespace between children was no longer displayed, 25% width working correctly. Too used to those automagic grids.
I process the tachyons src files as .scss, so it was easy to add some utility classes that extended the .flex class without having to keep considering it.
Other possible fixes were:

   one

   two

   three

one

two

three

one

two

three

Item content

Item content

Item content

div {
  font-size: 0;
}

Feel free to close this immediately.
Thank you!

UPDATE:
Flex is preventing the rest of the layout from collapsing now. Hmmm.

You are correct - display: inline-block renders white-space between elements and so that is why if you have four elements it is breaking to the next line. If you are setting with of elements that need to be on the same line I suggest floating them or using display table.
Your white space solutions can work - but that is brittle in my experience maintenance wise.
You should be able to remove the flex declaration for any breakpoint that isn't needed but let me know if you can't figure out how to do that.


Improve logic to better deal with current number of repositories.

As a bcgov organization user,
I want repomountie to be able to collect information about all repositories without hitting an API limit,
So that I am sure my repository doesn't get missed
Implementation Details:
The bot is a victim of its own success. When it starts, it gets a list of existing repositories, and goes through to see if any are non-compliant. To do this it needs to make quite a few API calls. This quickly hits the limit of API requests with GitHub. Would be good spread this out over a longer period of time so it does not choke and nothing gets missed.
I've explained to Alec. Can add context if needed.

I just added a user story above the issue body I think captures that to help with handing over the project and showing the backlog, we can add in a template to help with that


Revert   linker command file fixes: .bss/.noinit sections, z_mapped_start marker

Revert https://github.com/zephyrproject-rtos/zephyr/pull/60368 as it's breaking few tests, see https://github.com/zephyrproject-rtos/zephyr/issues/61572.

@SgrrZhf commented in #61572 that his observation is that __kernel_ram_start points to address 0x0. This doesn't surprise me and I think that it should already have been like this before. The move of the .bss and .noinit sections within the image shouldn't have any influence on this. I'll look into the pre/post diffs once again, but here's a general note on this observation:
The DTS for the ZynqMP (used in qemu_cortex_r5), the DTS fvp_baser_aemv8r.dts mentioned by SgrrZhf and probably a few more all specify the memory range used as RAM to start at address 0x0. From my point of view, this isn't valid (or at least unfortunate) in general and in particular doesn't work on an actual UltraScale chip (simulated in qemu_cortex_r5) unless the system is configured in a particular way.
Address 0x0 is, I'd guess in the majority of cases, the location of the exception vectors. As far as I can tell, there's two options for having the vectors at a different location: one is the HIVECS bit, which, at least on the Zynq and ZynqMP, requires a non-default memory configuration in order to enable memory located at the alternative high address of the vectors. The other one is using the VBAR register for a custom location, which the Cortex-A has, but the aarch32 Cortex-R doesn't have (there's something similar in Cortex-M, too). For any non-Cortex-M, non-ARM64-Cortex-R CPU, the HIVECS bit always gets cleared in relocate_vector_table(), btw.
The standard way of doing things is to have the vectors at the very beginning of the Zephyr image in RAM. This allows booting a stripped, non-ELF binary without entry point information (when copied to the right location), as the first instruction is the reset vector jump which will take you to __start. Regardless of whether we reach __start this way or via its location info from the ELF header, during early boot, the vectors are copied to their target location in RAM. At least for Cortex-A, the vectors section is padded to 4k due to the MMU page size. If the binary is loaded to address 0x0, the vectors are basically being copied to their current location.
The vectors section should always have been covered by __kernel_ram_start, with regards to that section, the z_mapped_start marker was the relevant fix to prevent the MMU on the Cortex-A from assigning the vectors page at the RAM start address as dynamically mapped memory at run-time. The following sections like .text, .data etc. have always been covered by z_mapped_start.
I'd say that in general, having the actual vectors location 0x0 and the base of the image in RAM overlap is unfortunate, as, at least on Xilinx chips, this wouldn't work in real life (QEMU simply doesn't care about the validity of your memory layout, aside from many other things, at least for the UltraScale). The Zynq-7000 has an offset of 0x00100000 for the RAM, the memory at address 0x0 is the On-Chip RAM (which has to be explicitly re-configured in case it should rather show up at the HIVECS location), and there's a gap between them. On the UltraScale, there's two Tightly Coupled Memory blocks starting at address 0x0, once again, only the minimum required to have working memory for the vectors at 0x0 is enabled by default. If you locate the image starting at 0x0, there's a black hole starting at either 0x10000 or 0x20000, based on the TCM configuration. Also, there's a mode in which the two TCM blocks only have half their size, but are transparently accessible under the same address core-local. In that case, there's two black holes within the TCM range before RAM starts behind that. There's the ancient #26577 addressing this issue for the UltraScale (the Zynq-7000 DTS correctly specifies the start of RAM at 0x00100000), at that time, the generic memory-region nodes didn't exist yet, so maybe it's about time to re-declare the ZynqMP's TCMs as off limits for normal RAM use.
On the one hand, I'd prefer some spatial separation between the vectors' zero-page and the start of the image in general, this copying the vectors over themselves somehow doesn't look pretty. On the other hand, I think that on most real-life SoCs, the memory at address 0 is special in terms of its technical implementation (for example, on the Zynq-7000 it already works even if the PLLs and therefore the external RAM haven't been set up yet) and that there's no guarantee that there's contiguous memory available from address 0x0 all the way into regular RAM.
I've just moved the RAM base address in the ZynqMP DTS upwards a little. Let's see if this helps with the problematic test case. If that's the case, I'll look into whether the issue is the address 0 and if __kernel_ram_start has always pointed there. I also can't exclude an issue within the MPU code due to the modified section layout (but probably with an identical zero __kernel_ram_start) yet.
I'll check once again what the situation was like before regarding __

I've dabbled back and forth a bit so far, and I think that the whole issue with memory protection test(s) failing isn't exclusively down to my changes to the linker command file. With the linker command file from before my modifications, the following test cases fail:
INFO    - 1) tests/kernel/mem_protect/protection/kernel.memory_protection.protection on qemu_cortex_r5 failed (Timeout)
INFO    - 2) tests/kernel/mem_protect/obj_validation/kernel.memory_protection.obj_validation on qemu_cortex_r5 failed (Timeout)
INFO    - 3) tests/kernel/mem_protect/mem_protect/kernel.memory_protection on qemu_cortex_r5 failed (Timeout)
INFO    - 4) tests/kernel/mem_protect/stack_random/kernel.memory_protection.stack_random on qemu_cortex_r5 failed (Timeout)
INFO    - 5) tests/kernel/mem_protect/stackprot/kernel.memory_protection.stackprot on qemu_cortex_r5 failed (Timeout)
... if in the ZynqMP's DTS, the sram0 memory region begins at a location other than 0. So it seems that there's some code relating to the MPU stuff that only works with a fixed RAM base @ 0x0. This shouldn't happen as, as mentioned above, it's highly unlikely that all real-life targets will support that address as RAM base.

Okay, I admit it: __kernel_ram_start wasn't @ 0 before, but at the start of the bss section. To make the difference between a complete, successful test execution and an early boot crash while using the old linker command file, all it takes is a different RAM base address. I can see the offset in the map file, relative to the RAM base, __kernel_ram_start stays the same. Checking what causes the crash with a non-zero RAM base now.

@ibirnbaum many thanks for the investigation work.

@fabiobaltieri @microbuilder I've looked into this issue a little bit further, although I haven't reached a conclusion yet. What I can say is that with the RAM base address moved to 1 MB (in the DTS and hardcoded in the ZynqMP MPU init code), the tests that caused the CI to fail on qemu_cortex_r5 do actually pass on actual ZynqMP hardware - with both the old linker command file and the changes proposed in #60368.
I do believe that in general, qemu_cortex_r5 definitely needs an overhaul regarding the MPU setup, not only is the RAM base address hardcoded at the time being, also, .rodata isn't considered at all - RAM is always full access. There's a few items that need to be linked to both the sections contained in the binary and the device tree.
In addition to that, I believe that qemu_cortex_r5 might also be broken on the QEMU side. As mentioned before, the memory protection tests pass on actual hardware while QEMU won't even boot. What I've discovered so far is that with the RAM moved to the proper base address, some code still accesses an address located somewhere in the BTCM. If I don't have an MPU mapping that covers at least the front half of the BTCM, Zephyr won't boot even up to the banner. However, on actual hardware, I don't need this mapping, so obviously, noone's touching the BTCM there, because my debugger didn't stop on any exception. Unfortunately, I'm still fighting gdb in order to be able to step through the code. Most gdb frontends fail because they're trying to load an aarch32 binary, while the Xilinx QEMU gdb stub identifies itself as aarch64, as the A-Cores are covered by gdb as well. I've made a little progress getting to the R-cores via gdb's command line, however, it's not really usable yet.
I'll keep digging. It's unfortunate that this target caused the CI to fail, I suspect that the issue is more on the target side than on the linker command file side. Basically the same issue the new linker command file was supposed to fix (bloated binaries due to .bss/.noinit being somewhere in the middle of the image) has recently also been reported for aarch64 in #62578.

@ibirnbaum thanks again for the investigation work, from what I understand of your explanation it sounds like qemu here is not representing the underlying hardware correctly and blocking a change that fixes something on real hardware. Sounds like qemu should be fixed then. Do you think that there would be an argument for disabling the tests on qemu-r5 in the meantime?

@ibirnbaum thanks again for the investigation work, from what I understand of your explanation it sounds like qemu here is not representing the underlying hardware correctly and blocking a change that fixes something on real hardware. Sounds like qemu should be fixed then. Do you think that there would be an argument for disabling the tests on qemu-r5 in the meantime?

I was going to suggest the same thing (disabling this platform for MPU tests on this platform, ideally adding a note why so it can be re-enabled in the future).
Also, FYI @wmamills who may have some insight into the QEMU problems since I believe this is coming from the Xilinx fork of QEMU.

@ibirnbaum Do you want to give this one more try, excluding qemu_cortex_r5 from the MPU tests since it's demonstrably not a problem with the code here, but the QEMU model? That might enable us to get this in before feature freeze.

@microbuilder What's the timeframe up to the feature freeze? In order to be sure that it's actually QEMU's R5 emulation that's malfunctioning, I'd like to run all of the memory protection testcases on my actual hardware prior to re-opening the linker command file PR. Due to this hardware only being bootable via JTAG, this is a relatively time-consuming process. Also, the same applies to my Cortex-A9 hardware, where I'd also like to run the tests on actual hardware, with the MMU in this case, which takes longer than just running them on qemu_cortex_a9.

@microbuilder What's the timeframe up to the feature freeze?

RC1 is due at Sept 29th: https://github.com/zephyrproject-rtos/zephyr/wiki/Release-Management

@fabiobaltieri @microbuilder Sorry, I won't get around to too much debugging over the next few days, and I'd still like to dig a little deeper regarding the issues with qemu_cortex_r5. I wouldn't want to exclude a target platform from those relatively important test cases on the basis that one or two manually executed tests executed on out-of-tree hardware worked better, without yet knowing why.
The bloated binaries are something that should definitely be fixed, but from my point of view, that might just as well happen with the next release rather than the upcoming one.
A word of warning in advance: I ran into issues with massive stack corruption with the recently overhauled aarch32 arch code, and I /think/ that the code related to the TRACING_OBJECT_TRACKING feature is to blame. I hope that I'll get around to narrowing the source of the problem down sometime towards next week, so that all the required info for at least opening an issue report is there.


Add configuration of min and max election timeout

What changes are proposed in this pull request?
Add configuration of min and max election timeout.
Why are the changes needed?
The maximum election timeout is currently fixed at twice the minimum election timeout. Adding this configuration can improve flexibility.
Does this PR introduce any user facing changes?
remove conf  alluxio.master.embedded.journal.election.timeout .
add conf  alluxio.master.embedded.journal.election.timeout.min and  alluxio.master.embedded.journal.election.timeout.max.

Codecov Report

Merging #13837 (32799f9) into master (db01aae) will decrease coverage by 26.06%.
The diff coverage is 25.80%.

@@              Coverage Diff              @@
##             master   #13837       +/-   ##
=============================================
- Coverage     43.61%   17.54%   -26.07%
+ Complexity     9083     2675     -6408
=============================================
  Files          1386     1386
  Lines         80546    80602       +56
  Branches       9804     9808        +4
=============================================
- Hits          35128    14144    -20984
- Misses        42465    65211    +22746
+ Partials       2953     1247     -1706

Impacted Files
Coverage Δ

.../master/journal/raft/RaftJournalConfiguration.java
0.00% <0.00%> (-82.86%)
:arrow_down:

...alluxio/master/journal/raft/RaftJournalSystem.java
0.00% <0.00%> (-64.84%)
:arrow_down:

.../alluxio/master/transport/GrpcMessagingServer.java
0.00% <0.00%> (-73.18%)
:arrow_down:

...common/src/main/java/alluxio/conf/PropertyKey.java
97.85% <100.00%> (-1.59%)
:arrow_down:

...mon/src/main/java/alluxio/shell/CommandReturn.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...mon/src/main/java/alluxio/util/ExceptionUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...n/src/main/java/alluxio/wire/AlluxioProxyInfo.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../src/main/java/alluxio/wire/AlluxioMasterInfo.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...src/main/java/alluxio/job/meta/JobIdGenerator.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...n/src/main/java/alluxio/stress/BaseParameters.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 662 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db01aae...32799f9. Read the comment docs.

@ggezer ping

@qian0817 Could you please drop your merge commit and instead replay your changes on top of a recent master branch?
We would like to keep the history clean by avoiding merge commits on PRs. @apc999 fyi.

alluxio-bot, merge this please


Western Windows1252 detected as SJIS

title is self explanatory :)
had the case with a little 4 lines extracted from Excel as CSV from a Windows computer

And with an example
Prénom;Nom;Email
invitéàçù01;invite01;test011&@barcom
invite02;invitéàçù01;test021bar.com
<EMAIL_ADDRESS>invitéàçù01;invite01;test99@barcom

You can save this text as Windows1252 with sublime or VSCode to reproduce the issue

I tested your text in Sublime text, and my result was same as yours.
But sorry, encoding.js is not supported Windows1252 now because this library is for Japanese encodings and Unicode.


Update Notifier Issue 14

Description
This PR adds a notification to the user when an npm update for dav-cli is available.  It uses the npm module update-notifier.
Related Issue
Add update notification functionality #14
https://github.com/DAVFoundation/dav-cli/issues/14
Motivation and Context
This change will notify users when their version of dav-cli is out of date.  It will be important during the rapid development cycle that users be aware when their version is not up to date.
How Has This Been Tested?
The update-notifier was tested by running against versions that both required and update and versions that did not.  The update-notifier acted according, only showing the notifier when the dav-cli was out of date compared to the most current version.
Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

My sincere apologies - I definitely did not want to comprise the entire project.  I'm not sure how that happened, although I think it could be due to running npm install multiple times during troubleshooting.
I reset the fork and now have pull request #19 reflecting the intended changes after doing the "compare".
Again sorry about.  Just fumbled my initial baby step.

It's Absolutely fine. We've all been there 😄
Didn't mean to be harsh or anything... I just want to instil in people a certain attention to detail when contributing to open source. To this day, after countless open source contributions, I still quadruple check every line before sending a PR.


[Fix] Listener in keyboard cvv to develop

Motivación y Contexto

No funcionaba el botón de continuar en el teclado en la pantalla de código de seguridad
Bug en jira

Codecov Report

:exclamation: No coverage uploaded for pull request base (develop@8d2b41f). Click here to learn what that means.
The diff coverage is 20%.

@@            Coverage Diff             @@
##             develop    #1109   +/-   ##
==========================================
  Coverage           ?   29.25%
==========================================
  Files              ?      326
  Lines              ?    14686
  Branches           ?     1337
==========================================
  Hits               ?     4297
  Misses             ?     9882
  Partials           ?      507

Impacted Files
Coverage Δ

...ain/java/com/mercadopago/SecurityCodeActivity.java
49.56% <20%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d2b41f...8f9f568. Read the comment docs.


Apache Superset is not starting line 50: /usr/bin/run-server.sh: No such file or directory

Apache Superset docker-compose is not working for a newly cloned project and by following the offical documentation: https://superset.apache.org/docs/installation/installing-superset-using-docker-compose
How to reproduce the bug

Clone the superset git locally
Run docker-compose -f docker-compose-non-dev.yml up

Expected results
The application should start.
Actual results
The web application is not starting and the log shows the following error:
superset_app            | Starting web app...
superset_app            | /app/docker/docker-bootstrap.sh: line 50: /usr/bin/run-server.sh: No such file or directory

Screenshots
If applicable, add screenshots to help explain your problem.
Environment
git checkout today 10.12.2021, no changes on my side just the latest repository

Same here, probably due to this change?  https://github.com/apache/superset/commit/aee5c9a3ea1006b194319d47c3e7f245e438ce6f

I confirmed that it works as of the previous commit (8c25f2f356e488120d7ce3ff852a4b6b9a049b3b).

That was it! When I pulled the images again, it worked.

I just checked out master and tried to run through Docker and I am getting the same error.  If I check out the 1.4.0 tag it starts correctly without errors.  According to the documentation master is supposed to work, right?
superset_app            | Starting web app...
superset_app            | Skipping local overrides
superset_app            | Starting web app...
superset_app            | /app/docker/docker-bootstrap.sh: line 50: /usr/bin/run-server.sh: No such file or directory
superset_app            | Skipping local overrides
superset_app            | Starting web app...
superset_app            | /app/docker/docker-bootstrap.sh: line 50: /usr/bin/run-server.sh: No such file or directory
superset_app            | Skipping local overrides
superset_app            | Starting web app...
superset_app            | /app/docker/docker-bootstrap.sh: line 50: /usr/bin/run-server.sh: No such file or directory
superset_app            | Skipping local overrides
superset_app            | Starting web app...
superset_app            | /app/docker/docker-bootstrap.sh: line 50: /usr/bin/run-server.sh: No such file or directory
superset_app            | Skipping local overrides

Still broken for me as well.  @amitmiran137 any reason this was closed?

@ofekisr can you check it out ?

The problem exists in 1.4.0. There shouldn't be so many differences and long periods of time between releases, because there are a lot of changes everywhere and there is no way to use default helm chart values with stable version. :(
Yesterday release is based on commit from OCTOBER 2021 which causes 3,5months of fixes and improvements missing in new release.
There is 285 changed files with 8,931 additions and 22,463 deletions difference between current master and release tag 1.4.0
@eschutho is there any roadmap for release 1.4.1 or any timetable when next release will cover those changes.
It would be good to add some tests for helm chart deployment for both latest and stable version.

first, why use docker-compose when you can use the helm chart?
second, just need to change the docker-compose bootstrap file to the new one.
@ninomllr @amitmiran137 @atpage

@ofekisr We are following these instructions:
https://superset.apache.org/docs/installation/installing-superset-using-docker-compose

@atpage @amitmiran137
anyway IMO it is not my change, if you look inside the Dockerfile you'll see the bootstrap script file is copied only inside the CI version while the docker-compose file reference the official release image

@wiktor2200 I have found this discussion for 1.4.0 bugs that are candidates for being fixed in 1.4.1 -> https://github.com/apache/superset/discussions/18119

Running
docker-compose -f docker-compose-non-dev.yml pull before running the up command should fix this. The issue is that the version of code you are on is not compatible with the docker image you have, so you need to pull a newer docker image that is compatible with this code

Running the pull command should be standard practice whenever you pull the latest source code

The problem exists in 1.4.0. There shouldn't be so many differences and long periods of time between releases, because there are a lot of changes everywhere and there is no way to use default helm chart values with stable version. :( Yesterday release is based on commit from OCTOBER 2021 which causes 3,5months of fixes and improvements missing in new release. There is 285 changed files with 8,931 additions and 22,463 deletions difference between current master and release tag 1.4.0 @eschutho is there any roadmap for release 1.4.1 or any timetable when next release will cover those changes.
It would be good to add some tests for helm chart deployment for both latest and stable version.

@wiktor2200 1.4.1 will only include bug fixes (not features) that address issues on 1.4.0, so of those 285 changes, only a small portion will apply. 1.4.1 is scheduled to go for vote in the next few days. Please add any bug fixes to the discussion here: https://github.com/apache/superset/discussions/18119

seeing same issue when deploying it using Helm and same on version 1.3.2.
I don't want to use latest as it shows version as v.0.0.0dev on UI.

1.4.1 still has the same issue for me

@manisha-tanwar @gai6948 have you tried @nytai's suggestion to pull the docker container again before running it?

It was problem with the helm chart, because running in docker-compose was perfectly fine.

Same issue for me with 0.5.8 chart with 1.4.1 or 1.4.0 image

Ok, that's what I expected, so for now, <= helm 0.4.0 works with all of the official Superset releases up to 1.4.1. Helm >=0.5.0 will be compatible with either the "latest" non release of Superset and the upcoming 1.5.0 release and is not compatible with <= Superset 1.4.1.

With x-superset-image: &superset-image apache/superset:1.4.1-dev in docker-compose-non-dev.yml running docker-compose -f docker-compose-non-dev.yml pull before docker-compose -f docker-compose-non-dev.yml up does not help:
superset_app            | Skipping local overrides
superset_app            | Starting web app...
superset_app            | /app/docker/docker-bootstrap.sh: line 50: /usr/bin/run-server.sh: No such file or directory

Same issue with latest superset helm chart, tag: 1.4.2.


Repository Not Found for url

I am facing this error
Repository Not Found for url: https://huggingface.co/api/models/jythjfjfh/common_voice_16_1_hi_pseudo_labelled/revision/main.
Please make sure you specified the correct repo_id and repo_type.
If you are trying to access a private or gated repo, make sure you are authenticated.
This is the terminal command I am running
python3 run_pseudo_labelling.py --model_name_or_path "openai/whisper-large-v3" --dataset_name "mozilla-foundation/common_voice_16_1" --dataset_config_name "hi" --dataset_split_name "train+validation+test" --text_column_name "sentence" --id_column_name "path" --output_dir "./common_voice_16_1_hi_pseudo_labelled" --wandb_project "distil-whisper-labelling" --per_device_eval_batch_size 1 --dtype "bfloat16" --attn_implementation sdpa --logging_steps 500 --max_label_length 256 --concatenate_audio True --preprocessing_batch_size 1 --preprocessing_num_workers 4 --dataloader_num_workers 4 --report_to "wandb" --language "hi" --task "transcribe" --return_timestamps True --streaming False --generation_num_beams 1 --push_to_hub

did you find any solution?
I face the same issue.

I also met this problem. How to solve it?
I just remove the --push_to_hubb label. But this is not a proper solution...

How was this issue solved?


Unable to control edge_alpha in ox.plot_graph

(what did I do, what did I expect to happen, and what actually happened)
Hello @gboeing,
thanks for the amazing repo!
I am using osmnx and networkx to visualize bicycle displacements in a given city. Instead of using the plot_routes command, I created a dataframe with the edges hit by cyclists, so that I can change the node_color, node_size, edge_color and edge_linewidth in the Graph according to the number of hits.
Everything works fine with the above arguments (and with edge_alpha, let's say, equal to 0.6), but returns an "TypeError: alpha must be a float or None" when I try to control the edge_alpha.
The thing I don't understand is that both edge_linewidth and edge_alpha require a Float, and the 2 functions ev and ea are basically identical.
What am I missing here?
I am using MacOS 10.14, Python 3.7, and OSMnx 0.8.2
Code snippet that independently and completely reproduces the issue

#creates 3 lists from the dataframe containing the hit edges and the number of hits
hu=list(df_counted["u_node"])
hv=list(df_counted["v_node"])
hc=list(df_counted["count"])

#for each edge, assign values to functions that control edge_color, edge_linewidth, and edge_alpha

for u,v,k in G.edges(keys=True, data=False):
    i = 0
    while i < n_unique_edges:
        if (u==hu[i] and v==hv[i]):
            ev[u,v] = 5 * (hc[i] / (hc[i] + 1))
            ec[u,v] = 'b'
            ea[u,v] = 0.5 * (hc[i] / (hc[i] + 1))
            i = n_unique_edges

        else:
            ev[u,v] = 0.5
            ec[u,v] = 'gray'
            ea[u,v] = 0.5 * (hc[i] / (hc[i] + 1))

        i = i + 1

fig, ax = ox.plot_graph(G, fig_height=7, node_color=nc, node_size=8, node_alpha=0.8, node_zorder=2,
edge_color=[ec[u,v] for u, v in G.edges(keys=False, data=False)], edge_linewidth=[ev[u,v] for u, v in G.edges(keys=False, data=False)],
edge_alpha=[ea[u,v] for u, v in G.edges(keys=False, data=False)])

You are passing plot_graph() an edge_alpha value of type list when as you can see in the docstring, it requires type float: https://osmnx.readthedocs.io/en/stable/osmnx.html#osmnx.plot.plot_graph
You didn't post your full error traceback, so some of this is guesswork, but this error is probably being thrown in matplotlib as I believe matplotlib requires a scalar for the alpha argument (hence, float rather than list).
If you want varying opacities, you can presumably leave the edge_alpha argument alone and instead pass a list of node_color values specified as RGBA, where A is the alpha value: https://matplotlib.org/api/colors_api.html

Thanks,
I am surely going to try with RGBA! Still, I don't get why my code works fine with edge_linewidth: I am passing a list, while it should accept only a float (as edge_alpha).
Here goes my full error traceback, sorry!```

Traceback (most recent call last):
File "route4.py", line 118, in
edge_alpha=[ea[u,v] for u, v in G.edges(keys=False, data=False)])
File "/miniconda3/lib/python3.7/site-packages/osmnx/plot.py", line 392, in plot_graph
lc = LineCollection(lines, colors=edge_color, linewidths=edge_linewidth, alpha=edge_alpha, zorder=2)
File "/miniconda3/lib/python3.7/site-packages/matplotlib/collections.py", line 1252, in init
**kwargs)
File "/miniconda3/lib/python3.7/site-packages/matplotlib/collections.py", line 170, in init
self.update(kwargs)
File "/miniconda3/lib/python3.7/site-packages/matplotlib/artist.py", line 888, in update
for k, v in props.items()]
File "/miniconda3/lib/python3.7/site-packages/matplotlib/artist.py", line 888, in
for k, v in props.items()]
File "/miniconda3/lib/python3.7/site-packages/matplotlib/artist.py", line 882, in _update_property
return func(v)
File "/miniconda3/lib/python3.7/site-packages/matplotlib/collections.py", line 782, in set_alpha
raise TypeError('alpha must be a float or None')
TypeError: alpha must be a float or None

Downstream, matplotlib allows a list on linewidths but it does not allow a list of alphas.

I was able to achieve my goal by passing the RGBA values to the edge_color and setting the edge_alpha=None, thanks again!

Hello PFerrix, I ave facing this same issue, could you share your fixed code, please? Thanks!

See also #429 for details.


chore: Update reflection database and create exemptions for new unimplemented DataTypes

Summary
Updates the built in reflection database (whilst making exceptions for two DataTypes that are listed by Roblox but not implemented yet.
Changes

Had to update the test snaps due to floating point precision issues; these are not material changes, and are likely due to a change to serde_yaml made a while back: https://github.com/dtolnay/serde-yaml/commit/6b836037b58ebb359e7c485fc6002b1e8214bd6c
Patch MeshId property of MeshPart to be ReadOnly (whilst it technically isn't true, it is not Scriptable and perhaps uniquely so behaves strangely if you try to update it in runtime. Not sure if this is the best solution, but it was all I could think of!)
Added exceptions for Font and DateTime datatypes (both of whom are either not available yet or not serializable - from my understanding)
Updated the .gitignore to ignore all target folders, for consistency

Suggestion: Perhaps we should update the reflection database every merge, via Actions?

@LPGhatguy Sorry about the delay here - no longer marked as a draft!

Hey, thanks for the PR!
I ended up doing a couple of these changes as separate commits so that they were a little easier to reason about.
How did target folders end up getting created in places other than the root folder for you? Even when I cd into rbx_binary and run cargo test or cargo build, it uses the target folder from the root of the project because it's in a workspace.

@LPGhatguy Sorry about the delay on this one .. I'm not sure how I managed that one! Probably just an accident.
I'll close the PR since it looks like each has indeed been replaced by other PRs that are now merged 👍


server: change database charset to utf8mb4

PoC for #166. yarn migration:up and yarn migration down works correctly for our local dump. Haven't tested it with a dump from production, yet.

Probably not needed anymore since we want to use the API in the future (instead of working with the DB directly)


[cluster-test] Switch all dependencies to rusttls

Cluster test was only libra component depending on openssl, we use rusttls everywhere else and should use it here as well

@bors-libra r+

:pushpin: Commit 29bd371 has been approved by ankushagarwal

:hourglass: Testing commit 29bd371a824456021b1128cc1f16da4265b13c7a with merge ed9b262b12926221ba5152a09997737dfa22d56b...

:sunny: Test successful - checks-actions_land_blocking_test, checks-circle_commit_workflow
Approved by: ankushagarwal
Pushing ed9b262b12926221ba5152a09997737dfa22d56b to master...

This PR makes me so happy :D bye bye openssl


fixed issues 6218(public access enforcement) & 6221 (special chars support in dinghy file)

Issue: https://github.com/spinnaker/spinnaker/issues/6218
Issue Summary: Private Azure Storage is not supported by Armory Spinnaker
Cloud Provider(s): AZS
Environment: DEV and Prod
Feature Area:
Description: When we convert the Azure Storage container access to private from public access, Spinnker front50 service unable to retrieve the pipelines from Azure Storage.
Steps to Reproduce: Configure Azure Storage container access to private from public access and try to pull the pipelines of any application with in Spinnaker UI.
Additional Details:
Observed that Spinnaker front50 container is expecting Azure blob storage container with public access enabled always. Application code is trying to set the permission to public access and getting exception “com.microsoft.azure.storage.StorageException: Public access is not permitted on this storage account.” as we disabled public access.
Mitigation / Fix: Removed the code snippet which is enforcing public access of expecting Azure blob storage container. Able to build & run the spinnaker front50 in local environment and getting the pipeline details from storage as expected.
Code snippet(actual exception was not getting logged due this it took more time for us to understand the cause)
Issue: https://github.com/spinnaker/spinnaker/issues/6221
Issue Summary: Issues with special character(space)handling in dinghy file during the new application & pipeline creation. Once we have space in one specific pipeline, it is breaking complete spinnaker pipelines which was integrated with AZS.
Cloud Provider(s):AZS(Azure)
Environment: DEV and Prod
Feature Area: Front50 get pipelines is returning empty set, even we have 50+ pipeline which does not have any special characters in it.
Description: Issues with special character(space)handling in dinghy file during the new application & pipeline creation. Once we have space in one specific pipeline, it is breaking complete spinnaker pipelines which was integrated with AZS.
Front50 get pipelines is returning empty set, even we have 50+ pipeline which does not have any special characters in it.
Steps to Reproduce:
Issues with special character(space)handling in dinghy file during the new application & pipeline creation. Once we have space in one specific pipeline, it is breaking complete spinnaker pipelines which was integrated with AZS.
Front50 get pipelines is returning empty set, even we have 50+ pipeline which does not have any special characters in it.
Additional Details:
As per analysis front50 service code is performing double encoding of special characters during cache refresh and load. Due to double encode, it is unable to locate the actual resource in Azure Storage Container.
Example Name: test spinnkerpipeline
Actual URL generated from code : test%2520spinnkerpipeline
Expected URL generated from code : test%20spinnkerpipeline
Proposed Fix: During the Container Blob Reference key generation, we have to decode the key to avoid double encode of special characters.
class: com.netflix.spinnaker.front50.model.AzureStorageService.java

Duplicate of #1018


Reflect name change from balena VPN to cloudlink

This change should reflect that balena VPN has been renamed to cloudlink.

Thanks for the review @vipulgupta2048 !

Namely, the masterclass and some instances in the SV API? Would you be willing to take those up in your own time?
Yes, I'll reach out to supervisor folks. Expect some PRs in the masterclasses. :joy:

Also, a suggestion. Can you add a definition to cloudlink on this page as well:
Done

@jaomaloy Are we sure that the changes you made to the CLI output for example (cmd usage docs, mis-use, update, process etc) that changed vpn -> cloudlink actually spit out those things now?

@rhampt , CLI is a good catch, I'll take a look at it and create an issue if there are possible vpn -> cloudlink instances

@jaomaloy I think if this is good to go, then you can certify your changes and merge this.

@balena-ci I self-certify!


Profile picture is not being gathered

Since it's a background image, it needs to be gathered via JS using page.$eval

Hey @leonardiwagner,
It's not a Background image since we can parse it at below Path

.pv-top-card-v3--photo .pv-top-card-section__image img

But, I tried getting that image and It returns null

solved at #35


Error: disallowed_useragent

I have follow Authenticating User Docs it show error:

Hi @nguyenthanhliemfc
This will most likely be a misconfiguration of the custom URL scheme in your Android app.


Azure Spring Apps rebranding

Hello Julien @jdubois ,
Could you please update the repo URL to https://github.com/microsoft/azure-spring-apps-training ? :)
Thank you,
Steve

This is unfortunately not possible, as it would break all the incoming links to this repo, and destroy its SEO.

After more discussion, the rebranding is done! Not sure what the impact on the SEO will be.


Remove call to column_for_attribute as it's depreciated in rails master

The title says it all, column_for_attribute is now depreciated to this pull requests removes the call and replaces it with a call to @object.class.columns_hash.

The method is not deprecated, only the return value of nil. Just for completeness.


Update base docker image guide

I hope the new structure won't break anything.

Whoa, never seen this error before :D


Update request: libheif 1.15.2 → 1.16.2

Package name: libheif
Latest released version: 1.16.2

Current version on the unstable channel: 1.15.2
Current version on the stable/release channel: 1.15.2

[x] Checked the nixpkgs pull requests

Notify maintainers

@gebner

Note for maintainers: Please tag this issue in your PR.

Done by PR #274003
(but it can take a while to reach whatever channels you use)


Alphafold-Multimer was run for more than 48h, I still couldn't get the structure of a complex with 4 subunits (total aa length: 2222+689+201+196).

Hi, author
python3 docker/run_docker.py \
  --fasta_paths=complex1.fasta \
  --max_template_date=2022-12-30 \
  --model_preset=multimer \
  --data_dir=/data  --gpu_devices=0,1,2,3 > complex1.log &

The program was always calculating the model until now.

I0107 01:49:08.176561<PHONE_NUMBER>59904 run_docker.py:255] I0107 01:49:08.175831<PHONE_NUMBER>97408 run_alphafold.py:191] Running model model_1_multimer_v3_pred_0 on complex1
I0107 01:49:08.178550<PHONE_NUMBER>59904 run_docker.py:255] I0107 01:49:08.178094<PHONE_NUMBER>97408 model.py:165] Running predict with shape(feat) = {'aatype': (3308,), 'residue_index': (3308,), 'seq_length': (), 'msa': (5799, 3308), 'num_alignments': (), 'template_aatype': (4, 3308), 'template_all_atom_mask': (4, 3308, 37), 'template_all_atom_positions': (4, 3308, 37, 3), 'asym_id': (3308,), 'sym_id': (3308,), 'entity_id': (3308,), 'deletion_matrix': (5799, 3308), 'deletion_mean': (3308,), 'all_atom_mask': (3308, 37), 'all_atom_positions': (3308, 37, 3), 'assembly_num_chains': (), 'entity_mask': (3308,), 'num_templates': (), 'cluster_bias_mask': (5799,), 'bert_mask': (5799, 3308), 'seq_mask': (3308,), 'msa_mask': (5799, 3308)}

The instance (p3.8xlarge) has four GPUs (4xNVIDIA V100).  All GPUs are available to calculate the model (--gpu_devices=0,1,2,3). But I noticed that 'run_alphafold.py' seems to use four GPUs, but why only the first one GPU memory is fulfilled?

How can I use the four GPUs with maximum memory usage at the same time to speed up the prediction?

Hi @echoduan, AlphaFold inference cannot be split over multiple devices. Most of the time elapsed for the first prediction will be MSA search. To get a final prediction for your structure, you should leave the script to run until completion, ideally without running additional tasks else on the workstation, as this could dramatically slow things down. Closing this issue, feel free to reopen this issue or open a new issue if this is still a problem.


Allow sites to specify PHP version

Each site should be able to specify their own PHP version via a php property under sites in the config, and Nitro should auto-install the required versions if they don’t exist.
sites:
 - hostname: example.test
   webroot: /nitro/sites/example.test/web
   php: "7.2"

Just an idea if this goes ahead, the machine should probably have a setting for the default CLI PHP version. or ability to change the CLI PHP version in use which ssh'd into the machine.

Sites are now PHP version specific as well as enabling xdebug per site. Please see the documentation here for upgrading instructions https://craftcms.com/docs/nitro/2.x

Jason you champ, thanks for making Nitro 2 so so good!!


Long user input doesnt get recognized

I have

context.Call(new FormDialog(new AskRoom(), options: FormOptions.PromptInStart),OnRoomQuestionAnswered);

If I enter a long input (for ex I want to book a room) function OnRoomQuestionAnswered doenst get callled. If I enter "I want to" or something short it gets called. Any idea?

What does AskRoom look like?  If your question requires an answer that is a single piece of text, you might consider using the prompts instead of form.

I have the same issue with prompts as well. when i give user 3 options and ask him to select one from it, skype doesnt show the text(until i restart my Skype). I have pasted my Sample input in #113 . Even with session.send() it has the same issue with long text. It works fine with webchat interface and emulator for me. I have not tried other channels.

Check the answer in #113 and if that does not address your issue, please reopen.


Add project_name as part of the container data

Changelog Description
This PR is to add project_name as part of the container data regarding to the implementation of https://github.com/ynput/ayon-core/pull/1005
Additional review information
Test along with the installed addon with https://github.com/ynput/ayon-core/pull/1005
Testing notes:

Launch Blender
Load Assets from any product type(s)

I guess we are not expecting more than that, right? As it looks correct to me as is now.

You should be able to completely manage it, like updating versions and even switch asset functionality (currently only within the project it was loaded from)


data-amount

Where can I specify the data-amount to be charged? Couldn't find anything in the documentation as well

The amount is charged using a charge object, you can check the documentation here -  https://stripe.com/docs/charges
Also, you must first tokenize the card information before proceeding to make charges,If you are not sure how to tokenize the data, use this part of the docs
https://stripe.com/docs/stripe-js
Also, have a look at Reference Section of StripeJs Elements for a better undestanding.
I hope everything is covered, the issue may be closed soon.

Thank you @pavittarx for explaining this and linking to the relevant sections of the documentation.  👍
@mohitgarg if you still have questions about your Stripe integration that can't be answered by reading the docs that @pavittarx linked to, please reach out to us using the form at https://support.stripe.com/email or come chat with us at #stripe on freenode.