[WIP] - Re-engineer dump workers

Start on model to pre-compute the dump data that can be easily serialized to a file.
This will be a polymorphic data model to support all the dumps and the plan is to store the data in a json blob, e.g. { id: 1, subject_id: 2, annotations: {}, workflow_id: 1, ... }. @marten @zwolf thoughts on the model name, data format?
Review checklist

[ ] First, the most important one: is this PR small enough that you can actually review it? Feel free to just reject a branch if the changes are hard to review due to the length of the diff.
[ ] If there are any migrations, will they the previous version of the app work correctly after they've been run (e.g. the don't remove columns still known about by ActiveRecord).
[ ] If anything changed with regards to the public API, are those changes also documented in the apiary.apib file?
[ ] Are all the changes covered by tests? Think about any possible edge cases that might be left untested.

Not because my idea is better (or worse) but just because we apparently had different ideas, let me also share what I thought the approach would be. I had a ClassificationExportRow in mind, with columns for each of the CSV columns, plus maybe one or two for scoping queries. So not polymorphic.
I would definitely suggest going with ExportRow rather than Export.

So we leave the subjects, workflow, etc dumps running the way they are and pay the read IOPS costs on them still (not as expensive as classifications i agree)?
ExportRow sounds good!

Oh, and keep in mind that one subject has multiple lines in the subjects.csv dump: one for each workflow it appears in.

How do we do subject retirement info that's currently in the classifications export? Have to remember to do cache invalidation on that.

retirement cache is setup per batch processed - https://github.com/zooniverse/Panoptes/blob/76b30dbd53012b1da7d986da725db3ff9839c5a6/app/workers/classifications_dump_worker.rb#L51

Yeah but presumably that will now get stuck into the ExportRow?

ah - good point, now I get you.

I guess, it's still too much to ask that they join a fresh subject dump and their classification dump to determine retirement status.

Closing this as it's out of data post the refactor in # - I still think this idea has merit:

Use the export row as the main source of data export.
backfill the export row model in batches while running dumps for new work

This way we know the latest export point and dont re-compute the state every time we serialize the dumps. Also noting this doesn't have to exist in panoptes either...


TO golang -- adds validation to regions

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR/1166/
Test PASSed.


Enable Apply chaining on ResourceOutput types in Go SDK

Enables a nice API for the awsx library:
See:

https://github.com/pulumi/pulumi/pull/9513

@stack72 I didn't see this build error locally, but it appears to have stopped the CI:

error: failed to gather package metadata: problem gathering data sources: 1 error occurred:
	* could not find docs for data source 'aws_ecr_credentials'. Override the Docs property in the data source mapping. See type tfbridge.DocInfo for details.

make: *** [Makefile:27: schema] Error 255
	```

Implemented with update to 3.31.1


Tilemap can't be edited in monaco editor

There is no tilemap in Monaco editor.

Dupe of https://github.com/microsoft/pxt-arcade/issues/1788


Update core.py

Can you provide more information for the reason for this change?

I see you have added a issue for this
#7
Including that to the description


Install wig_to_bigwig from TS

TODO: Please replace this header with a description of your pull request.
Installation sequence for tool-installers

[ ] Test using @galaxybot test this
[ ] Inspect CI output for expected changes
[ ] Deploy using @galaxybot deploy this if test install was successful
[ ] Merge this PR

@galaxybot test this

@galaxybot deploy this

We should also disable the distribution version?

It's hidden, I would not do that, some people are really into reproducibility.


TypeScript: SCIMMYRouters does not extend express.Router

Despite SCIMMYRouters extending express.Router in the JavaScript source, this is not the case in the generated Typescript type definitions:
    /**
     * Method invoked to determine a base URI for location properties in a SCIM response
     * @callback AuthenticationBaseUri
     * @param {express.Request} req - the express request to provide the base URI for
     * @returns {String|Promise<String>} the base URI to use for location properties in SCIM responses
     * @private
     */
    /**
     * SCIMMY HTTP Routers Class
     * @class SCIMMYRouters
     */
    export default class SCIMMYRouters {
        /**
         * Construct a new instance of SCIMMYRouters, validate authentication scheme, and set SCIM Service Provider Configuration
         * @param {Object} authScheme - details of the means of authenticating SCIM requests
         * @param {String} authScheme.type - SCIM service provider authentication scheme type
         * @param {AuthenticationHandler} authScheme.handler - method to invoke to authenticate SCIM requests
         * @param {AuthenticationContext} [authScheme.context] - method to invoke to evaluate context passed to SCIMMY handlers
         * @param {AuthenticationBaseUri} [authScheme.baseUri] - method to invoke to determine the URL to use as the base URI for any location properties in responses
         * @param {String} [authScheme.docUri] - URL to use as documentation URI for service provider authentication scheme
         */
        constructor(authScheme?: {
            type: string;
            handler: AuthenticationHandler;
            context?: AuthenticationContext;
            baseUri?: AuthenticationBaseUri;
            docUri?: string;
        });
    }

This forces to cast any instance of SCIMMYRouters into express.Router when trying to use it.
NB: I have quickly tested to add an @extend JS annotation, I did not work. I open this issue because I am a bit in a hurry but I will be glad to fix it myself if you have no time for that either.

Hi @fflorent,
Since the source files are written in JavaScript, the type declarations are automatically generated by the TypeScript compiler from the JSDoc comments. Unfortunately, the TypeScript compiler is actually quite limited in its handling of declaration generation from JSDoc comments, and does not seem to retain the express.Router extension (despite explicitly specifying it with the @extends tag as you mentioned). As a test, when I manually added the missing extends express.Router to the generated declaration, TypeScript then complained about express.Router not being a constructable class or function - despite this being perfectly acceptable in JavaScript.
The solution involved declaring an intermediary interface that extends express.Router, and adding the @implements JSDoc tag (which is retained by TypeScript) to the SCIMMYRouters class. This was defined in a preauthored routers.d.ts file that is merged with the generated declaration file. Testing locally, this seemed to appease TypeScript, which then seemed to recognise SCIMMYRouters instances as suitably extending the express.Router type.
This change should be available in the next release (v1.2.2) - please let me know if this resolves the issue for you.
Thanks,
Sam

Hi @sleelin,
Sorry for the late reply and thank you!
Unfortunately, I could not remove the casting here despite having upgraded to v1.2.2:
https://github.com/gristlabs/grist-core/blob/7ace7f80acf292b8e22b337bb01b64676059d501/app/server/lib/scim/v2/ScimV2Api.ts#L47
(currently in this PR: https://github.com/gristlabs/grist-core/pull/1199)

Thanks for following up @fflorent
It looks like there's a discrepancy between how the TypeScript compiler handles imported type references in JSDoc comments - when I originally tested, it was automatically adding this line at the start of the generated declarations, so I opted to omit it when merging the declaration file. It seems that didn't actually happen when the release build ran, and doesn't seem to be happening in my testing now either 😕
Since there's no reference to the express module, TypeScript doesn't recognise what the SCIMMYRouters interface is extending, which is why you still needed to cast it. I'm reopening the issue as it's clearly not resolved, but matters are also complicated by the fact that the express types aren't actually very useful here - the express.Router "class" which is extended does some magic that makes TypeScript not recognise it as a class that can be directly extended, and technically SCIMMYRouters should actually be exposing a method that returns a router instead of being an instantiable class itself.

Hi @fflorent,
I've just published v1.3.1 of scimmy-routers which should finally solve this issue and allow you to use the SCIMMYRouters instance without needing to cast it to express.Router!
If possible, please update to this version and report back as to whether this is now properly resolved.
Thanks,
Sam


Optimize K-means

Optimized the averaging step of K_means to use non-moving averages and arrays instead of hashmaps. The code also updates the new centroid in-place. Also fixed deprecation warning on all benchmarks in linfa-clustering (other benchmarks may have the same warnings). I'm curious as to the effect of my changes on benchmark speeds, since my machine isn't suited to running benchmarks.

Hi, thanks for the PR! I've been busy today but I will gladly review this tomorrow 👍🏻

I'm taking some more time to review this because on my machine these changes actually bring the performance down significantly, at least for what concerns the benchmarks, but I am also getting some outliers so I'd like to investigate this a little more. Also, I want to make sure that the performance is not impacted by the change in the bencher (it shouldn't but I would still like to check).
These are the results that I get when I run cargo bench k_means on my machine:

On branch master:

Benchmarking naive_k_means/10: Warming up for 3.0000 s
naive_k_means/10        time:   [1.0439 ms 1.1075 ms 1.1703 ms]
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
  5 (5.00%) high severe
naive_k_means/100       time:   [5.1121 ms 5.4133 ms 5.7409 ms]
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
naive_k_means/1000      time:   [35.332 ms 37.381 ms 39.503 ms]
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild
Benchmarking naive_k_means/10000: Warming up for 3.0000 s
naive_k_means/10000     time:   [233.40 ms 243.14 ms 253.00 ms]

On branch kmeans-opt:

naive_k_means/naive_k_means/10
                        time:   [5.3906 ms 8.3629 ms 11.660 ms]
Found 19 outliers among 100 measurements (19.00%)
  19 (19.00%) high severe
naive_k_means/naive_k_means/100
                        time:   [16.939 ms 24.063 ms 31.794 ms]
Found 24 outliers among 100 measurements (24.00%)
  24 (24.00%) high severe
Benchmarking naive_k_means/naive_k_means/1000: Warming up for 3.0000 s
naive_k_means/naive_k_means/1000
                        time:   [66.666 ms 94.997 ms 126.49 ms]
Found 19 outliers among 100 measurements (19.00%)
  1 (1.00%) high mild
  18 (18.00%) high severe
Benchmarking naive_k_means/naive_k_means/10000: Warming up for 3.0000 s
naive_k_means/naive_k_means/10000
                        time:   [284.80 ms 436.71 ms 613.42 ms]
Found 13 outliers among 100 measurements (13.00%)
  3 (3.00%) high mild
  10 (10.00%) high severe

Have you tried running the benches on your machine? Do you get similar results?
My machine is definitely not the top for benchmarking so the fault for the worse performance could be on the machine itself, If you get different results from me please share them so that I know the results that I should expect

It's also slower on my machine, but I'm on a 2-core i7-6500U that runs faster without Rayon so I didn't take the results too seriously. I mainly wanted someone with a better machine to confirm my results. I'm also curious as to what flamegraphs you get from the benchmarks because my flamegraphs for both version indicated that the assignment step was the bottleneck, not the averaging step.

I'm seeing similar results as you in my benchmarks, but my flamegraphs indicate that the averaging step isn't the bottleneck.

These are my results with cargo bench k_means -- --verbose.
With just the benchmark fixes
naive_k_means/naive_k_means/10
                          time:   [1.1976 ms 1.2291 ms 1.2623 ms]
                          change: [-14.458% -7.8971% -1.1469%] (p = 0.03 < 0.05)
                          Performance has improved.
  Found 4 outliers among 100 measurements (4.00%)
    4 (4.00%) high mild
  slope  [1.1976 ms 1.2623 ms] R^2            [0.5064945 0.5050213]
  mean   [1.2193 ms 1.2825 ms] std. dev.      [132.98 us 187.69 us]
  median [1.1873 ms 1.2554 ms] med. abs. dev. [105.86 us 182.06 us]
  naive_k_means/naive_k_means/100
                          time:   [7.4920 ms 7.7974 ms 8.1227 ms]
                          change: [-0.9476% +5.6558% +12.591%] (p = 0.10 > 0.05)
                          No change in performance detected.
  Found 3 outliers among 100 measurements (3.00%)
    2 (2.00%) high mild
    1 (1.00%) high severe
  mean   [7.4920 ms 8.1227 ms] std. dev.      [1.2859 ms 1.9477 ms]
  median [7.1343 ms 7.8455 ms] med. abs. dev. [921.75 us 1.7574 ms]
  naive_k_means/naive_k_means/1000
                          time:   [46.172 ms 48.628 ms 51.174 ms]
                          change: [+3.4581% +10.997% +19.390%] (p = 0.00 < 0.05)
                          Performance has regressed.
  Found 1 outliers among 100 measurements (1.00%)
    1 (1.00%) high mild
  mean   [46.172 ms 51.174 ms] std. dev.      [10.782 ms 14.653 ms]
  median [44.011 ms 49.039 ms] med. abs. dev. [8.9856 ms 15.345 ms]
  naive_k_means/naive_k_means/10000
                          time:   [282.75 ms 296.74 ms 311.53 ms]
                          change: [-5.9549% +0.8865% +8.3379%] (p = 0.81 > 0.05)
                          No change in performance detected.
  Found 3 outliers among 100 measurements (3.00%)
    3 (3.00%) high mild
  mean   [282.75 ms 311.53 ms] std. dev.      [59.775 ms 87.050 ms]
  median [261.28 ms 300.02 ms] med. abs. dev. [46.431 ms 80.503 ms]

With all changes
naive_k_means/naive_k_means/10
                          time:   [5.0783 ms 7.6619 ms 10.578 ms]
                          change: [+312.70% +512.85% +725.71%] (p = 0.00 < 0.05)
                          Performance has regressed.
  Found 19 outliers among 100 measurements (19.00%)
    1 (1.00%) high mild
    18 (18.00%) high severe
  mean   [5.0783 ms 10.578 ms] std. dev.      [11.457 ms 16.126 ms]
  median [1.0480 ms 1.1344 ms] med. abs. dev. [127.76 us 227.01 us]
  naive_k_means/naive_k_means/100
                          time:   [23.279 ms 33.091 ms 43.804 ms]
                          change: [+195.19% +324.39% +472.32%] (p = 0.00 < 0.05)
                          Performance has regressed.
  Found 24 outliers among 100 measurements (24.00%)
    24 (24.00%) high severe
  mean   [23.279 ms 43.804 ms] std. dev.      [41.019 ms 62.016 ms]
  median [5.4170 ms 6.4924 ms] med. abs. dev. [1.2616 ms 2.6658 ms]
  naive_k_means/naive_k_means/1000
                          time:   [77.660 ms 111.70 ms 148.71 ms]
                          change: [+57.755% +129.70% +219.54%] (p = 0.00 < 0.05)
                          Performance has regressed.
  Found 18 outliers among 100 measurements (18.00%)
    18 (18.00%) high severe
  mean   [77.660 ms 148.71 ms] std. dev.      [140.19 ms 217.17 ms]
  median [29.247 ms 33.316 ms] med. abs. dev. [6.4509 ms 13.765 ms]
naive_k_means/naive_k_means/10000
                          time:   [337.02 ms 513.33 ms 720.10 ms]
                          change: [+9.3530% +72.989% +143.89%] (p = 0.03 < 0.05)
                          Performance has regressed.
  Found 10 outliers among 100 measurements (10.00%)
    10 (10.00%) high severe
  mean   [337.02 ms 720.10 ms] std. dev.      [730.82 ms 1.2520 s]
  median [170.21 ms 188.99 ms] med. abs. dev. [34.806 ms 56.924 ms]

The perf problems seems to be caused entirely by the outliers, since the median is so much lower than the mean.

I've tried profiling the code with flamegraph as well as measuring the times directly and compute_centroids has never shown up as a bottleneck. The only way to improve benchmark performance beyond the original version was to track averages using an empty HashMap. Any other way of tracking averages (using a vector, using a pre-filled HashMap) ends up severely degrading the performance. Vector/Array bounds-checks may have been the culprit in earlier iterations, but I have absolutely no idea why this solution is so much better. One thing to note is that hashing functions aren't called for HashMaps under the size of 32, so this solution (and the solution in master) may perform differently on more than 32 clusters.

I've tried profiling the code with flamegraph as well as measuring the times directly and compute_centroids has never shown up as a bottleneck. The only way to improve benchmark performance beyond the original version was to track averages using an empty HashMap. Any other way of tracking averages (using a vector, using a pre-filled HashMap) ends up severely degrading the performance. Vector/Array bounds-checks may have been the culprit in earlier iterations, but I have absolutely no idea why this solution is so much better. One thing to note is that hashing functions aren't called for HashMaps under the size of 32, so this solution (and the solution in master) may perform differently on more than 32 clusters.
To see how weird this is, go to line 272 of algorithm.rs and add
(0..n_clusters).for_each(|i| {
    averages.insert(i, (Array1::zeros(n_features), F::from(0.0).unwrap()));
});

and see how that affects the benchmarks.

I will try to run the benchmarks now. This seems really strange, one thing which stood out to me that when you are pre-filling both hashmap and vector your performance degrades, so probably there lies the problem. In commit 561d9b70006b6d you could also use a 2d array directly with (num_clusters + 1, num_features) where the last row is your count, then normalize and drop the last row.

Without changes to algorithm.rs:
naive_k_means/naive_k_means/10
                        time:   [1.5642 ms 1.6362 ms 1.7137 ms]
                        change: [-9.0982% -3.8257% +1.9977%] (p = 0.19 > 0.05)
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe
Benchmarking naive_k_means/naive_k_means/100: Collecting 100 samples in estimated 5.9021 s (6                                                                                naive_k_means/naive_k_means/100
                        time:   [8.9471 ms 9.3102 ms 9.6769 ms]
                        change: [-0.8811% +4.6403% +10.543%] (p = 0.10 > 0.05)
                        No change in performance detected.
Benchmarking naive_k_means/naive_k_means/1000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.9s, or reduce sample count to 80.
Benchmarking naive_k_means/naive_k_means/1000: Collecting 100 samples in estimated 5.8972 s (                                                                                naive_k_means/naive_k_means/1000
                        time:   [45.065 ms 47.675 ms 50.373 ms]
                        change: [-11.544% -4.7009% +3.0629%] (p = 0.21 > 0.05)
                        No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild
Benchmarking naive_k_means/naive_k_means/10000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 28.8s, or reduce sample count to 10.
Benchmarking naive_k_means/naive_k_means/10000: Collecting 100 samples in estimated 28.840 s                                                                                naive_k_means/naive_k_means/10000
                        time:   [318.61 ms 336.68 ms 358.62 ms]
                        change: [+13.476% +22.160% +31.329%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

With changes:
naive_k_means/naive_k_means/10
                        time:   [1.3526 ms 1.3857 ms 1.4196 ms]
                        change: [-17.931% -14.437% -10.805%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild
Benchmarking naive_k_means/naive_k_means/100: Collecting 100 samples in estimated 5.3013 s (6                                                                                naive_k_means/naive_k_means/100
                        time:   [11.366 ms 12.488 ms 13.704 ms]
                        change: [+21.267% +34.128% +49.536%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe
Benchmarking naive_k_means/naive_k_means/1000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.3s, or reduce sample count to 70.
Benchmarking naive_k_means/naive_k_means/1000: Collecting 100 samples in estimated 6.3159 s (                                                                                naive_k_means/naive_k_means/1000
                        time:   [46.636 ms 50.043 ms 53.764 ms]
                        change: [-4.1604% +4.9669% +14.524%] (p = 0.30 > 0.05)
                        No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe
Benchmarking naive_k_means/naive_k_means/10000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 26.3s, or reduce sample count to 10.
Benchmarking naive_k_means/naive_k_means/10000: Collecting 100 samples in estimated 26.330 s                                                                                naive_k_means/naive_k_means/10000
                        time:   [277.86 ms 291.15 ms 305.68 ms]
                        change: [-20.009% -13.525% -6.7332%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  6 (6.00%) high mild
  1 (1.00%) high severe

cpu info

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 69
model name	: Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz
stepping	: 1
microcode	: 0x26
cpu MHz		: 1846.742
cache size	: 3072 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips	: 4591.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 69
model name	: Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz
stepping	: 1
microcode	: 0x26
cpu MHz		: 1686.786
cache size	: 3072 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips	: 4591.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 69
model name	: Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz
stepping	: 1
microcode	: 0x26
cpu MHz		: 1706.545
cache size	: 3072 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 2
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips	: 4591.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 69
model name	: Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz
stepping	: 1
microcode	: 0x26
cpu MHz		: 1791.553
cache size	: 3072 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 2
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips	: 4591.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual

I've looked a bit more and the problem seems to be with the benchmarks. For some reason if I move all the initialization code out of the benchmark function I observe more reasonable performance numbers. I'm still observing runtime spikes with cluster size of 100, which looks to be RNG-dependent.

which is still strange, because only the code in bencher.iter(|| ) should be relevant

I will try to run the benchmarks now. This seems really strange, one thing which stood out to me that when you are pre-filling both hashmap and vector your performance degrades, so probably there lies the problem. In commit 561d9b7 you could also use a 2d array directly with (num_clusters + 1, num_features) where the last row is your count, then normalize and drop the last row.

just saw that this was your original version

reducing the number of iterations to 100 seems to make the benchmark results more reasonable

can you try to reinitialize the rng for each new parameter set, this gives me much more consistent results

I did some more digging and the real reason behind the perf issues was because some inputs run for all 1000 iterations, which doesn't usually happen in the benchmarks. This happens because somewhere along the way the centroids are calculated as NaN, which messes up the algorithm.

Performance issues were caused by empty clusters producing an average of NaN (bug in my code, not master). I'm setting empty centroids to 0 right now but there may be a better value. This may also indicate an issue with the random cluster initialization we're using (can we use a better initialization?).

Maybe it could be a good idea to look into kmeans++ for center initialization? I see that the kmeans crate implements it so maybe that could be used as a reference. Provided that you have the time of course 👍🏻

I'm getting similar issues with rand::rngs::SmallRng but with thread_rng everything seems to work fine. Cluster collapse should terminate the algorithm and return an error IMO or the API has to handle them specifically (either by setting centroids to NaN or using optional centroids but we can't store everything in contiguous memory anymore).
The cluster collapse is caused by using standard normal distributions with a shift (https://github.com/rust-ml/linfa/blob/master/algorithms/linfa-clustering/src/utils.rs#L44) which makes the sample distribution very sparse and then using random initialization by sampling the dataset just four time which makes it very likely that two centroids are in the same cluster. @Sauro98 yep that would solve it. For a quickfix: you can add a parameter to utils::generate_blobs for the standard deviation and then set σ=10.0

btw: have you heard about the iai project? It's from the same author as criterion and can directly measure the cache/memory access and instruction counts instead of the wall-clock. The beauty is that this can be reliable deployed to CI systems (as opposed to criterion.rs) and only need a single pass. We could probably use the metrics for a dashboard like the rustc performance and publish it with every new release

so what are the actions here now: should we merge the PR? Does the performance gain justify it to have two implementations, one for increment K-Means and not?

btw: have you heard about the iai project? It's from the same author as criterion and can directly measure the cache/memory access and instruction counts instead of the wall-clock. The beauty is that this can be reliable deployed to CI systems (as opposed to criterion.rs) and only need a single pass. We could probably use the metrics for a dashboard like the rustc performance and publish it with every new release

That project sounds promising, especially since it can be integrated with the CI system, it would give benchmarks some more importance, which is cool, and having a dashboard like the one you linked in the website I believe could get some more people interested in the project, I will definitely explore it a little bit 👍🏻

so what are the actions here now: should we merge the PR? Does the performance gain justify it to have two implementations, one for increment K-Means and not?

This evening I'll be able to run the benchmarks again with the latest changes to feel the extent of the improvements. Why do you mention two implementations? Doesn't this PR just overwrite the existing one?

This evening I'll be able to run the benchmarks again with the latest changes to feel the extent of the improvements. Why do you mention two implementations? Doesn't this PR just overwrite the existing one?

sorry I meant that in context of scikit: it contains two implementation KMeans and MiniBatchKMeans. The moving average is currently only used in our single-fit implementation, but could be used for a incremental version (by implement https://github.com/rust-ml/linfa/blob/master/src/traits.rs#L38) in the future.

The PR changed the averaging algorithm from moving average to standard average. For handling cluster collapse I vastly prefer returning an Error. I'm definitely against setting to NaN because it causes perf drops in the benchmarks. Center initialization and incremental fit are out of scope for this PR.

sorry I meant that in context of scikit: it contains two implementation KMeans and MiniBatchKMeans. The moving average is currently only used in our single-fit implementation, but could be used for a incremental version (by implement https://github.com/rust-ml/linfa/blob/master/src/traits.rs#L38) in the future.

Just by reading scikit-learn's user guide it would seem to imply that they do use two different approaches for the mini-batches and regular k-means, so maybe it would make sense to keep the non-moving average implementation for k-means if it increases performance. With a future incremental fit implementation in mind, maybe it is better to keepthe helper mod so that there already is a moving average struct available?

Performance issues were caused by empty clusters producing an average of NaN (bug in my code, not master).

What is the behaviour of the implementation in master in such cases?

when we merge I would like to keep the moving average implementation as well: just ran the benchmark new version vs. old version:
Benchmarking naive_k_means/10: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.0s, enable flat sampling, or reduce sample count to 50.
naive_k_means/10        time:   [1.4124 ms 1.4485 ms 1.4861 ms]
                        change: [-5.3063% -2.6335% +0.1510%] (p = 0.06 > 0.05)
                        No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
naive_k_means/100       time:   [8.6071 ms 8.9256 ms 9.2593 ms]
                        change: [-5.9313% -1.2229% +4.2687%] (p = 0.64 > 0.05)
                        No change in performance detected.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild
Benchmarking naive_k_means/1000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.6s, or reduce sample count to 80.
naive_k_means/1000      time:   [54.948 ms 58.965 ms 63.304 ms]
                        change: [+6.7258% +16.787% +27.683%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
Benchmarking naive_k_means/10000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 32.6s, or reduce sample count to 10.
naive_k_means/10000     time:   [303.20 ms 317.37 ms 332.34 ms]
                        change: [-7.3419% -1.4865% +5.0620%] (p = 0.65 > 0.05)
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

On my machine:
master:
Benchmarking naive_k_means/10: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.8s, enable flat sampling, or reduce sample count to 60.
naive_k_means/10        time:   [1.2289 ms 1.2922 ms 1.3656 ms]
                        change: [-14.414% -5.6616% +3.8992%] (p = 0.25 > 0.05)
                        No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
  4 (4.00%) high mild
  7 (7.00%) high severe
naive_k_means/100       time:   [5.3927 ms 5.6523 ms 5.9290 ms]
                        change: [-3.3688% +3.8261% +11.909%] (p = 0.33 > 0.05)
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild
naive_k_means/1000      time:   [38.752 ms 41.189 ms 43.716 ms]
                        change: [-7.9562% +1.1587% +10.389%] (p = 0.80 > 0.05)
                        No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild
Benchmarking naive_k_means/10000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 27.1s, or reduce sample count to 10.
naive_k_means/10000     time:   [250.79 ms 261.14 ms 271.86 ms]
                        change: [-0.2075% +5.6954% +11.811%] (p = 0.06 > 0.05)
                        No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

kmeans-opt:
Benchmarking naive_k_means/naive_k_means/10: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.6s, enable flat sampling, or reduce sample count to 60.
naive_k_means/naive_k_means/10
                        time:   [1.2419 ms 1.2810 ms 1.3237 ms]
                        change: [-9.7741% +1.1279% +12.853%] (p = 0.87 > 0.05)
                        No change in performance detected.
Found 16 outliers among 100 measurements (16.00%)
  3 (3.00%) low mild
  6 (6.00%) high mild
  7 (7.00%) high severe
naive_k_means/naive_k_means/100
                        time:   [3.6659 ms 3.8628 ms 4.0931 ms]
                        change: [+1.2612% +8.0094% +15.932%] (p = 0.02 < 0.05)
                        Performance has regressed.
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) high mild
  8 (8.00%) high severe
naive_k_means/naive_k_means/1000
                        time:   [23.170 ms 24.227 ms 25.356 ms]
                        change: [-10.836% -3.4829% +4.3460%] (p = 0.38 > 0.05)
                        No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
  8 (8.00%) high mild
  2 (2.00%) high severe
Benchmarking naive_k_means/naive_k_means/10000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 11.1s, or reduce sample count to 40.
naive_k_means/naive_k_means/10000
                        time:   [104.28 ms 107.76 ms 112.22 ms]
                        change: [-25.250% -19.050% -11.754%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe

I would say that the improvement in speed is quite good 👍🏻

What is the behavior of the implementation in master in such cases?
Since master uses moving averages, it just sets the centroid to 0.
I'll look into the perf of moving average vs standard average. Also, the benchmarks for this PR has more outliers than master, which is strange.

okay then let's merge and collect what's missing in an issue :+1: I have to rerun the benchmark cause my cache was invalid
18.03.2021 21:00:07 Yuhan Lin @.***>:

What is the behavior of the implementation in master in such cases?
Since master uses moving averages, it just sets the centroid to 0.
I'll look into the perf of moving average vs standard average. Also, the benchmarks for this PR has more outliers than master, which is strange.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub[https://github.com/rust-ml/linfa/pull/97#issuecomment-802248017], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AAHRRKJ5INF7Z35JAHWSJWLTEJLUDANCNFSM4ZF7D3QQ].
[data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEIAAABCCAYAAADjVADoAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAAnSURBVHic7cEBDQAAAMKg909tDwcUAAAAAAAAAAAAAAAAAAAAAPBjRFIAASHKmKkAAAAASUVORK5CYII=###24x24:true###][Verfolgungsbild][https://github.com/notifications/beacon/AAHRRKLFJVEQLCSK3RRRF23TEJLUDA5CNFSM4ZF7D3Q2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOF7IVKUI.gif]

nah you can always find the file in the git history, no need to add dangling files :laughing:

without changes:
naive_k_means/10        time:   [1.4574 ms 1.4977 ms 1.5400 ms]
                        change: [+1.0481% +4.3349% +7.5120%] (p = 0.01 < 0.05)
                        Performance has regressed.
naive_k_means/100       time:   [8.8972 ms 9.2581 ms 9.6355 ms]
                        change: [-1.6644% +3.7247% +9.4769%] (p = 0.19 > 0.05)
                        No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
Benchmarking naive_k_means/1000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.1s, or reduce sample count to 80.
Benchmarking naive_k_means/1000: Collecting 100 samples in estimated 6.0698 s (100 iterations                                                                                naive_k_means/1000      time:   [46.133 ms 48.795 ms 51.541 ms]
                        change: [-24.440% -17.248% -9.3971%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild
Benchmarking naive_k_means/10000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 29.2s, or reduce sample count to 10.
Benchmarking naive_k_means/10000: Collecting 100 samples in estimated 29.184 s (100 iteration                                                                                naive_k_means/10000     time:   [312.75 ms 327.89 ms 345.63 ms]
                        change: [-3.0299% +3.3159% +9.9612%] (p = 0.36 > 0.05)
                        No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

with changes:
naive_k_means/naive_k_means/10
                        time:   [1.3423 ms 1.3659 ms 1.3937 ms]
                        change: [+4.5462% +6.5943% +8.6149%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild
naive_k_means/naive_k_means/100
                        time:   [5.5286 ms 5.6984 ms 5.9126 ms]
                        change: [+12.994% +16.524% +20.862%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe
naive_k_means/naive_k_means/1000
                        time:   [30.290 ms 31.026 ms 31.879 ms]
                        change: [+4.2413% +6.5199% +9.3345%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 17 outliers among 100 measurements (17.00%)
  8 (8.00%) high mild
  9 (9.00%) high severe
Benchmarking naive_k_means/naive_k_means/10000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 13.2s, or reduce sample count to 30.
naive_k_means/naive_k_means/10000
                        time:   [126.78 ms 127.38 ms 128.09 ms]
                        change: [+0.0205% +1.1580% +2.1152%] (p = 0.02 < 0.05)
                        Change within noise threshold.
Found 13 outliers among 100 measurements (13.00%)
  5 (5.00%) high mild
  8 (8.00%) high severe

:+1: if no objections are raised I would merge this now


Custom close function in prompt

I'd like to custom close action in prompt

I'm building an application with a wrapper of prompt and I'll want to control the close event. I couldn't find props to control that, so I guess this is an interesting new feature in prompt.

I think you mean this? @close="myFunctionClose"

Yes

I think you mean this? @close="myFunctionClose"

Another thing I guess is essential in a dialog is a boolean to enable or disable click outside to close

@iagocavalcante when you make a wrapper with .sync atributte, this try to change a prop, (and the props is immutable), but you can make a computed with get and set to change this.
look a example with you code.
https://codesandbox.io/s/n3zq0l6zqj

I understand this now, in this case, I can call the method inside the computed to manipulate some states.
But I think this is better to manipulate the event @close or @hide in the vs-prompt to pass the function I desire, what you think?

it's a very good idea


Reloading notifications does not remove selection state

The cell that was previously selected will stay selected

Bug Report Dump (Auto-generated)

Version 1.22.0<PHONE_NUMBER>)
Device: iPad7,3 (iOS 12.0)
TestFlight: true

Is it supposed to? If you select a cell and view the issue, then reload, you're still viewing the issue and it should be selected.
Am I missing something?

Fair enough!


Throwing exception updating state React component

Hey there, following bellow my current issue:

This exception was always throwing for the last tests to be executed. But, when I execute this tests with the cache clean, this exception happens on tests no stopping them. For what saw, this exception is not related with the test broked but seems to me a issue about immediate execution of setState promise.
// the handler to callback
onRefresh() {
  this.setState({ refreshing: true });
  setTimeout(() => this.setState({ refreshing: false }), 300);
}

// the assert
it('inbox notifications list is refreshing', () => {
  const flatlist = getFlatlist();
  flatlist.props.onRefresh();
  expect(getFlatlist()).toMatchSnapshot();
});

I'm really, really confused about that, any helps it's valid for me! Thanks.

I have troubles understanding what's going on in this code. Please format it better, ideally create a repository with a repro, and post on Stack Overflow or Discord, as this is not a help forum. If you're certain such behavior is a bug, feel free to create a new issue (obeying the template).


Splitting the flatcc runtime dependency into parsing and building.

Even though LTO will strip what we don't use it still means we are
compiling more than we need and risking inclusion of things we don't want.
I still don't like how the compiler depends on the runtime flatcc files
but that's a task for another day.

https://gist.github.com/hcindyl/dd72759018fc3d414ff22343921ed969
rv32 baremetal target build.ninja snippet with libflatcc_runtime.a dependency based on this PR.

Darn, will take a look. Good idea checking the ninja output!

Ok think I fixed it - no more flatcc_runtime dep from simple_embedding_test!

The RV64 Linux and ARM64 Android runtimes still build libflatcc_runtime.a and flatcc/runtime/builder.c and flatcc/runtime/json_printer.c objects.
https://source.cloud.google.com/results/invocations/a1ed384c-062d-4d36-a3e9-9033318e8766/targets/iree%2Fgcp_ubuntu%2Fcmake%2Flinux%2Friscv64%2Fpresubmit/log
https://source.cloud.google.com/results/invocations/bd2f0a0a-55c4-4dbb-a82b-4ed66ee30a0c/targets/iree%2Fgcp_ubuntu%2Fcmake%2Fandroid%2Farm64-v8a%2Fpresubmit/log
The dependency is not shown in rv32 baremetal, so we could address that in a later PR if you want unblock baremetal first.

I think (but could be wrong a second time ;) that those are because that build is producing Linking CXX executable iree/tools/iree-dump-module - but the non-tool files shouldn't be requiring it.


please keep tagging releases

Last skydns release was a while ago and Kubernetes 1.3.0 already vendored unreleased snapshot of skydns. :(
Please tag new release, vendoring by tags is useful.
Also semantic versioning is good:

Debian Upstream Guide.
http://semver.org/
https://en.wikipedia.org/wiki/Software_versioning

Thanks.

Ack.

Thanks.
Also tagging could be useful for compatibility with gopkg.in.


Look for other strategies to avoid overfitting

Since we currently have a small amount of training data, it would be relevant to check overfitting approaches more in depth.

Never close an issue without referencing to the commit that fixes the issue, or if no fix is provided, explain why the issue can be closed.

Stale


All layers flash on page load

Since CSS is added via JavaScript, the "hide" class isn't available until the javascript loads. This causes all layers to flash on page load before being hidden.

This is fixed in release 2.1.0
Closing this ticket.


How can i apply rflint with sonarQube ?

I need to test all the files in the projects and get a report for all .robot  in one file, is this possible ?
Another thing, can we add other rules and integrate it with SonarQube ?

HI, Is there any update for this issue?

I haven't ever used sonarQube so I can't specifically speak to that. As for
getting a report of all robot files in one file, you can certainly do that
in the normal way by redirecting the output to a file.
$ rflint -r /path/to/tests > rflint.txt

On Wed, Sep 18, 2019 at 2:48 AM lolotoms<EMAIL_ADDRESS>wrote:

HI, Is there any update for this issue?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/boakley/robotframework-lint/issues/52?email_source=notifications&email_token=AAISMYXCED5T3HBA3AQX4Z3QKHMNPA5CNFSM4FEKPAZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD67EUGY#issuecomment-532564507,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAISMYWS3Q33F6PX6CTKQQ3QKHMNPANCNFSM4FEKPAZA
.


Smileys panel size on small resolutions

#91

@mike-gusev-ae thanks for the chat conversations, much things become clear. Nice fix.


Importing Multiple Tickers - NaN Values

Hi,
If I import one ticker, I get no NaN values, but when importing multiple tickers together, it gives lots of NaN values in the dataset

I'm having this problem today also.

I'm having this problem today also. I changed the code to import tickers one at a time which is several times slower and it leaves the row blank instead of populating the value with NaN.
I created a colab notebook that recreates the error. https://colab.research.google.com/drive/1W_alDteC5qtdxSTtSpqRbOBm6qZMPUpI?usp=sharing

I have switched to just importing one ticker at a time and now I am getting the NaN values again. I assume this is just yahoo lowering the rate limit even more?

Options:

stop spamming
fetcher smarter
yfinance_cache


pgstac seems to require globally unique item IDs contrary to STAC spec

I've been loading some STAC items into my pgstac instance, and it seems that pgstac requires globally unique Item IDs. For example, if I create a new collection, and then add an item that I've already added, with the same ID as an existing item, then I get the following error:
$ pypgstac load items bec-items.json
How to deal conflicting ids (insert, insert_ignore, upsert) [insert]:
loading bec-items.json into items using insert
Traceback (most recent call last):
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\Scripts\pypgstac.exe\__main__.py", line 7, in <module>
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\typer\main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\typer\main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\pypgstac\pypgstac.py", line 43, in load
    typer.echo(asyncio.run(load_ndjson(file=file, table=table, dsn=dsn, method=method)))
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\asyncio\base_events.py", line 642, in run_until_complete
    return future.result()
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\pypgstac\load.py", line 246, in load_ndjson
    await load_iterator(f, table, conn, method)
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\pypgstac\load.py", line 230, in load_iterator
    await copy(iter, table, conn)
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\pypgstac\load.py", line 134, in copy
    await conn.copy_to_table(
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\asyncpg\connection.py", line 859, in copy_to_table
    return await self._copy_in(copy_stmt, source, timeout)
  File "C:\Users\rwilson3\Documents\mambaforge\envs\anglo\lib\site-packages\asyncpg\connection.py", line 1041, in _copy_in
    return await self._protocol.copy_in(
  File "asyncpg\protocol\protocol.pyx", line 506, in copy_in
asyncpg.exceptions.UniqueViolationError: duplicate key value violates unique constraint "items_p1999w52_datetime_id_idx"
DETAIL:  Key (datetime, id)=(2000-01-01 00:00:00+00, 4096a_0_10.0_32_7_-46) already exists.

Is this intentional? The STAC spec states that:

The ID should be unique within the Collection that contains the Item.

This is correct, currently pgstac is set to that items have a globally unique id. The key word in the spec is should as opposed to must. There is some level of overhead in keeping multi-column primary keys rather than single column and the use case / catalog that PGStac was initially created for prioritized reducing that overhead.

Thanks - that makes perfect sense.

currently pgstac is set to that items have a globally unique id.

Actually, I don't think this is true. The example above shows that there is a uniqueness constraint on the combination of datetime and id. If someone inserted another Item with the same id but a different datetime then it would not violate the uniqueness constraint. This issue appears to have been raised for PgSTAC < 0.5.0; for versions of PgSTAC >= 0.5.0 the same would apply to unique combinations of collection and id.
This does not change the original response to this post, but does have implications for stac-utils/stac-fastapi#381.


NPM Version

Where is the supported NPM version for this project listed?
Description
I'm running into a lot of errors running this and after searching around it looks like a few others have mentioned that the npm version needs to be reduce for this project to work. If that is the case can that be listed somewhere in the README.MD to save others time?
Can you add "engines" to save us sometime please
{
    "engines": {
        "node": "version"
    }
}

Hello
Not sure what is the last version of NPM supported for this project.
I would have to recheck/retest.
Thanks in advance.

Hello
Last version of NPM bundled with Node18 LTS is supported.
The project needed some update because some libs where broken and build was not working as intendeed.
Has been updated in the meanwhile should be working now.
Thanks a lot!


As an enthusiastic Squeak user IWLT, when in window type overview, I would like to switch and select a window type, thus focusing on the most recently used window of this window type

13 balloons


Fee selector - no translated option

Steps:

Go to Send transaction.
Click fee selector.
Result: One option is not translated

Fixed.

Fixed.


Python 3 compatibility

(without leaving Python 2 behind)

This is the main reason I hate Python 3 :)


Unable to test when mocking decorated component via MockComponent

Hi,
Is there any workaround to deal with an error while testing with spectator & ng-mocks?
Child component has a decorator @UntilDestory() and when I use MockComponent(ChildComponent) I cannot run test.
Getting error like with disabled Ivy (but Ivy is enabled).
Cannot read property 'get' of undefined
Sth like this:
describe('ParentComponent', () => {
 let spectator: Spectator<ParentComponent>;

 const createComponent = createComponentFactory({
    component: ParentComponent,
    detectChanges: false,
    declarations: [MockComponent(ChildComponent)]
  });

 beforeEach(() => {
    spectator = createComponent();
  });

 test('should be created', () => {
   expect(spectator.component).toBeTruthy();
 });
});

@Component({
  selector: 'parent',
  template: '<child></child>'
})
export class ParentComponent {}

@UntilDestroy()
@Component({
  selector: 'child',
  template: 'anything'
})
export class ChildComponent {}

If I use shallow: true w/out using MockComponent() everything is working fine.
Any idea?
   "@angular/common": "~9.1.11",
   "@angular/compiler": "~9.1.11",
   "@angular/core": "~9.1.11",
   "@ngneat/until-destroy": "^7.3.2",
   "@ngneat/spectator": "^5.13.1",
   "jest": "^26.4.2",
   "ng-mocks": "^10.1.2",

@krzyhan have you checked this section https://github.com/ngneat/until-destroy#potential-pitfalls?

@krzyhan could you provide a minimal reproducible example? This error is thrown Cannot read property 'get' of undefined when Ivy is actually disabled and it can be disabled in testing environment.

I'm closing the issue. If you can reproduce it, let us know.

Sorry guys,
I updated Angular & and all dependencies to the latest version and it's working fine.
Thanks for replying.


Set up an empty Express Boilerplate with dotenv

What does this PR do?
This PR aims at setting up express and server project
Description of Task to be completed?

Set up an empty Node.js - express application.

By this We mean a full directory with all folders that you think developer will need. this might include but not limited to
src/ controllers, models, routes helpers,validations, etc
3.Test

feel free to contact the rest of the team and get ideas from them because this is the first skeleton of the application.

Install and setup dotenv for the API to manage environment variables. Ensure packages like sequelize and passport among the rest can fetch data about the current environment correctly via dotenv. Add documentation on how to utilize dotenv in the API.

Set the server and make sure it is running

How should this be manually tested?

Clone the repo
Cd into that repo
run  npm install
After that run npm run dev

What are the relevant pivotal tracker/Trello stories?

#184752075
#184752076
#184752078

The setup is great for me.


Return the Features Scores

I think it would be nice to return the features scores. Or have it as optional parameter.

Sorry for the delay on this. I've just added an additional parameter called return_scores (default to false), which returns the computed relevance and redundancy.

Thanks for this great package and this special feature! Is it also possible to additionally return the final score used for sorting? I would like to make the number of features actually used dependent on a threshold on this score. Or is there a simple formula so that I can calculate this using redundancy and relevance?

Did you find any formula so that I can calculate this using redundancy and relevance?

Did you find any formula to calculate this using redundancy and relevance?

according to the original mRMR author, the final score for MID is calculated by max(V-W), where W is redundance and V is relevance.
It is be max(v/W) for MIQ scheme.
https://home.penglab.com/proj/mRMR/BIBM07_mRMR_071103_handout.pdf


Angular Schematics for adding and updating this library into projects

Using Angular Schematics for adding and updating this library into projects
Detailed description
This is a feature request for Angular schematics to be used in this project to add dependencies for new projects, e.g.: VMware Clarity design system uses Angular schematics is added to new projects.

Describe in detail the issue you're having.

Documentation for adding this library and its dependencies into an existing Angular code base is inconsistent between the official Carbon Design System website and this library's documentation.
There are also steps to modify SCSS files which is repetitive for new projects.

Is this a feature request (new component, new icon), a bug, or a general issue?

General issue.
Additional information

https://angular.io/guide/schematics

This is definitely something we'd like to have 💯 We just haven't had the bandwidth to take it on ... If you or anyone else wanted to contribute a first pass at this it would be very appreciated!


Update process-related docs

naming.md: add the rule to not move tags
releases.md: how to handle self-references in projects
README.md: add Xen-troops process section
Add description for the inter-repo dependencies

Comments are addressed. Ready for re-review.

I have updated naming.md and releases.md.
Please re-review.

For the textual part (except YAML, discussion of which eventually may introduce more changes that I will need to re-review), with one minor comment above: Reviewed-by: Artem Mygaiev<EMAIL_ADDRESS>

Comments are addressed.

" and require management review & approval." is added;
special handling of the moulin-based projects is removed

After the latest comment, only the second commit was modified "releases.md: describe release branches".
Ready for the re-review.

This approach looks good to me
Acked-by: Oleksii Moisieiev<EMAIL_ADDRESS>

s/naming.md: updates on tags and branched/naming.md: updates on tags and branches

The misprint in the first commit message is fixed.
Reviewers are added.
Should be merged only after the https://github.com/xen-troops/meta-xt-prod-devel-rcar/pull/143

Please proofread and fix language. With that done
Reviewed-by: Artem Mygaiev<EMAIL_ADDRESS>


Fixed a couple of issues with the I64_TRUNC_FXX_U opcodes.

Re-enabled a couple of the skipped conversion tests that were failing due to bugs in the unsigned float to i64 truncation implementations, and fixed those bugs.

Looks like you are catching up with the codebase pretty quickly, please, let us know if something is not working well for you in the development experience (e.g. you should be able to run individual tests from the IDE etc. etc.).

Thanks, @andreaTP - so far so good. The trickiest bit was getting used to Maven (briefly dabbled with Ant many years ago, then switched to Gradle and never looked back) & playing around with the test gen last night. Aside from a couple idiosyncrasies around test generation (need the right sequence of clean/verify sometimes), it's been pretty easy going - I've found it to be a clean and well-organized codebase.

Apparently Thomas is busy those days, I think that this is an improvement going in the right direction.
Also, the diff is pretty self-contained, all of the tests are already passing plus the 2 re-enabled here.
I'll take ownership to make sure we fix it on top of the latest main whenever we hear back from Thomas.

Sorry guys for the lack of response. I'm currently dealing with some non-work related stuff that needs my attention.
Please go ahead and continue at your own pace. I hope I can join you guys soon again.

Sorry guys for the lack of response. I'm currently dealing with some non-work related stuff that needs my attention.

It's no problem! Let's not hold up any PRs unless it's dire. As soon as you get one approval you can merge. We can always circle back on our own timeline since we're all async, just leave a comment that the person can come check when they're ready. Because we don't have any prod users or anything, the risk of stagnating is lower than the risk of pushing potentially problematic code.

Everyone of you who have commit access is trustworthy enough to merge any changes on your own :)

Sounds good!


sap.m.SinglePlanningCalendar missing growing functionality

It would be nice if the SinglePlanningCalendar would support a growing functionality.
The problem is if the entity set which is bound against the appointments aggregation returns more entries than the size limit of the ODataModel, some appointments won't be shown. As a workaround it's possible to just increase the size limit but this is not very performant since a lot of entries have to be read by the OData service.
I think this could be implemented in such a way:
First the displayed start date and end date of the current view must be known. If that the case then the bound entity set can be filtered by the range of those. For example if the view shows appointments between 2020-03-01 and 2020-03-07:
/Appointments?$filter=StartDate ge datetime'2020-03-01T00:00' and EndDate lt datetime'2020-03-08T00:00'

Is the date range which is shown in the current view changed, a new requests is triggered. That way each request only returns the appointments which have to be displayed.

@mzimnn You can possibly use the event startDateChange to fire off a new oData request with date range filters

I have implemented a simple custom control.
It does nothing fancy, it just calls the OData service for the date range of the current view but does not check if the range was already read. Furthermore, it's not working with the month view right now because I'm using the internal method _getFirstAndLastRangeDate() to get the date range and the month view doesn't return a correct end date.

I have implemented a simple custom control.
It does nothing fancy, it just calls the OData service for the date range of the current view but does not check if the range was already read. Furthermore, it's not working with the month view right now because I'm using the internal method _getFirstAndLastRangeDate() to get the date range and the month view doesn't return a correct end date.

@mzimnn I am using the control as well, and I found the month view to be problematic too. The route I chose was to load a month's worth of data in one go, and checking on navigation between views/dates, if the data is already loaded or not for the startDate/endDate of the view. To work out the correct start date/end date of the view, I used moment.js to add relevant number of days depending on the view selected by the user. For week - 7, fortnight- 15, month -42, as the control shows a grid of 6x7 boxes(so your month view can display previous or next month days). Add 42 to the startDate of the month view date and you can then work out if the data is already loaded or not in the oData model, by querying by the entity key. Hope this provides some alternative ideas to you until the UI5 teams provides a proper "growing" solution.

@ravish-garg Thanks for giving some insights about how you solved this.
I have found the method SinglePlanningCalendarMonthGrid._getVisibleDays() which calculates all days shown in the month view. That way the start date and end date can easily be retrieved.
I added this to my custom control which means it supports now the month view as well.
For now the custom control pretty much covers my use case but an official growing functionality would be a lot nicer.

Hello @mzimnn,
Thank you for sharing your enhancement proposal. I've created an internal incident<PHONE_NUMBER>. The status of the issue will be updated here in GitHub.
Regards,
Dobrin

This enhancement will be covered in backlog item BGSOFUIBALKAN-5876.


Update ADAL Android link

the directory listing link has some MIME type issue and is downloaded instead of viewed.
Call the link Maven to be consistent with adal4j link

@0ac1 : Thanks for your contribution! The author, @CelesteDG, has been notified to review your proposed change.

@0ac1 : Thanks for your contribution! The author, @rwike77, has been notified to review your proposed change.

@0ac1  Change looks good, thanks.  #sign-off


Figure out MVP backend architecture/technologies

The prototype right now is a static-site with no backend. Depending on what the MVP looks like, we'd need a solution for:

persisting users
authenticating users
supporting multiple users simultaneously
logging/analytics
serving the client app (maybe)
making requests to the data hub routing services (maybe)
anything else?

Acceptance criteria:

a host provider is selected (likely HHS' AWS, but need a confirmation)
an MVP database is chosen (dynamo, RDS for postgres, self-hosted postgres, or other)
an MVP backend is chosen (node/express, lambda/api gateway, something else?)

For data storage / sync / transfer I recommend we use existing solutions rather than rolling our own. Implementation would require changing the way we store documents a tiny bit, but would be overall much easier. Examples include:

https://github.com/share/sharedb : "ShareDB is a realtime database backend based on Operational Transformation (OT) of JSON documents."

Realtime synchronization of any JSON document
Concurrent multi-user collaboration
Synchronous editing API with asynchronous eventual consistency
Realtime query subscriptions
Simple integration with any database - MongoDB, PostgresQL (experimental)
Offline change syncing upon reconnection
etc.

https://pouchdb.com/ : "PouchDB is an in-browser database that allows applications to save data locally, so that users can enjoy all the features of an app even when they're offline. Plus, the data is synchronized between clients, so users can stay up-to-date wherever they go."

... designed from the bottom-up to enable easy synchronization between different databases.
It is framework-agnostic, and you can use it with Angular, React, Ember, Backbone, or your framework of choice
your backend needs to speak the CouchDB replication protocol.
PouchDB is just a script tag and 46KB (gzipped) away in the browser

https://turtle-db.github.io/ : "Web applications will work seamlessly online or offline, and developers can leave the backend to turtleDB - it will handle all data synchronization and conflict resolution between users. Works with MongoDB out of the box!"

fairly new, haven't used this one

https://github.com/pubkey/rxdb/blob/master/README.md : "To replicate data between your clients and server, RxDB provides modules for realtime replication with any CouchDB compliant endpoint and also with custom GraphQL endpoints. "

first read about this here: https://news.ycombinator.com/item?id=24340802
seems a bit cutting edge but also moving very fast
fairly in depth example here: https://slacker.ro/2020/01/06/building-an-offline-first-web-app-with-rxdb-hasura/

If possible it probably makes sense to just default to any such system that our devs have experience with, if anyone else has used one of these?

Thanks for the initial research.
I haven't used any of these, but we can check with the the new engineers next week. Of the options listed, Pouch and RxDB seems interesting! It's a bit hard to evaluate without playing around with them.
I've used firebase way back when for real-time syncing, but sounds like google may be out of the question. I've used dynamodb for tiny scrappy projects, but not for any real-time work.
If it helps to evaluate, here are some of the queries we'd need:
MVP must-haves:

uploading patients (bulk)
adding to queue
removing from queue
submitting test result

MVP maybes:

uploading patient (single)
archiving patient
update global settings
update ordering provider
exporting csv (should happen on the backend, could happen on the frontend)
updating testResult (ie: has it been reported or not)
adding device
updating device
removing device

Post-MVP:

patient search (post-mvp)
...lots of stuff, tbd

Theoretically on the must-haves, those can all be sync based operations (sync from local). I think most of the maybes should be the same (also doable via syncing local data). That makes this a lot easier.

AWS also has appsync
https://aws.amazon.com/appsync/?sc_ichannel=ha&sc_icampaign=acq_awsblogsb&sc_icontent=mobiledev-resources
Might be a little more work to setup for an MVP tho, but could be a better long-term solution

Some of this is already in other tickets so I'm going to pull them out of this for now:
User Management / Auth is here: https://github.com/CDCgov/prime-central/issues/89
Analytics (placeholder) here: https://github.com/CDCgov/prime-central/issues/67

@sharmaneil not sure we can guarantee our hosting platform will be AWS. @RickHawesUSDS has more details here. But wanted to warn you early on.

In that case, I'm going to try out one of the databases in Pete's suggestions instead of picking something AWS-specific. If we don't decide soon, it'll create tech debt if we ever want to switch databases, but that may be unavoidable.
For our non-prod apps, cloud.gov seems to have a limited set of managed database options:

redis
s3
postgres, mysql, oracle
https://cloud.gov/docs/services/relational-database/

Don't know what the LoE is to spin up a different database

@sharmaneil Suggest Postgres because it is open-source, works on dev machines, and can be deployed anywhere including the environments being considered. You can also just store JSON in Postgres, if that makes things go faster.

Building off of the recs and research so far, here's an idea:
MVP: postgres + graphql  + hasura (backend)

hasura - an open source Backend as a Service to create graphql APIs

this should minimize our need to write our own backend, and comes with a lot of handy tooling around graphql
comes with JWT auth support
has postgres adapters

graphql

comes with subscriptions built in, so it lays the groundwork for sync (for multiple simultaneous users).
USDS has engineers who know and use this, so we have lean on them for support if needed

this can be dockerized and deployed in any environment, which is helpful if we end up going the self-hosting route
cons:

could take some time to learn and setup
hasura has a paid, managed version. It is unclear if the benefits of hasura exist in both the open source and paid versions.

Post-MVP: add RxDB for offline support

RxDB - this is a wrapper around client-side databases. It uses observables, which is a javascript programming paradigm that deals with streaming data.
RxDB could be configured to use graphql to sync with the db instead of the more common couchDB path.
cons:

RxDB + graphql is a little fuzzy in terms of documentation
more things to learn and setup
not sure how this plays with redux

Not wed to anything; we can discuss more early next week.

Another option for offline support:  Redux Persist
Pros:

seems easy to use, esp since we are already using redux
has support for optimistic, pessimistic offline requests, as well as rollbacks if things don't work

Cons:

doesn't have any proactive conflict resolution strategy baked in, but maybe the app is simple enough that we don't care.

Some of these options still look like building a lot of stuff to make them work (i.e. Redux Offline seems to primarily be built to handle network resilient API calls rather than data synchronization out of the box, so we have to build that API). I think we'd be best off avoiding anything more complex that basic data sync for early stages, but we keep coming back in this direction so maybe that's the way we need to go.

@pete-usds - can you give an order of magnitude or time difference for what it means to build a lot of stuff vs. not build a lot of stuff? Want to make sure we're coming from the same set of assumptions about what is acceptable vs. not.

I think we should have a sync discussion with the expanded eng team today/tomorrow to see what skillsets folks have and figure out what path makes sense. Doesn't seem like we're moving toward a consensus, and half the engineers aren't in this thread.
I'd like to roll out a locally-running solution to all of this -- wired to the frontend -- for this sprint, so making a decision soon here would be helpful

@sharmaneil I think GraphQL makes a lot of sense for the PDI backend. No experience with Hurusa, but it looks like a fit on the surface.

Did the synchronous discussion happen? Also, let me know if I can be helpful on:

[ ]  shared understanding of what the priorities are for the next prototype version (ie: multi-user, sync, offline, etc)

Decision
Based on the latest discussion with @RickHawesUSDS @benwarfield-usds and @nickrobison-usds, the tech stack decision is:

Java Spring Boot
Postgres
GraphQL

Rationale
Benefits

we need to move fast, and If Ben and Nick would be doing most of the backend work, it makes sense to do what they are comfortable with
GraphQL seems like a meaningful quality-of-life upgrade over REST from a front-end perspective, and Java <-> GraphQL bindings are established (ie: https://www.graphql-java.com/tutorials/getting-started-with-spring-boot/)
USDS has experience in graphQL, so we have folks we can lean on if needed.
this is a familiar tech stack to contractors

Tradeoffs

the server code should be relatively small given the size of our app, so we can always do a rewrite later, esp if we get more javascript folks and fewer java engineers in the future. USDS has a lot of node/TS engineers, so this might prove a reality
unless engineers are comfortable with both javascript and java, our team would be split into frontend vs backend instead of full stack.

Final acceptance criteria checkbox is addressed by this Epic from Alicia: https://github.com/CDCgov/prime-central/issues/105
tl;dr: multi-user sync, but not offline.
Going to go ahead and close this out, woo!


Facebook's "BSD+Patent" License situation?

Its more like an inquiry rather than a issue, on where do Superset stand with the current Facebook's BSD+Patent situation. Apache has moved it to the Category X and have given a time-frame of 31st August (to all Apache projects using components with aforementioned license) to port to other solutions.
What other alternatives to React are being discussed and is there any Roadmap for that?
Thanks

Closing in favor of #3148


Implement listCommands

Closes #155

Moves the shared files to handler
Created a command.go in handler to store a map of all commands

There are merge conflicts now. May you please resovle them?

Should be good now!

Thank you!


File verifiers are run regardless of whether the contents have changed

Description
File verifiers run regardless of whether the contents have changed.  This makes it expensive to run expensive verifiers (because they run on every chef run), as well as introduces chef-client failure modes where a verifier begins failing despite there being no chef-deployed change to the file's contents (e.g. the verifier binary may have changed)
Chef Version
Tested against 13.10.0 but I'm confident this is also the case in master
Platform Version
CentOS8
Replication Case
Define a file (or template) resource with a verify property set, and the contents of the rendered file are already correct, but observe from the chef-client debug logs that verifiers are still run.
Client Output
[2021-03-11T08:03:01-08:00] DEBUG: Running verification[visudo -c -q -f %{path}] on /tmp/....

I'll submit a PR for a fix for this shortly.

Closed with merge.


Mute partio error prints

Description
This patch mutes the error printed to std::err from inside Partio when it fails to read a pointcloud.  OSL's pointcloud_search and pointcloud_find already reports errors "the proper way" via errorfmt, so a failed read will still thrown an error. Plain prints like this can sneak past log-limiting filtering  and result in very large log files.
Currently, running something like
shader partioerror(){
float test[1];
if(pointcloud_search("non-existing",P,1.0,1,"test",test))
printf("Success");
}
in testshade, will print
Partio: No extension detected in filename
ERROR: pointcloud_search: could not open "non-existing"
with the patch, it will just print
ERROR: pointcloud_search: could not open "non-existing"
Tests
I haven't added any tests yet
Checklist:

[X ] I have read the contribution guidelines.
[X] I have updated the documentation, if applicable.
[X ] I have ensured that the change is tested somewhere in the testsuite (adding new test cases if necessary).
[X ] My code follows the prevailing code style of this project. If I haven't
already run clang-format v17 before submitting, I definitely will look at
the CI test that runs clang-format and fix anything that it highlights as
being nonconforming.

@olegul You need to click the right things to add yourself to the CI. You're already approved on the SPI end, but you need to accept it.


add the option to crop silence segments in training set curation

First we try to crop from the end of the dataset
Then, we try from the beginning.
If that doesn't work we crop from silence segments larger than the window_size.
This is done by preventing training windows from starting at points within the silence segments.
Since silence segments can follow non-silence segments or precede them the amount of data being ignored actually depend on whether the silence segment happens to be right in the beginning of the file or right in the end. We make sure to account for that.

@yardencsGitHub I think I see at a high level what you're doing here, and I'm willing to merge in if it makes it possible for you to create training sets of smaller duration and we're sure it works without introducing bugs in anything else
Next time please let's discuss proposed enhancements, and how we might implement them, before you go off and work a whole bunch on your own.
I made some changes for clarity; they should not have changed the behavior of what you added (I hope).
But there's a couple things that are not clear to me. Main things are

[ ] why specifically window size + 2 time bins? Right now this seems like a "magic number" in the code
[ ] we also have a "magic number" where we assume that there are unlabeled segments, and that their label is zero. Instead of this I think we should do the following:

check whether'unlabeled' in labelmap

which we already pass in to the crop method as a parameter
if so, put it in a variable and use it in the code block where you find the onsets and offsets of silent periods
if not, fail gracefully by raising an informative error

I think it might be easier to discuss face-to-face instead of via messages on a PR. I'll send you an email -- let's talk by Zoom tomorrow sometime

@yardencsGitHub this last commit adds changes we just went through
please do test as discussed and then once you think it's working, I will squash the fixup commits into yours, then merge in

Summary of tests with the last commit:

Ran 'learncurve' with a total training data set of 24000 sec from canary llb16

I was able to create subsets of 150 sec or larger.
Without this branch, when only allowing cropping from the beginning or end of the dataset, I could reliably subset 360 sec training sets or larger.

When requiring datasets too small for reliable curation I was able to 'exit' the runs in known points involving the validity checks for the desired cropping.

Thank you @yardencsGitHub -- I am going to be extra paranoid and re-run a couple of the BirdsongRepository learning curves to test further. But that should finish up today and then I'll squash + merge in

I re-ran with Bird00 and Bird01 from the BirdsongRecognition dataset. Neither crashed.
I did not validate the results from Bird00 but ran with all dataset durations used previously, [ 30, 45, 75, 120, 180, 480,]
For Bird01 I ran with just [30, 45, 75] then inspected the metrics csv after the learncurve ran without error. Accuracy was ~95%, roughly the same as before.
There is still some prettying up of the code that could be done but I will resist the urge and just merge in, so we can move forward with other things.


Re-pushed changes from ACAG Sprint 1 due to disappeared working branch

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/cloud-adoption-guide/intermediate-stage/overview.md
:white_check_mark:Succeeded
View

docs/cloud-adoption-guide/intermediate-stage/subscription-controls.md
:white_check_mark:Succeeded
View

docs/cloud-adoption-guide/intermediate-stage/subscription-multiple.md
:white_check_mark:Succeeded
View

docs/cloud-adoption-guide/intermediate-stage/subscription-ownership.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Thanks for the feedback. We definitely need to shorten the time to merge PRs and the associated feedback loop - this content was in my original PR from last Friday (2/23). Telmo and I agreed we'd still push it, because acag-intermediate-working is a ongoing working branch with frequent merges as we refine the content. Because it's from a week ago, a lot of this content will be replaced or restructured based on what Telmo and I have been working on together this week (and early next).

That said, I'll add any feedback items above that might be missing from the working outline for governance content that Telmo and I reviewed this week.


How to dynamically change the dimensions of a Draggable element?

It looks like style attribute gets overridden automatically so I haven't been able to set width or
height of a Draggable element even when I try to force it.
For example, I am rendering multiple Draggable elements in a loop and would like each to have it's own dimensions based on model attributes, e.g. a garden with multiple beds of different sizes. I've tried inline CSS, wrapping the Draggable element in a div, etc. but nothing works.
this.props.plantings.map((planting) => {
  return(
    <Draggable grid={[10, 10]} bounds="parent" {...dragHandlers} style={{width: planting.width, height: planting.length}}>
         <span className='box'>
           {planting.crop_name}
          </span>
     </Draggable>
   )
})

You need to use the <DraggableCore> component and then control its lifecycle yourself, if you wish to programmatically adjust it.


New Pageflow Logo

Use new Pageflow logo in default theme and editor blank slate.

Coverage remained the same at 99.156% when pulling 7c0f18a7fde96094d17cb502e0451916db499dd9 on tf:logo-changes into aa7620ff61c456d3394e6e2f4549d7b5ac784dd5 on codevise:master.

Coverage remained the same at 99.156% when pulling 7c0f18a7fde96094d17cb502e0451916db499dd9 on tf:logo-changes into aa7620ff61c456d3394e6e2f4549d7b5ac784dd5 on codevise:master.


⚠ Mobitopup has degraded performance

In b201fb7, Mobitopup (https://napcard.voz.tech) experienced degraded performance:

HTTP code: 200
Response time: 11596 ms

Resolved: Mobitopup performance has improved in 1810df4 after 17 minutes.


update Terraform sample to use getSecret permissions instead of GetCert

What happened in your environment?
Once we release Ratify RC7, We need to update Terraform sample to  use getSecret permissions instead of GetCert.
Both should be updated:
https://github.com/deislabs/ratify/blob/main/terraform/azure/main.tf#L82
https://github.com/duffney/secure-supply-chain-on-aks/blob/main/terraform/main.tf#L133
What did you expect to happen?
No response
What version of Kubernetes are you running?
No response
What version of Ratify are you running?
No response
Anything else you would like to add?
No response
Are you willing to submit PRs to contribute to this bug fix?

[ ] Yes, I am willing to implement it.

@duffney Are you able to update Terraform sample to use getSecret permissions?

I've got the change ready, but it looks like the latest version of the helm chart isn't updated with the getSecret change and fails to retrieve certificate information to validate the signatures. What are your thoughts on adding both permissions to support the two variants?


Optimizes project loading in few scenarios

Main changes are in 2a920f2 and 421ea3b

2a920f2: When we try to load ancestor tree for finding all references or other such multi project operations, earlier we would load all child projects if project referenced the project in which we are searching. That is say if we are looking for something in src\compiler it would load whole tree set by src/tsconfig which is a solution file. But we dont need to load all children, we need to load only those that reference compiler directly or indirectly. In our case that might be all cases but in big mono repo like scenarios thats not true, so this only loads the projects that truly could reference compiler.

421ea3b When opening project or any other operation that involves finding default project for a file, earlier we would load project tree one by one. So if the default project is referenced indirectly, first that indirect project will be loaded only to find that it does not contain default project. Now, we just query what's the projectReference the file redirects to and try operation on that first so this avoids having to load these indirect projects in file open or reload like scenarios. So give this in our source code scenario if src\tsconfig.json did not refererence src\compiler then opening in file in compiler would first load services project and then src\compiler before this change. Now it will directly. For purpose of this understanding you also have to assume that tsconfig in compiler is not named tsconfig but something else which is when solution is found instead of default project.

2a920f2 LGTM, but I didn't really understand 421ea3b (because I'm unfamiliar with the concepts).

I’m in the same boat as @amcasey. Could you give another example of what the latter change does?

Consider this folder structure:
|-compiler/compiler.ts
|-services
| -testrunner
|-tsconfig.json
| - compiler.tsconfig.json
|- services.tsconfig.json
| - testrunner.tsconfig,json
// tsconfig.json
{ "references":[  { "path": "services.tsconfig,juson" }, { "path": "compiler.tsconfig.json"} ]

// services.json
{ "references":[  { "path": "compiler.tsconfig,juson" } ]

In this scenario, when compiler.ts is opened, the first tsconfig found is tsconfig.json is solution that references two projects.
Before this change, we woild get Resolvedreferences of tsconfig.json and load them one by one (and recursively their own resolved references) till we find the project in which compiler.ts is not project reference source file. So we would load project tsconfig.json. services.tsconfig.json and then compiler.tsconfig.json to finally find the defaut project for compiler.ts
Withi this since resolved project references when tsconfig.json is loaded (which has to be loaded to know whether compiler.ts is par of it or not) we already have fileNames that services.tsconfig.json referes to and compiler.tsconfig.json refers to. so instead of loading services.tsconfig.json, we load compiler.tsconfig.json directly. That is exactly what https://github.com/microsoft/TypeScript/pull/41126/commits/421ea3bbc12f4472649262f4daad07adf5416120#diff-118c4b8aa4d105b15ea845c35502f62698a445fb0ded70523e8f7e6867c0fb1cL2198 shows.


Version stripe API to fix test breakage

What kind of a pull request is this?

QA / Code Review

Code Checklist

[ ] The code follows dotkom code style
[ ] The code passes the defined tests
[ ] I have added tests for the code I added
[ ] I have provided documentation for the code I added
[ ] The code is ready to be merged

(Possible) Breaking Changes

[ ] The changes are backwards compatible

[ ] I have updated the build configuration
[ ] These changes requires changes to configuration in production

[ ] I have applied the required changes in production
[ ] I cannot apply the required changes in production before this is deployed.

Description of changes
Screenshot(s) if appropriate

2500 PR's whooo 🍰


Can't find service cron.service on CentOS

The service is called crond on CentOS

Fixed in https://github.com/nozel-org/serverbot/commit/43c8911fc75fb1504fb8062d68f7c5826cb9b29b


added missing services to scripts and fixed a URL typo in README

fixed LunarCrush URL under FAQ
created services files for missing in scripts

@NeoEGY thanks a lot!


Compressing depth frames exported to image files: can colorizer be used?

Required Info

Camera Model
D435

Firmware Version
5.11.1

Operating System & Version
Windows 10 / Ubuntu 18.10

Kernel Version (Linux Only)
NA / 4.18.0-17-generic

Platform
PC

SDK Version
2.20.0

Language
Python

Segment
NA

Issue Description
Realsense bag files are very large. We are collecting a lot of data. I'd like to use lossy compression to store the frames extracted from a bag file, so that I can delete the original bags. For the color stream, JPEG conversion substantially reduces the required storage, but this cannot be applied to the 16 bit format of the depth files without shrinking the dynamic range by 2 orders of magnitude. TIFF compression does not seem to shrink the file size substantially, but the conversion tools I'm using (e.g., Imagemagick) do not seem to support JPEG 2000. Dropping to 8 bits seems like it would make for a much more substantial loss of information, relative to the loss incurred by standard compression techniques. Am I wrong about this? If not, then are there any other options for storing 16 bit grayscale using lossy compression?
I'm wondering if the librealsense colorizer could be useful. Is it possible to convert the grayscale images, using the colorizer, when the frames have already been exported to image files (i.e., they are not wrapped in a frame object)? I'm wondering if this would make it possible to apply standard JPEG compression to 8x3 bit colorized images, without sacrificing dynamic range.
I don't know a great deal about image and video storage. Any tips will be appreciated.

This might not be an issue anymore. It requires further testing, but I've gotten some improved results by adjusting TIFF compression settings.

Hi @realsense01,
You can also try to compress rosbag file. There is one option in the setting of realsense-viewer.


Improved testing and time handling

This merge brings a few small bug fixes, improvements to the test base and automated handling of datetime.timedelta objects in the execution loop. In detail:

Handle and interpret datetime.timedelta objects in the execution loop
Use timedelta objects in example tests
Add baseline tests for grid creation and field sampling functionalities
Fix standalone field visualisation
Allow setting output_interval without specifying an output file

Apart from two minor issues (level of verbosivity in test.py and whether to import timedelta as delta or not), I think this is a very nice housekeeping PR! Cleans up some of the code!

Cool, addressed comments, rebased and green travis. This should be good to go.


Improving cell grouping UX

I think the cell grouping experience can definitely be improved here and think that it would benefit a lot of users! Not to make this into a spec, and understanding that this won't be "to do now", but just brainstorming here:

Ability to create more than 2 groups
Intuitive key bindings for grouping cells, running them, and then ungrouping them
Would be nice to have the grouping persist via git (e.g. through metadata)
Creating custom names for groups
Being able to easily copy and paste a group of cells to a different notebook
Being able to always start a notebook with a specific group (e.g. "I use similar imports all the time")

Given that we have a new way built into core for renning dependant cells, it was discussed that this feature would be deprecated in favour of whats in core.
Hence closing this issue.


Merge pull request #18 from Trisia/master

m

merge

merge

merge

merge

merge

merge

merge

merge

merge

merge

merge

merge

merge

merge

merge

mergemergemergemergemergemergemergemergemergemergemergemergemerge