angular build with prod causes runtime errors

monaco-editor version: 0.18.1
**Browser:chrome
**OS:win10
Steps or JS usage snippet reproducing the issue:
The problem in this  issues
It's not my issues but I have the same with it.
error-demourl
When I run with ng serve the monaco has no bug
But When I run with ng serve --prod(production mode),the monaco has bug on Console

I think the bug belong to angular-cli or monaco ,but in angular-cli ,past few months, it's not fixed.So I hope monaco can see it and give a solution.

This looks like it is an issue specific to @angular-devkit/build-optimizer --- https://github.com/angular/angular-cli/issues/14033 . We can leave this issue here open for a while, but there is no code change planned on our side to address this...

Angular appears to have no plans to change this behavior in the build optimizer. You need to turn off the build optimizer, or have the optimizer ignore Monaco.
However, if the error stems from not following best practices (e.g, object property access should be side-effect free), it could be worthwhile to explore changing Monaco so it plays nicely with uglifyjs.  It could benefit the maintainability of Monaco, since developers working on Monaco could unintentionally assume those best practices are valid here.
Both sides present valid arguments. Angular shouldn't impose assumptions on third-party library code, but also making changes to conform to those fairly reasonable assumptions could be overall beneficial to maintainers.

Ignoring the tree shaking optimization for the Monaco Editor sounds reasonable.
The Monaco Editor is shipped from "tree shaken" VS Code sources -- see here


steroids.app.absolutePath doesn't work in read-only on Android adhoc

Looking at this table: http://docs.appgyver.com/tooling/wrapper/android/android-feature-table/ it says that crosswalk adhoc build should work in read-only mode with steroids.app.absolutePath however when I try to use it to pass a file url to a custom plugin it doesn't work. It works fine on scanner build.

Cordova File plugin cannot access Android assets (the read-only location where application files are stored in non-Scanner builds). This is an issue with the Cordova File plugin, so I'll close this issue.
Thanks for pointing out that this should be clarified in that document, this is now documented.

Well, now it explains that files can be headed using http://localhost but it doesn't seems to work. Can't access the files using the http protocol from my cordova plugin, it doesn't work

That's true. http://localhost is intercepted by the WebView, there's no real local webserver. Therefore your Cordova plugin doesn't know anything about it.
For Cordova plugins you must access the filesystem directly:

steroids.app.absolutePath = location of your application (read-only on standalone)
steroids.app.absoluteUserFilesPath = writable location for user supplied content

Just for the record, you should add that steroids.app.absolutePath only works on iOS, because when using Android that location can't be accessed/readed from Cordova plugins (like the file plugin or low latency audio plugin). As we discussed in other thread (can't locate it right now, sorry), the workaround I'm using is to copy the files (using a secret feature on application.coffee) to absoluteUserFilesPath location and then use them from there.

Thanks for the clarification. :)


/live/track/get/send should report send index

Currently, /live/track/get/send does not return the index of the send queried, so responses can't be matched with queries.
(Reported by Urs)

Now implemented.


Alphabetize contents. Add Configuration section

@ziadoz If you didn't want the contents alphabetized let me know, I'll revert and you can pull in just the configuration section -- just thought it would be easier to scan when looking for a specific library.

+1 :+1:

Closing since this would break all existing PRs. Can you add your configuration section in a separate PR?

Yep will do. P.S. Hi jose!

What about second file, which would be generated from the current, primary, README.md? It should be relatively easy to collect all links and sort them. Then we can have multiple views on the same list.

I don't want to maintain two lists, so no. Alphabetizing can wait :)

Yeah, it can definitely wait.

https://github.com/ziadoz/awesome-php/pull/557 @josegonzalez


Last methods for nosql_execution_Store.go covered

What changed?
Happy paths for all methods in the file are done. Will be working on covering the missed lines from now onwards.

Why?
increase code coverage.

How did you test it?
unit tests

Potential risks

Release notes

Documentation Changes

Pull Request Test Coverage Report for Build 018eaabe-a476-460d-a048-ae1ca6bd850a
Details

0 of 0   changed or added relevant lines in 0 files are covered.
38 unchanged lines in 7 files lost coverage.
Overall coverage increased (+0.05%) to 66.308%

Files with Coverage Reduction
New Missed Lines
%

service/history/task/transfer_standby_task_executor.go
2
87.42%

common/task/parallel_task_processor.go
2
93.06%

service/history/queue/timer_queue_processor_base.go
3
77.82%

service/matching/taskListManager.go
3
80.2%

service/history/task/fetcher.go
6
85.05%

service/history/task/cross_cluster_task_processor.go
8
80.79%

service/history/execution/mutable_state_task_refresher.go
14
66.77%

Totals

Change from base Build 018eaaaf-ac42-4f44-9bc6-b90f44d2a673:
0.05%

Covered Lines:
97018

Relevant Lines:
146315

💛  - Coveralls


Add support to"interrupt" a response and send multiple images

Please add a button to cancel the answer from the assistant and make it possible to send several images, not only one per session. That would be a great addition to this awesome app.
I would also love a "stop" condition like oogaboogas custom stopping strings to prevent the model from continue to talking to itself.
Thanks!

Interesting
Will add to TODOs:

Support batching
Stop condition

For the stop condition, I'll have to look into how it's done. Could you point to where custom stopping strings are? I'm not too familiar with oogabooga's text ui repo.


failure at TASK [Ensure internet-monitoring environment is running.]

I encountered this following error during [Ensure internet-monitoring environment is running.] step.
The error:
fatal: [<IP_ADDRESS>]: FAILED! => {"actions": [{"id": "grafana", "status": "Pulling", "what": "service"}, {"id": "ping", "status": "Pulling", "what": "service"}, {"id": "speedtest", "status": "Pulling", "what": "service"}, {"id": "nodeexp", "status": "Pulling", "what": "service"}, {"id": "prometheus", "status": "Pulling", "what": "service"}], "changed": true, "cmd": "/usr/bin/docker compose --ansi never --progress plain --project-directory /home/pitemp/internet-monitoring/ up --detach --no-color --quiet-pull --no-build --", "containers": [], "images": [], "msg": "Error when processing ping: Error\nError when processing nodeexp: Error\nError when processing speedtest: Error\nError when processing prometheus: no matching manifest for linux/arm/v6 in the manifest list entries", "rc": 18, "stderr": " grafana Pulling \n ping Pulling \n speedtest Pulling \n nodeexp Pulling \n prometheus Pulling \n ping Error \n nodeexp Error \n speedtest Error \n prometheus Error \nno matching manifest for linux/arm/v6 in the manifest list entries\n", "stderr_lines": [" grafana Pulling ", " ping Pulling ", " speedtest Pulling ", " nodeexp Pulling ", " prometheus Pulling ", " ping Error ", " nodeexp Error ", " speedtest Error ", " prometheus Error ", "no matching manifest for linux/arm/v6 in the manifest list entries"], "stdout": "", "stdout_lines": []}
PLAY RECAP *********************************************************************************************************************************
<IP_ADDRESS>                  : ok=11   changed=1    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0

It looks like you may be running this on either an older Pi (armv6) or a very old OS release that's 32-bit only. Unfortunately this project requires something newer.

Oh yeah, its an armv6. Thanks for the quick response.


elm-server currently doesn't install on arch due to attoparsec

$ sudo aura -A elm-server
aura >>= Determining dependencies...
aura >>= Dependency checking failed for these reasons:
The dependency `haskell-attoparsec` demands version `<0.11`but the most recent version is `<IP_ADDRESS>-1`.

elm-server no longer exists


Don't propagate click when clicking on header filter

Without this clicking on the header filter icon would propagate to the header and thus cause it to sort the column as well.
Also added typing to the onCommand event

Pull Request Test Coverage Report for Build 238855287

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.02%) to 39.067%

Totals

Change from base Build 238736697:
0.02%

Covered Lines:
18960

Relevant Lines:
44370

💛  - Coveralls


v0: Fix hex cache generation

Was a simple order of operation bug. I ended up just removing uf2 caching, since it will never be needed in v0.

Can we also fix master?

Master was already fixed, parentheses were added around the ternary operator


compile bromides

one example is this https://liquidzulu.github.io/brain/note/the-error-in-regarding-the-man-made-as-immutable/

wrong repo


Add support for Porterbuddy

Closing this since even though this fixed the symptom of the issue, it does not fix the underlying issue. It has been solved in https://github.com/Dintero/Dintero.Checkout.WooCommerce.V2/pull/26 however


seclists: 2023.3 -> 2023.4

Diff: https://github.com/danielmiessler/SecLists/compare/2023.3...2023.4
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Priorities
Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 271414 run on x86_64-linux 1

3 packages built:

rockyou
seclists
wordlists

@fabaff imho it is better that the folder directory should replace wordlists by payloads because "wordlists" belong to small category of the resources offered by seclists. In general payloads could cover seclists resources in a broader manner.


Update dependencies prior to Wegue v2

Updated any remaining non-dev dependencies, on top of all previous PRs in the V2 series.
Replaces #295.
I can't run the Karma tests easily on my system & don't have time for a workaround right now.
It would be worth running npm run test after all the updates.

I just checked out the changeset of this PR and created a new Wegue app with npm run init:app. Unfortunately this layout seems to be a bit off. There seems to be an Element (white) on top of the map, which shifts down the map itself:

Also for the sidebar layout:

Thanks @chrismayer for the feedback - a bit silly of me - perhaps these are changes introduced by the Vuetify upgrade. I will test further next week and update 👍

Found the issue in custom css for AppLogo.vue:
  .v-avatar.v-avatar--tile.wgu-app-logo {
    position: absolute;
    z-index: 2;
  }

Should be replace with:
  .v-avatar.wgu-app-logo {
    position: absolute;
    z-index: 2;
  }

And the prop rounded="0" passed to v-avatar to have a square tile.
I will update all PRs.

I also updated some other minor deprecations from Vuetify 2.2.x --> 2.3.x in commit https://github.com/wegue-oss/wegue/pull/333/commits/42ead057b461b77b84e6875800ee6d1ef7fd3ea0
There were no relevant upgrades from 2.3.x --> 2.4.x and no upgrades at all from 2.4.x --> 2.6.x (LTS).
I cherry-picked the commit onto all subsequent stacked PRs.

It should also be noted that all edits in https://github.com/wegue-oss/wegue/commit/42ead057b461b77b84e6875800ee6d1ef7fd3ea0 were to the components directory, so no manual updates during the V1 --> V2 upgrade should be required.
There is however one minor deprecation to the app-starter, found in commit https://github.com/wegue-oss/wegue/pull/335/commits/a12cd0d4dcf7d79c7ac8e67a3152768aa1b15519:
The update is a very simple on for WguAppTemplate.vue.
<v-content app>
...
<v-content />

to
<v-main app>
...
<v-main />

This should probably be noted for any upgrade guide we may include for users migrating apps V1 -> V2.

I just pulled your updates and changed <v-content> to <v-main> in WguAppTemplate.vue. Now everything looks good to me:

Very nice! Thanks again.

@spwoodcock Would you mind resolving the conflict, so this PR gets mergable again? In my point of view the base PR #320 also has the conflict and should be an whitespace change only.

Hi @spwoodcock, thanks for your ongoing efforts!
It looks like the merge in 6d9c8579a6b7cdee5be773e61024a2af143274b4 has gone the wrong way, as a lot of those methods in AttributeTable were removed in the master (see 3d7116395049b1f03a640824b0fb0a9511e8107e)
Sorry for the brief comment, these are currently busy days for me but I hope I can test it more throughoutly in the upcoming weeks.
Cheers, Felix

Hey @fschmenger sorry about the notifications - I was testing to see what would fix the conflicts, as I didn't want to rebase the entire branch again.
I ended up just checking out the master for the files, as they are only whitespace related.
Please check everything is all good 👍

Cherry-picked 23c9451d425646de04879c00e66850f91869d13e to make this mergable.

Merged master to only show specific commits - ready to test and merge.

Thanks for merging in the current master @spwoodcock.
Unfortunately the unit tests in the CI fail. Not sure if this was the case before the merge or if they were OK before. They also fail for #333 and #334 and it is locally reproducible when you execute npm run test with the code of this PR.
I did a fresh checkout of the code of this PR and created the Wegue example apps. There are some linting problems in the AttributeTableWin component, which can easily be fixed with this commit.
After that everything worked as expected at least as far as I could see in some visual tests by clicking through the modules in the example apps.

Thanks @chrismayer!
Apologies for not being very thorough and checking the tests after merge - I'm also pretty pushed for time currently.
We planned to use Wegue as our geospatial module in EnviDat/WSL in CH, but I am leaving and priorities have shifted for now.
I'll make contributions to fix these PRs and add cloud native format support (COG, flatgeobuf, geoparquet, COPC) in the near future.

@spwoodcock I had a look at the test suite and why it is failing. This commit should fix it. Are you willing to pull it into this PR or should I open a new PR based on this PR with the additional fix?

Much appreciated @chrismayer! I added the commit =)

So I did some more tests by upgrading a current application to this branch and it works very well. No problems so far, so I finally will merge this :tada:
If any problems will occur afterwards we will handle them in follow-up PRs as always.
I also will come up with an update of the migration guide and I am going to update the v2 roadmap (#327).
Thanks again to all, who were involved. This is another great step forward :+1:


ForgifiedFabricAPI doesn't work in newer Forge version

Here's the crash log if necessary
crash-2023-10-06_16.22.56-fml.txt

This is intended, see #15 for more information


Duplicate claim links on resources page

"Signing a claim" and "Claim Signing" sound like the same thing and both routes are 404'd.

https://github.com/lbryio/lbry.tech/commit/7cf147bab180a00fc9c8af70fc13f03dcd4e5a9e


hasOne documentation confusion

From the docs for hasOne:

Target Model
Constructor of Model targeted by join.
[foreignKey] string
ForeignKey in the Target model. By default, the foreignKey is assumed to be the singular form of this model's tableName, followed by _id / _{{idAttribute}}.
[foreignKeyTarget] string
Column in the Target model's table which foreignKey references, if other than Target model's id / idAttribute.

The docs here say that foreignKey exists in the target model, and references a column foreignKeyTarget, which also exists in the target model. I think this is backwards: doesn't foreignKeyTarget reference a column which exists in this model, not the target model?

No, the docs are correct. Maybe an image with some example tables would help to visualize where the foreign keys are.
hasOne is like hasMany but with a single target instead of multiple, so it is also the opposite of belongsTo. What this means is that the target model will have a key like this_model_id and this is what you use the foreignKey parameter for.
The last parameter (foreignKeyTarget) is only needed if the foreign key in the target model references a key in this model other than the default id (or the value of idAttribute).


No links

Hi, I have installed your addon as well as installed almost all addons on Stremio. However, when I search for any show, there are no links displayed.
For eg, I was searching "The Kapil Sharma Show" which is available on your website but on Stremio, I do not get results from your addon or other addons.
Screenshot here: https://imgur.com/a/jjMYPlL
Two links shown in the screenshot above, they do not play. Why am I not getting more links for the show, either from your addon or from other addons? If I search for English movies, then I do get lots of links.
Can you please suggest any solutions? Thanks.

Hi @TheKidRock, This is add-on only scraping data from tamilblaster site. Currently, it will only scrap the newest movie data. If you search a movie and if there is stream data, that means that movie not been scraped.

Searching movies and series functionality is added #20. This will only search the movies which are already scrapped in the db.
Searching directly from tamilblasters is need to bypass Cloudflare validation.


Some features dont work in the florisboard organisation

Description
Hello,
when looking at repositories in the florisboard organisation some of the features aren't working.
I've created a personal access token with the standard settings except the delete-repo permission as i'm not using that feature.
For example, the release tab at the top is not displayed, or the repo-header-info right next to the repo name.

Those features are working in other organisations I'm a member in. https://github.com/goxlr-on-linux/goxlr-utility

How to replicate the issue + URL
Just go to the repository and the features aren't there :man_shrugging:
https://github.com/florisboard/florisboard
Extension version
24.9.21
Browser(s) used
Firefox, Iridium (Chrome)
Token ⚠

[X] I set a token in the options page. I understand that some features require a token to work.

...

I see features working:

Check for errors in the console. This is likely to be a token issue. Since you're part of the org, maybe the token is disabled for it internally.

Weird.
I've tested it with 2 different browsers on 2 different PCs with and without incognito mode on. Could it be that my personal access token is broken for this organization?

Likely a restriction within the organization: https://docs.github.com/en/organizations/managing-programmatic-access-to-your-organization/setting-a-personal-access-token-policy-for-your-organization#enforcing-an-approval-policy-for-fine-grained-personal-access-tokens
The restriction doesn't apply to me because the repo is public and my access is read-only; my guess is that since your account has write access, the organization is able to block it entirely.

Hey there,
admin of the florisboard org here. We were able to fix the issue, it was indeed an issue with the token settings of the org. In particular: Going to Organization settings > Personal Access tokens > Settings > Tokens (Classic) you can restrict the usage of classic tokens:

Then Refined GitHub does not work on repos for that org. Allowing it fixes the issue and Refined works without issues.


DO NOT MERGE: fix memory overallocs in parts of PromQL engine

DRAFT, DO NOT MERGE
Summary
Fixes memory overallocations in parts of PromQL engine, specifically for case *VectorSelector: in evaluator.eval(expr Expr) in promql/engine.go.
This reduced memory in use from 6.03GB to 228.18MB, which is broadly 26x/98% improvement for my sample data and query, if we trust pprof and Go's memory stats :)
NOTE: this somewhat fixes the https://github.com/prometheus/prometheus/pull/4513 which probably incorrectly accounts for the samples used in the query rather than allocated samples. If so, perhaps the fix should be within that space.
NOTE2: the savings depend on the query range, downsampling interval (step), and number of "holes" in data. I think.
NOTE3: the current PR includes instrumentation code, please ignore.
Root cause and patch
The gist of the issue is best shown in code based on https://github.com/prometheus/prometheus/blob/master/promql/engine.go#L1129
case *VectorSelector:
    checkForSeriesSetExpansion(ev.ctx, e)
    mat := make(Matrix, 0, len(e.series))
    it := storage.NewBuffer(durationMilliseconds(LookbackDelta))
    for i, s := range e.series {
        it.Reset(s.Iterator())
        ss := Series{
            Metric: e.series[i].Labels(),

            // ==> This pre-allocates ss.Points with capacity equal to `numSteps`.
            Points: getPointSlice(numSteps),
        }

        for ts := ev.startTimestamp; ts <= ev.endTimestamp; ts += ev.interval {
            _, v, ok := ev.vectorSelectorSingle(it, e, ts)
            if ok {
                if ev.currentSamples < ev.maxSamples {
                    // ==> This is where we actually add stuff to ss.Points
                    ss.Points = append(ss.Points, Point{V: v, T: ts})
                    ev.currentSamples++
                } else {
                    ev.error(ErrTooManySamples(env))
                }
            }
        }

        // ==> At this point the cap(ss.Points) may be very much larger than len(ss.Points).
        // ==> Since we keep the fully allocated array for the duration of the query eval,
        // ==> and since the number of series can be quite large, this can lead to excessive
        // ==> memory usage over what we need.
        if len(ss.Points) > 0 {
            mat = append(mat, ss)
        }

        // ==> The patch is to shrink ss.Points by reallocating to a smaller array.
        // ==> Here I do so if we overallocated more than 20% (arbitrary number).
        {
            pointsOverallocLimit := 1.2
            pointsOverallocRatio := float64(cap(ss.Points)) / float64(len(ss.Points))
            if pointsOverallocRatio > pointsOverallocLimit {
                pointsCopy := make([]Point, len(ss.Points))
                copy(ss.Points, pointsCopy)
                ss.Points = pointsCopy
            }
        }

    }
    return mat

Impact of overallocation and the patch
Running 1-month ranged query over synthetic 1-week data:
Data:

Thanos-bench generator profile: realistic-k8s-1w-small
mint: 2019-10-11T03:00:15+01:00
maxt: 2019-10-18T03:00:15+01:00

Query:

query: count({__name__=~".+"}) by (__name__)
mint: 2019-10-01T00:00:0+01:00
maxt: 2019-11-01T00:00:0+01:00
resolution: step=4838

// BEFORE PATCH:

PromQL: PROMQL_ALLOC_PATCH_OFF: true                                <=| Patch disabled
PromQL: Expression: {__name__=~".+"}
PromQL: VectorSelector: Series Count: 845000
PromQL: VectorSelector: pointsAllocSize: 468975000; Size: 7503.60M  <=| This is what we allocated
PromQL: VectorSelector: pointsNeededSize: 670000; Size: 10.72M      <=| This is what's actually needed
PromQL: VectorSelector: pointsOverAllocRatio: 699.96x               <=| This is how much we over-allocated
PromQL: VectorSelector: Current Samples: 670000

// memory impact, including garbage
MEM STATS DIFF:   	promql-eval 	promql-eval - AFTER 	-> Delta
    HeapAlloc  : 	804.21M 	7.26G 			-> 6.46G
    HeapSys    : 	1.01G 		8.12G 			-> 7.11G
    HeapObjects: 	12.60M 		14.53M 			-> 1.94M

// AFTER PATCH:
// memory impact, including garbage
MEM STATS DIFF:   	promql-eval 	promql-eval - AFTER 	-> Delta
    HeapAlloc  : 	985.59M 	925.55M 		-> -60.05M
    HeapSys    : 	1.00G 		2.15G 			-> 1.14G
    HeapObjects: 	14.69M 		14.53M 			-> -151.87K

Pprof figures:
// BEFORE PATCH:
Showing nodes accounting for 6.03GB, 99.71% of 6.05GB total

// AFTER PATCH:
Showing nodes accounting for 228.18MB, 100% of 228.18MB total

// DELATA:
-5.8GB --> -96%

Pprof charts
Original bytes in use:

Patch bytes in use:

Diff bytes in use:

Pprof files
eval-end-orig.pb.gz
eval-end-patch.pb.gz
eval-start-orig.pb.gz
eval-start-patch.pb.gz

Neat!

What does it look like with a more typical query? This change will increase allocs, and cpu usage. The existing PromQL benchmarks would also be useful.

What does it look like with a more typical query?

Overallocs look broadly similar. I think the interesting thing here is that surprising amounts of memory can be over-allocated depending on the data and query controlled by the user: range, step. I managed to contrust a query over the same data which wanted to allocated 65GB (!) when the actual need was like 256M.
Here are the stats for k8s_app_metric0 with everything else being exactly same including data and other query parameters:
Query:

query: k8s_app_metric0
mint: 2019-10-01T00:00:0+01:00
maxt: 2019-11-01T00:00:0+01:00
resolution: step=4838

// BEFORE PATCH
PromQL: PROMQL_ALLOC_PATCH_OFF: true
PromQL: Expression: k8s_app_metric0
PromQL: VectorSelector: Series Count: 16900
PromQL: VectorSelector: pointsAllocSize: 9379500; Size: 150.07M   <=| This is what we allocated
PromQL: VectorSelector: pointsNeededSize: 13400; Size: 0.21M      <=| This is what's actually needed
PromQL: VectorSelector: pointsOverAllocRatio: 699.96x             <=| This is how much we over-allocated
PromQL: VectorSelector: Current Samples: 13400

// memory impact, including garbage
MEM STATS DIFF:   	promql-eval 	promql-eval - AFTER 	-> Delta
    HeapAlloc  : 	25.70M 		151.65M 		-> 125.96M
    HeapSys    : 	65.90M 		200.11M 		-> 134.22M
    HeapObjects: 	323.45K 	294.41K 		-> -29.05K

// AFTER PATCH:
// memory impact, including garbage
MEM STATS DIFF:   	promql-eval 	promql-eval - AFTER 	-> Delta
    HeapAlloc  : 	25.44M 		25.00M 			-> -440.93K
    HeapSys    : 	65.90M 		65.90M 			-> 0
    HeapObjects: 	319.56K 	294.47K 		-> -25.09K

Pprof figures:
// BEFORE PATCH:
Showing nodes accounting for 127.61MB, 100% of 127.61MB total

AFTER PATCH:
Showing nodes accounting for 5620.68kB, 100% of 5620.68kB total

Note that overallo ratio is exactly same here at 699.96x, even if absolute figures are different.

This change will increase allocs, and cpu usage.

Here is my current thinking on this:

We allocate only 1 extra array per loop cycle, but that object can be several orders of magnitude smaller to replace large ones. Overall the I think this should have a positive impact reducing GC pressure etc and that one extra array alloc per loop is a drop in a ocean in the grand scheme of things.

Also we don't do anything if we are overallocated less than 20% (arbitrary figure at the moment). So for some scenarios this patch can have zero impact.

CPU cycles. Yes true but should we worry about it? memcpy of contigious areas of memory is fast. To be fair, I didn't do any benchmarks so there is a bit of hand-waving there :)  If people insist, I can do some perf benchmarks around this.

The existing PromQL benchmarks would also be useful.

More than happy to run these if people could point me in the right direction and guide a bit. I'm not at all familiar with this side of the project.

More than happy to run these if people could point me in the right direction and guide a bit.

Run all benchmarks in the promql directory.

So... I removed all instrumentation and run the benchmarks.
Unless I did something completely dumb, I'm inclined to interpret results as "near-zero performance impact with marginal differenced bordered on error" but I may be mistaken.

Original: https://github.com/prometheus/prometheus/commit/7293c859abce40ae8c526c0f0134bcb894f282c7
Patch: this PR https://github.com/prometheus/prometheus/pull/6372/commits/f9e2ab03bba09b6ca6c1711b28d349b5ddf026d0

go test -run=NONE -bench=. ./promql/... | tee old.txt
go test -run=NONE -bench=. ./promql/... | tee new.txt
benchcmp old.txt new.txt  > diff.txt

Did I run the correct thing?

Highlights:

ns/op

benchmark                                                                                old ns/op      new ns/op      delta
# deltas border or error
BenchmarkRangeQuery/expr=a_hundred,steps=1000-16                                                                       6434642        6480290        +0.71%
BenchmarkRangeQuery/expr=changes(a_hundred[1d]),steps=1000-16<PHONE_NUMBER><PHONE_NUMBER>     -0.52%

# some take longer
BenchmarkRangeQuery/expr=a_one,steps=100-16                                                                            25623          27219          +6.23%
BenchmarkRangeQuery/expr=a_one,steps=1000-16                                                                           78269          82208          +5.03%

# but some are equally faster
BenchmarkRangeQuery/expr=a_one_-_b_one,steps=100-16                                                                    112892         105559         -6.50%
BenchmarkRangeQuery/expr=a_one_-_b_one,steps=1000-16                                                                   785382         728117         -7.29%
BenchmarkRangeQuery/expr=a_ten_-_b_ten,steps=1000-16                                                                   7162044        6463838        -9.75%
BenchmarkRangeQuery/expr=a_ten_-_b_ten,steps=10000-16                                                                  77495123       63063953       -18.62%

allocs/op

# *most* zero deltas
BenchmarkRangeQuery/expr=a_one,steps=1-16                                                                              107            107            +0.00%

# only three show small icrease
BenchmarkRangeQuery/expr=a_hundred_-_b_hundred,steps=1000-16                                                           17895          18575          +3.80%
BenchmarkRangeQuery/expr=a_ten_-_b_ten,steps=10000-16                                                                  53296          54529          +2.31%
BenchmarkRangeQuery/expr=a_hundred_-_b_hundred,steps=10000-16                                                          155247         160374         +3.30%

bytes/op

# *most* are near-zero deltas
BenchmarkRangeQuery/expr=a_one,steps=1000-16                                                                           8328          8323          -0.06%
BenchmarkRangeQuery/expr=a_ten,steps=1-16                                                                              13307         13306         -0.01%
BenchmarkRangeQuery/expr=a_ten,steps=10-16                                                                             13306         13306         +0.00%
BenchmarkRangeQuery/expr=a_ten,steps=100-16                                                                            14026         14026         +0.00%
BenchmarkRangeQuery/expr=a_ten,steps=1000-16                                                                           21066         21075         +0.04%
BenchmarkRangeQuery/expr=a_hundred,steps=1-16                                                                          80003         80206         +0.25%

# two strange ones showing some increase
BenchmarkRangeQuery/expr=a_ten_-_b_ten,steps=10000-16                                                                  3201351       3550912       +10.92%
BenchmarkRangeQuery/expr=a_hundred_-_b_hundred,steps=1000-16                                                           2811254       3035373       +7.97%

# equally some showing decrease
BenchmarkRangeQuery/expr=changes(a_one[1d]),steps=1000-16                                                              578778        547794        -5.35%
BenchmarkRangeQuery/expr=a_ten_-_b_ten,steps=1000-16                                                                   359869        348921        -3.04%

Attached are the benchmark files. Of course it would be nice if someone could verify my results independently.
old.txt
new.txt
diff.txt

Your PR only helps with sparse series, and the existing benchmarks use data which is not sparse at all. We'll likely need some new benchmarks here.

OK, I made a Big Dumb Thing with copy by confusing source and destination, which I have now fixed. Thank you go test for picking this up :)
The memory benchmarks still stand though! (basically I was replacing the samples with all zeros, doh!).
I have re-run the benchmarks with this fix though, and results are interesting, and also a bit confusing. Maybe because it's late in the evening or I did some Dumb Thing again.
Basically, benchmarks show that eval is now faster. A lot of tests show 7-20% faster. Why? is there a bug somewhere either in the PR or the way I ran benchmarks? Some background activities on my laptop skewed this? Is it some GC-related thing or GO's malloc thing?
For now, see the updated benchmarks.
old.txt
new.txt
diff.txt

Your PR only helps with sparse series, and the existing benchmarks use data which is not sparse at all. We'll likely need some new benchmarks here.

Yes I guess so. It would be extremely helpful if someone either created these sparse benchmarks or helped me do so because I really have no idea what I’m doing when it comes to PromQL :)
It would be nice if benchmarks were already in place somewhere so I could use them.

A lot of tests show 7-20% faster. Why?

It's possible some of our existing benchmarks are not actually sparse, which should be fixed if that's the case.

It would also be nice if someone did a really careful review of this patch for correctness and as a bonus did both men and perf validation. I was a bit rocked by that “copy” fiasco and don’t trust myself at the moment!

One thing to watch on laptops as well is power saving stuff which can vary the frequencies of the cores, I'd suggest turning that off. Though that usually shouldn't affect microbenchmarks like these.

Also, if you broke something the unittests should catch it.

Also, if you broke something the unittests should catch it.

Yes they did :) Tests are green now though. Quick approve and merge ;)
On a serious note, if all is green and if people are happy with this fix and benchmarks as they stand, will this be a LGTM and a merge? If not, what would I/we need to do to make this happen?

There'll be more benchmarks and tweaks required anyway, but first we should determine if it's actually a win.

Ok let’s start with the win.
“6GB -> 0.2GB” == win
Right?

That's only one case, it might make other cases slower or use more memory.

That's only one case, it might make other cases slower or use more memory.

The perf benchmarks suggest it’s all good? Of course this assertion is only as good as the benchmarks but in the absence of any other measurement, how do we make a judgement if this PR is ok? What do we need to do to assure ourselves it is indeed the case?

I think what I’m asking for, to be a bit blunt perhaps, is this:

what are the concrete steps I need to take for the merge to happen?

More benchmarks? If so what kind and can anyone help?
Explaining and scrutinising the method and data I used? Happy to do that if that’s what blocks this PR.
Over alloc ratio or 1.2 is questionable? Which ratio is good then? 1.5? 1.8? 2.0? Let’s discuss.
Real life test ran by volunteers? That would be great?
Anything else? A list of concrete steps would help.
I think I provided all the pprof files and perf benchmarks (ran on a laptop of course) to show there is a benefit for some cases without perf degradation as far as the tests show.
So what do I need to do to move this forward? Concrete steps?

Nice!

Real life test ran by volunteers? That would be great?

That would do as well, but probably some docker image would be nice to have for this, we can ask on prometheus-dev mailing list if needed as well.

So what do I need to do to move this forward? Concrete steps?

I think we need to justify this with microbenchmarks, so we would probably need to add a case to benchmarks for sparse series, so series that are living, for example, 1/(30*24) of the requested time range like in your case, which is sometimes realistic. We definitely over-allocate in those cases as we allocate assuming all samples for each step for all series are present and I totally see some fix like this to make sense.
I cannot still understand though how shrinking helps here though as we already allocated this memory in return make([]Point, 0, sz) as I understand and once we put it properly we just retrieve this from the sync pool or allocate the too large slice again. (: Do we have idea why shrinking helps here?

How about running prombench for few hours (6-10) on this?

what are the concrete steps I need to take for the merge to happen?

We don't exactly have an ISO-9000 certified process, so we'll likely go back and forth a bit as we ensure the change is good and ways this may cause issues occur to us.

Over alloc ratio or 1.2 is questionable? Which ratio is good then? 1.5? 1.8? 2.0? Let’s discuss.

I'd go for 2 by default barring data indicating otherwise, to match how Go slices generally work.

How about running prombench for few hours (6-10) on this?

Microbenchmarks will be the most relevant here, and we'll want relevant ones added anyway to judge any future changes to this code.

Do we have idea why shrinking helps here?

We're also not considering the interaction with sync pool, nor using it for the new slices, so that means more garbage, thus more memory pressure and more CPU to collect it.

I cannot still understand though how shrinking helps here though as we already allocated this memory in return make([]Point, 0, sz) as I understand and once we put it properly we just retrieve this from the sync pool or allocate the too large slice again. (: Do we have idea why shrinking helps here?

A very good question @bwplotka!  Also thanks @brian-brazil, helpful comments (that ISO comment made my chuckle) and yes, sync pool looks to be the key.
I dug around trying to understand what's going on and I found some interesting things.
TL;DR: The sync.Pool makes things fun and intersting, I've no idea what I'm measuring anymore, but the patch still seems to help!

Points are not always put back into pool. It depends on the query. Is it a bug or by design?
The way I did this is here: https://github.com/ppanyukov/prometheus/compare/master...ppanyukov:feature/promql-pool-debug
expr               : k8s_app_metric0
poolGetCount       : 16900
poolGetSuccessCount: 0
poolPutCount       : 0          <= not putting back

expr               : k8s_app_metric0
poolGetCount       : 33800
poolGetSuccessCount: 0
poolPutCount       : 0          <= not putting back

expr               : k8s_app_metric0
poolGetCount       : 50700
poolGetSuccessCount: 0
poolPutCount       : 0          <= not putting back

expr               : count by(__name__) ({__name__=~".+"})
poolGetCount       : 895750
poolGetSuccessCount: 0
poolPutCount       : 670000     <= this does put back

expr               : k8s_app_metric0
poolGetCount       : 912650
poolGetSuccessCount: 16900
poolPutCount       : 670000

expr               : k8s_app_metric0
poolGetCount       : 929550
poolGetSuccessCount: 33800
poolPutCount       : 670000

expr               : k8s_app_metric0
poolGetCount       : 946450
poolGetSuccessCount: 50700
poolPutCount       : 670000

Calling `runtime.GC()` wipes `sync.Pool`
In instrumented version I was writing pprof heap dumps. This, for some reason, called runtime.GC(). Which, as it turns out, seems to wipe whatever we have in sync.Pool.
This certainly would have an impact on any measurements. I have removed the call to runtime.GC() and things became much more interesting and fun after that!

Use of `sync.Pool` seems to make pprof heap profiling near-useless?
It looks like whatever is put into sync.Pool does not show up as "bytes in use" in pprof heap profiles. Unless I am doing it wrong? I wouldn't be surprised TBH. If anyone can shed some light on this that'd be great.

Use of `sync.Pool` shows different figures depending on how many times you run queries
With sync.Pool the more we run things, the more we reuse byte slices. Correct? And since things in sync.Pool are not shown as "bytes in use" in heap profile (see above), we get different profiles depending how many times we run queries. Combine this with the fact that some queries do not put anything back into the pool. Now we get different figures depending which queries we run in which order and how many times.
Questions: what am I measuring here?? And how to do it right?

The `HeapSys` stat seems to be most useful metric with `sync.Pool`?
I found that the only reliable figure here is HeapSys.
What I also found is that it "stabilises" at some point when I run queries, that's the effect of sync.Pool I presume.

But! The patch still seems to help!
If we assume that HeapSys is reliable, I found this patch still helps with memory utilisation.
First, the sync.Pool is still working (I observe things being put and successfully taken from it).
Second, the HeapSys is way lower with the patch. On my machine it was 13.8GB without patch vs 2.2GB with the patch.
So it does seem to help. I think the way this works is:

we eiher alloc new array or get it from pool;
we shrink it if required;
the shrunk array is put back in the pool (sometimes);

The question remains: are we doing too many allocs and put pressure on GC with this patch?

This is about it for now. I'm very confused at this point and don't know what/if we want to do next :)
Probably would be nice to confirm why things are not being put back into the pool first? And fix that if it's a bug?

I'd go for 2 by default barring data indicating otherwise, to match how Go slices generally work.

Yes, seems reasonable, I considered exactly same number too. How about maybe even go to 2.2? Since we are using sync.Pool we perhaps don't want to be overaggressive/precise in the trimming?
More general question: what kind of tests/benchmarks can we have to help us confirm 2.0 or 2.2 are good numbers?  If tha's possible at all.

Points are not always put back into pool. It depends on the query. Is it a bug or by design?

Hmm, they're meant to. Are you closing the query properly when it's complete? I'd have to see the specific cases, but it's probably an oversight. Let's not worry about it right now if we don't have to.

Calling runtime.GC() wipes sync.Pool

That depends on your Go version, it's only half wiped now I believe.

And since things in sync.Pool are not shown as "bytes in use" in heap profile (see above)

They are in use on the heap, so should be showing.

Use of sync.Pool seems to make pprof heap profiling near-useless?

It does make performance profiling fun alright.

Questions: what am I measuring here?? And how to do it right?

That's the big question here. In an ideal world we wouldn't need to have the pool for performance, but that's not how Go works currently.

More general question: what kind of tests/benchmarks can we have to help us confirm 2.0 or 2.2 are good numbers? If tha's possible at all.

To be honest I wouldn't worry about getting it perfect in this PR. Once we've figured out what benchmarks etc. we need, finding the optimal value can be done later.

It occurs to me that what we probably want benchmark wise here is to manually set the contents of the syncpool before each iteration of the tests for this. As against the existing tests (which we can re-use the test cases for) where we want the sync pool to mature naturally to simulate the best case with regular refreshes/evaluations.

Yes excellent idea. If we could come up with a good set of tests and benchmarks for this, it would be generally a good thing regardless of the fate of this PR :)
Brian do you think it’s worth to sync up offline on slack or something at some point for a more fluid discussion and to avoid noise here and then post the summary here?

We're on #prometheus-dev on Freenode if you want to chat more realtime, but I think my above idea will be sufficient benchmark wise.

Quick update on this. I'm a bit sidetracked with other things at the mo but this PR is not forgotten. We are going to run some tests of this patch on the real data sets using Thanos and we will see what it shows.
The biggest blocker for me right now is "what microbenchmarks exactly do I need to create?", as I'm not knowledgable enough of Prometheus and existing benchmarks etc so it will take some learning.  If someone could take a stab at these in a separate PR, that would be super awesome.

We did some tests with this patch in the Thanos querier, using real data.  We just hit the querier with various heavy queries, with a couple of minutes between each query.
We first tried sum(up) over an 8 week period.  We only have data for about 4 of those weeks in this environment so it's partial results, but I think this kind of query is somewhat normal anyway.
With the patch off:
level=debug name=<querier_name> ts=2019-12-06T11:16:07.105677359Z caller=proxy.go:376 queryTotalSize=420.09MB queryLocalSize=160.02MB
level=debug name=<querier_name> ts=2019-12-06T11:16:07.10942091Z caller=proxy.go:376 queryTotalSize=440.09MB queryLocalSize=220.06MB
level=debug name=<querier_name> ts=2019-12-06T11:16:07.109481432Z caller=proxy.go:191 queryTotalSize=440.09MB
...
PromQL: PROMQL_ALLOC_PATCH_ON: false
PromQL: Expression: up
PromQL: VectorSelector: Series Count:    286829
PromQL: VectorSelector: Current Samples: 20112
PromQL: VectorSelector: pointsAllocInitial:   71994079; Size: 1151.91M
PromQL: VectorSelector: pointsAllocNeeded:    20112; Size: 0.32M
PromQL: VectorSelector: pointsAllocFinal:     71994079; Size: 1151.91M
PromQL: VectorSelector: pointsOverallocLimit:        2.00
PromQL: VectorSelector: pointsOverAllocRatioInitial: 3579.66x
PromQL: VectorSelector: pointsOverAllocRatioFinal:   3579.66x
PromQL: VectorSelector: pointsOverAllocDelta:        0.00x
expr               : sum(up)
poolGetCount       : 1080832
poolGetSuccessCount: 15
poolPutCount       : 106400
MEM STATS DIFF:         promql-eval     promql-eval - AFTER     -> Delta
    HeapAlloc  :        885.60M         1.76G                   -> 874.95M
    HeapSys    :        1.81G           1.88G                   -> 66.72M
    HeapObjects:        9.51M           10.37M                  -> 855.72K

And with the patch on:
level=debug name=<querier_name> ts=2019-12-06T11:23:44.44175381Z caller=proxy.go:376 queryTotalSize=420.05MB queryLocalSize=160.02MB
level=debug name=<querier_name> ts=2019-12-06T11:23:44.441867962Z caller=proxy.go:376 queryTotalSize=440.05MB queryLocalSize=220.02MB
level=debug name=<querier_name> ts=2019-12-06T11:23:44.441949684Z caller=proxy.go:191 queryTotalSize=440.05MB
...
PromQL: PROMQL_ALLOC_PATCH_ON: true
PromQL: Expression: up
PromQL: VectorSelector: Series Count:    286829
PromQL: VectorSelector: Current Samples: 20112
PromQL: VectorSelector: pointsAllocInitial:   71994079; Size: 1151.91M
PromQL: VectorSelector: pointsAllocNeeded:    20112; Size: 0.32M
PromQL: VectorSelector: pointsAllocFinal:     20112; Size: 0.32M
PromQL: VectorSelector: pointsOverallocLimit:        2.00
PromQL: VectorSelector: pointsOverAllocRatioInitial: 3579.66x
PromQL: VectorSelector: pointsOverAllocRatioFinal:   1.00x
PromQL: VectorSelector: pointsOverAllocDelta:        -3578.66x
expr               : sum(up)
poolGetCount       : 286830
poolGetSuccessCount: 0
poolPutCount       : 15055
MEM STATS DIFF:         promql-eval     promql-eval - AFTER     -> Delta
    HeapAlloc  :        1.24G           1.40G                   -> 164.35M
    HeapSys    :        1.27G           1.68G                   -> 402.65M
    HeapObjects:        11.03M          9.55M                   -> -1.49M

We also tried the pathological query count({__name__=~".+"}) by (__name__) over a 6 day period, again covering a partial data time
Patch off:
level=debug name=<querier_name> ts=2019-12-06T11:55:39.537519487Z caller=proxy.go:376 queryTotalSize=3900.84MB queryLocalSize=3900.84MB
level=debug name=<querier_name> ts=2019-12-06T11:55:39.53760166Z caller=proxy.go:191 queryTotalSize=3900.84MB
….
PromQL: PROMQL_ALLOC_PATCH_ON: false
PromQL: Expression: {__name__=~".+"}
PromQL: VectorSelector: Series Count:    2696665
PromQL: VectorSelector: Current Samples: 8744867
PromQL: VectorSelector: pointsAllocInitial:   676859462; Size: 10829.75M
PromQL: VectorSelector: pointsAllocNeeded:    8744867; Size: 139.92M
PromQL: VectorSelector: pointsAllocFinal:     676859462; Size: 10829.75M
PromQL: VectorSelector: pointsOverallocLimit:        2.00
PromQL: VectorSelector: pointsOverAllocRatioInitial: 77.40x
PromQL: VectorSelector: pointsOverAllocRatioFinal:   77.40x
PromQL: VectorSelector: pointsOverAllocDelta:        0.00x
expr               : count by(__name__) ({__name__=~".+"})
poolGetCount       : 2697090
poolGetSuccessCount: 269
poolPutCount       : 322
MEM STATS DIFF:         promql-eval     promql-eval - AFTER     -> Delta
    HeapAlloc  :        8.65G           18.64G                  -> 9.99G
    HeapSys    :        8.79G           19.19G                  -> 10.40G
    HeapObjects:        97.48M          110.30M                 -> 12.82M

Patch on:
level=debug name=<querier_name> ts=2019-12-06T11:44:34.426667882Z caller=proxy.go:376 queryTotalSize=3900.84MB queryLocalSize=3900.84MB
level=debug name=<querier_name> ts=2019-12-06T11:44:34.426772731Z caller=proxy.go:191 queryTotalSize=3900.84MB
….
PromQL: PROMQL_ALLOC_PATCH_ON: true
PromQL: Expression: {__name__=~".+"}
PromQL: VectorSelector: Series Count:    2696665
PromQL: VectorSelector: Current Samples: 8744867
PromQL: VectorSelector: pointsAllocInitial:   676862915; Size: 10829.81M
PromQL: VectorSelector: pointsAllocNeeded:    8744867; Size: 139.92M
PromQL: VectorSelector: pointsAllocFinal:     8760984; Size: 140.18M
PromQL: VectorSelector: pointsOverallocLimit:        2.00
PromQL: VectorSelector: pointsOverAllocRatioInitial: 77.40x
PromQL: VectorSelector: pointsOverAllocRatioFinal:   1.00x
PromQL: VectorSelector: pointsOverAllocDelta:        -76.40x
expr               : count by(__name__) ({__name__=~".+"})
poolGetCount       : 4955190
poolGetSuccessCount: 542
poolPutCount       : 917824
MEM STATS DIFF:         promql-eval     promql-eval - AFTER     -> Delta
    HeapAlloc  :        8.92G           17.35G                  -> 8.43G
    HeapSys    :        9.06G           18.05G                  -> 9.00G
    HeapObjects:        99.13M          111.68M                 -> 12.55M

Honestly, these look kind of mixed to me.  Of course there is the issue of when GCs happened as well.
It's notable that we are not actually extracting things from the pool that often.  I would be inclined to try getting rid of the pool altogether (as in https://github.com/prometheus/prometheus/pull/6427) to make this easier to reason about.  But in any case, the pool is only partially related to whether we should do the shrink.

Re escape analysis:

Escape analysis is rewritten in 1.14: https://go-review.googlesource.com/c/go/+/170322/
Go 1.14 Beta announced today: https://twitter.com/golang/status/1207027024893792256?s=20

Don't know if this would cause anything to escape less w/o running tests and deving into details of this change.

If it happens to make PromQL without the pooling no longer do heap allocations for samples, then it's of interest here.

Yes, I will try out 1.14 and see what happens.
In the meantime, we've been running some tests on the real data on the real clusters to see what positive impact (if any) various patches may have. Doing so, we've come up with an idea which should make things better.
Idea 1:

When we make a new correctly-sized slice and copy points there, we are dropping the over-allocated slice on the floor. This becomes garbage. Which is wrong. Why don't we reuse it?

Also, considering there is "let's not use sync.Pool" conversation floating around, here is another idea:
Idea 2:

Switch off sync.Pool. But keep the resizing patch. And in addition to that, keep the overallocated slice for reuse in the next loop iteration.

I did some local test runs on OS X with synthetic data to see how that works. Summary:

Using HeapSys is probably more reliable metric than anything else?
Most numbers seem to show improvement
Both "sync.Pool" and "no sync.Pool" versions showed comparable numbers
But some numbers are confusing, esp HeapObjects
Also I don't trust local tests running on OS X, we've seen things work differently on Linux in k8s.
Will need to run benchmarks and more realistic tests

# All numbers are after running same query 3 times

ORIGINAL (uses sync.Pool)
    HeapSys    : 	11.54G

PATCH1 (uses sync.Pool + resize overalloced slice)
    HeapSys    : 	2.28G

PATCH1+PATCH2 (uses sync.Pool + resize overalloced slice + put overalloced slice back to sync.Pool)
    HeapSys    : 	1.88G

PATCH3.1 (Original code, No Sync.Pool at all)
    HeapSys    : 	13.62G

PATCH3.2: No Sync.Pool at all, + resize + keep overalloced slice for next loop iteration
    HeapSys    : 	1.81G

Full logs
ORIGINAL (uses sync.Pool)
=========

real	1m19.310s
user	0m0.014s
sys	0m0.017s

PromQL: PROMQL_ALLOC_PATCH_ON : false
PromQL: PROMQL_ALLOC_PATCH2_ON: false
PromQL: Expression: {__name__=~".+"}
PromQL: VectorSelector: Series Count:    845000
PromQL: VectorSelector: Current Samples: 670000
PromQL: VectorSelector: pointsAllocInitial:   468975000; Size: 7503.60M
PromQL: VectorSelector: pointsAllocNeeded:    670000; Size: 10.72M
PromQL: VectorSelector: pointsAllocFinal:     468975000; Size: 7503.60M
PromQL: VectorSelector: pointsOverallocLimit:        2.00
PromQL: VectorSelector: pointsOverAllocRatioInitial: 699.96x
PromQL: VectorSelector: pointsOverAllocRatioFinal:   699.96x
PromQL: VectorSelector: pointsOverAllocDelta:        0.00x
expr               : count by(__name__) ({__name__=~".+"})
poolGetCount       : 2535150
poolGetSuccessCount: 1864997
poolPutCount       : 2535000

MEM STATS DIFF:   	promql-eval 	promql-eval - AFTER 	-> Delta
    HeapAlloc  : 	10.43G 		11.46G 			-> 1.03G
    HeapSys    : 	10.53G 		11.54G 			-> 1.01G
    HeapObjects: 	55.59M 		68.35M 			-> 12.76M

PATCH1 (uses sync.Pool + resize overalloced slice)
=======
real	1m11.209s
user	0m0.014s
sys	0m0.016s

PromQL: PROMQL_ALLOC_PATCH_ON : true
PromQL: PROMQL_ALLOC_PATCH2_ON: false
PromQL: Expression: {__name__=~".+"}
PromQL: VectorSelector: Series Count:    845000
PromQL: VectorSelector: Current Samples: 670000
PromQL: VectorSelector: pointsAllocInitial:   468975000; Size: 7503.60M
PromQL: VectorSelector: pointsAllocNeeded:    670000; Size: 10.72M
PromQL: VectorSelector: pointsAllocFinal:     670000; Size: 10.72M
PromQL: VectorSelector: pointsOverallocLimit:        2.00
PromQL: VectorSelector: pointsOverAllocRatioInitial: 699.96x
PromQL: VectorSelector: pointsOverAllocRatioFinal:   1.00x
PromQL: VectorSelector: pointsOverAllocDelta:        -698.96x
expr               : count by(__name__) ({__name__=~".+"})
poolGetCount       : 2535150
poolGetSuccessCount: 0
poolPutCount       : 2010000

MEM STATS DIFF:   	promql-eval 	promql-eval - AFTER 	-> Delta
    HeapAlloc  : 	974.87M 	1.12G 			-> 143.49M
    HeapSys    : 	2.28G 		2.28G 			-> 0
    HeapObjects: 	14.50M 		14.55M 			-> 48.07K

PATCH1+PATCH2 (uses sync.Pool + resize overalloced slice + put overalloced slice back to sync.Pool)
=============

real	1m6.615s
user	0m0.014s
sys	0m0.015s

PromQL: PROMQL_ALLOC_PATCH_ON : true
PromQL: PROMQL_ALLOC_PATCH2_ON: true
PromQL: Expression: {__name__=~".+"}
PromQL: VectorSelector: Series Count:    845000
PromQL: VectorSelector: Current Samples: 670000
PromQL: VectorSelector: pointsAllocInitial:   468975000; Size: 7503.60M
PromQL: VectorSelector: pointsAllocNeeded:    670000; Size: 10.72M
PromQL: VectorSelector: pointsAllocFinal:     670000; Size: 10.72M
PromQL: VectorSelector: pointsOverallocLimit:        2.00
PromQL: VectorSelector: pointsOverAllocRatioInitial: 699.96x
PromQL: VectorSelector: pointsOverAllocRatioFinal:   1.00x
PromQL: VectorSelector: pointsOverAllocDelta:        -698.96x
expr               : count by(__name__) ({__name__=~".+"})
poolGetCount       : 2535150
poolGetSuccessCount: 2534997
poolPutCount       : 4545000

MEM STATS DIFF:   	promql-eval 	promql-eval - AFTER 	-> Delta
    HeapAlloc  : 	802.34M 	1.55G 			-> 745.27M
    HeapSys    : 	1.81G 		1.88G 			-> 67.11M
    HeapObjects: 	12.55M 		22.66M 			-> 10.11M

PATCH3.1 (Original code, No Sync.Pool at all)
=============

real	1m23.233s
user	0m0.015s
sys	0m0.018s

PromQL: SYNC_POOL_IS_OFF: true
PromQL: PROMQL_ALLOC_PATCH_ON (sync.Pool) : false
PromQL: Expression: {__name__=~".+"}
PromQL: VectorSelector: Series Count:    845000
PromQL: VectorSelector: Current Samples: 670000
PromQL: VectorSelector: pointsAllocInitial:   468975000; Size: 7503.60M
PromQL: VectorSelector: pointsAllocNeeded:    670000; Size: 10.72M
PromQL: VectorSelector: pointsAllocFinal:     468975000; Size: 7503.60M
PromQL: VectorSelector: pointsOverallocLimit:        2.00
PromQL: VectorSelector: pointsOverAllocRatioInitial: 699.96x
PromQL: VectorSelector: pointsOverAllocRatioFinal:   699.96x
PromQL: VectorSelector: pointsOverAllocDelta:        0.00x
expr               : count by(__name__) ({__name__=~".+"})
poolGetCount       : 0
poolGetSuccessCount: 0
poolPutCount       : 0

MEM STATS DIFF:   	promql-eval 	promql-eval - AFTER 	-> Delta
    HeapAlloc  : 	9.39G 		8.06G 			-> -1.34G
    HeapSys    : 	11.81G 		13.62G 			-> 1.81G
    HeapObjects: 	38.23M 		16.30M 			-> -21.93M

PATCH3.2: No Sync.Pool at all, + resize + keep overalloced slice for next loop iteration
=============

real	1m8.178s
user	0m0.014s
sys	0m0.015s

PromQL: SYNC_POOL_IS_OFF: true
PromQL: PROMQL_ALLOC_PATCH_ON (sync.Pool) : true
PromQL: Expression: {__name__=~".+"}
PromQL: VectorSelector: Series Count:    845000
PromQL: VectorSelector: Current Samples: 670000
PromQL: VectorSelector: pointsAllocInitial:   468975000; Size: 7503.60M
PromQL: VectorSelector: pointsAllocNeeded:    670000; Size: 10.72M
PromQL: VectorSelector: pointsAllocFinal:     670000; Size: 10.72M
PromQL: VectorSelector: pointsOverallocLimit:        2.00
PromQL: VectorSelector: pointsOverAllocRatioInitial: 699.96x
PromQL: VectorSelector: pointsOverAllocRatioFinal:   1.00x
PromQL: VectorSelector: pointsOverAllocDelta:        -698.96x
expr               : count by(__name__) ({__name__=~".+"})
poolGetCount       : 0
poolGetSuccessCount: 0
poolPutCount       : 0

MEM STATS DIFF:   	promql-eval 	promql-eval - AFTER 	-> Delta
    HeapAlloc  : 	871.80M 	1.56G 			-> 683.37M
    HeapSys    : 	1.81G 		1.88G 			-> 67.11M
    HeapObjects: 	13.52M 		22.38M 			-> 8.86M

HeapAlloc is generally what I'd use. I'd be interested in dropping the pool, even if it was a small performance hit. It makes optimisation hard.

I'd be interested in dropping the pool

Just it benchmarks with/without pool. Killing the pool makes some benchmarks a bit slower, slightly more allocs in some of them. But it addition bytes/op show some crazy increases.
benchmark                                                                                old bytes     new bytes     delta
BenchmarkRangeQuery/expr=a_one,steps=1000-16                                                                           8468          24817         +193.07%
BenchmarkRangeQuery/expr=a_ten,steps=100-16                                                                            14171         31769         +124.18%
BenchmarkRangeQuery/expr=a_ten,steps=1000-16                                                                           21219         184729        +770.58%
BenchmarkRangeQuery/expr=a_hundred,steps=100-16                                                                        86337         262313        +203.82%
BenchmarkRangeQuery/expr=a_hundred,steps=1000-16                                                                       146745        1781832       +1114.24%
BenchmarkRangeQuery/expr=rate(a_ten[1m]),steps=1000-16                                                                 24738         188476        +661.89%
BenchmarkRangeQuery/expr=rate(a_hundred[1m]),steps=1000-16                                                             176835        1812550       +925.00%
BenchmarkRangeQuery/expr=rate(a_one[1m]),steps=10000-16                                                                40259         203686        +405.94%
BenchmarkRangeQuery/expr=rate(a_ten[1m]),steps=10000-16                                                                139119        1779392       +1179.04%
BenchmarkRangeQuery/expr=rate(a_hundred[1m]),steps=10000-16                                                            1156695       17518945      +1414.57%

Interesting?

That's not unexpected, it's those allocs that the pool is preventing. Any increases in allocations would need to be a lot smaller than that.

Short update.
I've added to the current patch to put the overallocated slice back into pool in the resize path, so that we reuse it in the next loop iteration.
In the absence of benchmarks for sparse datasets, the current benchmarks do not show much of interest:

Performance: some bits faster, some bits slower. Overall the patched version seems consistently faster to run all benchmarks (orig: 414.809s vs patch: 411.111s). Considering this is all on a laptop with CPU stepping etc (I ran each kind twice and took second result), variation is probably within margins?

Allocs: absolutely zero diffs. Except two tests which show reduction of -1.48% and -12.97%

Bytes/op: broadly no difference except a couple of cases.

I also tried bechmarks with 1.14Beta, no significant differences in anything there either.

Files:
orig.txt
patch.txt
diff.txt
PS. I may have a look into new benchmarks over xmas if I get bored with day time TV :)

Also quick question:
What does make this _, v, ok := ev.vectorSelectorSingle(it, e, ts) return OK or NOT OK?
Is it possible to figure out up-front how often we wil get OK? So that we can size the points array better?
I looked at the code and it's a bit complicated :)
Full extract:
for i, s := range e.series {
    for ts := ev.startTimestamp; ts <= ev.endTimestamp; ts += ev.interval {
        _, v, ok := ev.vectorSelectorSingle(it, e, ts)  // <-- this
        if ok {
            if ev.currentSamples < ev.maxSamples {
                ss.Points = append(ss.Points, Point{V: v, T: ts})
                ev.currentSamples++
            } else {
                ev.error(ErrTooManySamples(env))
            }
        }
    }
}

False means no sample was returned. You can presume that it's always true.

False means no sample was returned. You can presume that it's always true.

Yes, I get it :)
What I don't get is what would lead to samples being found or not. There is some correlation between data, query, step, and range. But I can't quite figure out what exactly it is.
OK, let's make it even simpler. Given same data, query, and range. Just varying "step" parameter I observe differences in how often the sample data is found. Why? What drives this?

A series may have gone stale, or target disappeared.

Did you get a chance to work on benchmarks to cover this?

We have looked at this pull request during our bug scrub.
Given the lack of updates from the author, we have decided to close this pull request.
Thank you for your contribution.


hautelook/TemplatedUriBundle package is gone

Starting from yesterday https://github.com/hautelook/TemplatedUriBundle gives 404.
Hautelook has marked as private the repository and it seems not probable that it will be back in a reasonable timeline.
This package is a fork of the original repository ( to which I still have access).
I have already updated https://packagist.org/packages/hautelook/templated-uri-bundle to point to this package.
If you are have installed this package in the past, you just need to run composer update hautelook/templated-uri-bundle and that should be it (it is needed only to update the composer.lock file).

related issue https://github.com/goetas/TemplatedUriRouter/issues/1

Hi Asmir !
I think composer update hautelook/templated-uri-bundle doesn't work for me.
I think you need to tag the repository as version 3.3.2

The latest release for hautelook/templated-uri-bundle is 3.3.1, why is it needed a 3.3.2? running composer update should just update the composer.lock with the new repository reference, right?

After the command run your lock file should have this changes
Index: composer.lock
<+>UTF-8
===================================================================
diff --git a/composer.lock b/composer.lock
--- a/william/composer.lock	(revision dcf7abc8988adb4172afda14056715d5717c278a)
+++ b/william/composer.lock	(date<PHONE_NUMBER>563)
@@ -2706,12 +2706,12 @@
             "target-dir": "Hautelook/TemplatedUriBundle",
             "source": {
                 "type": "git",
-                "url": "https://github.com/hautelook/TemplatedUriBundle.git",
+                "url": "https://github.com/goetas/TemplatedUriBundle.git",
                 "reference": "1e25566f46dabfccac11735a5c36c56f181c5782"
             },
             "dist": {
                 "type": "zip",
-                "url": "https://api.github.com/repos/hautelook/TemplatedUriBundle/zipball/1e25566f46dabfccac11735a5c36c56f181c5782",
+                "url": "https://api.github.com/repos/goetas/TemplatedUriBundle/zipball/1e25566f46dabfccac11735a5c36c56f181c5782",
                 "reference": "1e25566f46dabfccac11735a5c36c56f181c5782",
                 "shasum": ""
             },
@@ -2756,8 +2756,8 @@
                 "xml"
             ],
             "support": {
-                "issues": "https://github.com/hautelook/TemplatedUriBundle/issues",
-                "source": "https://github.com/hautelook/TemplatedUriBundle/tree/3.3.1"
+                "issues": "https://github.com/goetas/TemplatedUriBundle/issues",
+                "source": "https://github.com/goetas/TemplatedUriBundle/tree/3.3.1"
             },
             "time": "2020-07-10T07:33:54+00:00"
         },
@@ -2766,12 +2766,12 @@
             "version": "3.3.0",
             "source": {
                 "type": "git",
-                "url": "https://github.com/hautelook/TemplatedUriRouter.git",
+                "url": "https://github.com/goetas/TemplatedUriRouter.git",
                 "reference": "1a56975f78d1654fdafce8cd24ffbecb488b2add"
             },
             "dist": {
                 "type": "zip",
-                "url": "https://api.github.com/repos/hautelook/TemplatedUriRouter/zipball/1a56975f78d1654fdafce8cd24ffbecb488b2add",
+                "url": "https://api.github.com/repos/goetas/TemplatedUriRouter/zipball/1a56975f78d1654fdafce8cd24ffbecb488b2add",
                 "reference": "1a56975f78d1654fdafce8cd24ffbecb488b2add",
                 "shasum": ""
             },
@@ -2816,8 +2816,8 @@
                 "xml"
             ],
             "support": {
-                "issues": "https://github.com/hautelook/TemplatedUriRouter/issues",
-                "source": "https://github.com/hautelook/TemplatedUriRouter/tree/master"
+                "issues": "https://github.com/goetas/TemplatedUriRouter/issues",
+                "source": "https://github.com/goetas/TemplatedUriRouter/tree/3.3.0"
             },
             "time": "2020-04-21T06:39:53+00:00"
         },

No in my case. hautelook/TemplatedUriBundle is a sub-dependency of another package.
composer update hautelook/templated-uri-bundle just keeps the composer.lock file unchanged.
Maybe a composer issue or a missed an update option. I will check further.

Sorry. It seems to work now. I ran composer update hautelook/templated-uri-bundle without any effect this morning. Now it's OK
Thank you for your advice


nil is not a legal json value

The output of cddlc includes nil, which is not a legal value in JSON. The correct output value for these instances is null.

Indeed, if you don't specify an output format, you get Ruby format.
For JSON output  use -tjson (or -tneat, which selects a specific JSON formatter -- I might add others later).
For CDDL output use -tcddl.


Time Synchronization Problem between GPU and CPU in NPKit

Hello! In NPKit, we have a thread NpKit::CpuTimestampUpdateThread() looping to update the cpu timetamp and the updated value is given to a pointer cpu_timestamp_. To synchronize between CPU and GPU, NPKit has CPU SYNC event and GPU SYNC event happening at almost the same time and documents the values read from the pointer cpu_timestamp_ and clock64().
However, from my experiments, I think the cpu timestamp obtained in CPU SYNC event is not the correct value because cache coherence in the system may not be strong enough to ensure every update in the NpKit::CpuTimestampUpdateThread() writes to the memory and we may not get the most up-to-date value in CPU SYNC event even if we always use volatile in the code. Could I ask whether your team have noticed the problem and do you have any way to settle it? Thanks a lot!

Hi,
We didn’t encounter the issue you mentioned when collecting CPU timestamps from GPU. We however noticed that some bad NUMA binding might make CPU timestamp less up-to-date.
According to this code piece updating latest FIFO pointer from CPU to GPU (https://github.com/Azure/msccl-executor-nccl/blob/main/src/transport/net.cc#L1102), writing a volatile pointer allocated by ncclCudaHostCalloc should be a feasible way to make CPU memory writes visible to GPU. Note that, however, that code piece applies two extra memory buffers, which enforces memory operation order, but should not change whether some CPU write is finally visible to GPU or not.

Thanks for the reply! Could you please kindly elaborate on Some bad NUMA binding might make CPU timestamp less up-to-date. because I suspect my problem is caused by the same reason. By the way, is there an effective way to tackle the synchronization problem in such system. Thanks a lot!


⚠ Bofrost.hr has degraded performance

In db993e8, Bofrost.hr (https://www.bofrost.hr) experienced degraded performance:

HTTP code: 500
Response time: 9389 ms

Resolved: Bofrost.hr performance has improved in df4dfc6 after 8 minutes.


Provide concise documentation of tini's parameters

Please put the defined parameters (from https://github.com/krallin/tini/blob/3c5368682eb238d2177bfef98f9ac4bbc28aa9a0/src/tini.c#L238 onwards) also in the README.md in a concise table.
Can  -e EXIT_CODE be repeated?

Can -e EXIT_CODE be repeated?

It can, yes. I'll update the code to mention that.
As for putting those in the README — I'd rather keep it focused on practical examples. Running --help on the binary can get you all this info.

Ok fine for me as well ;) thanks!


Fix/user agent

What is the problem that this seeks to address? This PR contains a lot of changes that have nothing to do with the user agent.


GitOps triggered workflow not registered in ChaosCenter

What happened:
When a scenario is updated in GitHub Repository, it will trigger a workflow in ChaosCenter. From the server, I can see the pods been scheduled out, experiment is triggered. However, scenario run is not registered in Chaos Center. The logs seems to point to litmus server rejecting it as a duplicated event, however it is actually been run. Refer to logs below
Litmus-server log shows changes are synced and workflow triggered

Delegate's subscriber log: it shows discharded event, but workflow is actually triggered

The experiment is actually running

What you expected to happen:
two possibilities

the scenario is not triggered automatically (ideally), as we don't want every small change trigger an experiment run.
the scenario run is triggered and it actually show up in Chaos Center.

Where can this issue be corrected? (optional)

How to reproduce it (as minimally and precisely as possible):

Create a scenario file and sync to GitOps Repository.
Update the scenario to trigger an update and scenario run.

Anything else we need to know?:

Hey @avaakash @amityt , can i work on this issue?

Hi @Nageshbansal, sure you can work on this issue.

can you please guide me how to resolve this?


Lack the implementation of KNN and EM

line 149 in the class KNN(Algorithm) is,

https://github.com/yfzhang114/AdaNPC/blob/c280d6be2b524a895a554a7bb650ae99f5b11eea/domainbed/algorithms.py#L147-L149

Eq.(4) in the paper is，

Thanks for your project.
I have a question about Where is the detailed $L_{KNN}$, concretely, the codes of Eq.(4) (see above) and EM algorithm? Detailed questions are shown below:
Although line 149 in the class KNN(Algorithm) (see above) ,  replaces F.cross_entropy(a, b) with F.nll_loss(torch.log(a), b), I think it still does not achieve Eq.(4) function.
Because F.nll_loss(torch.log(a), b) and F.cross_entropy(a, b) have the same results, referring to How is Pytorch’s Cross Entropy function related to softmax, log softmax, and NLL.
Besides, there seems to be no implementation of EM in this project.

EM means we update the memory and train the network alternatively. The update of the whole memory bank is detailed in https://github.com/yfzhang114/AdaNPC/blob/c280d6be2b524a895a554a7bb650ae99f5b11eea/domainbed/scripts/train.py#L250, which is just we said in the paper We only periodically update $B_{k,\theta,D_S}$ and keep them fixed for the remaining time,
Here the implementation is similar to contrastive learning, where we use classification tasks instead of minimizing representation similarity directly.


[0.8.0 Dev] Panel not showing/throwing "Scales is offline" error on server creation

Could not reproduce, are you sure you didn't have a Scales instance running in the background?


Revert "build(deps): bump node-fetch from 2.6.7 to 3.3.2 (#359)"

Issue #, if available:
N/A
Description of changes:
This reverts commit da35ca94230a63bc9c17a8e453a90814540248bc.
The node-fetch dependency update is causing some issues in the backend AWS CodePipeline for Finch infrastructure. Revert the dependency update to unblock pipeline and have follow-up action items for addressing gaps between dependency update workflows and backend pipeline.
Testing done:

[x] I've reviewed the guidance in CONTRIBUTING.md

License Acceptance
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

So the blocked pipeline isn't necessarily that big of a deal as our infra is already deployed. Do you think this would be a big undertaking to investigate+fix root cause?

So the blocked pipeline isn't necessarily that big of a deal as our infra is already deployed. Do you think this would be a big undertaking to investigate+fix root cause?

It is a fair question. I am a little biased because I pushed the update through and don't want to be the one blocking the pipeline. 😅
It is a little difficult to gauge; I am still coming up to speed on this system and not very familiar with with Node development. What are your thoughts towards effort here based on your experience with the infrastructure?

Could be related to this: https://github.com/node-fetch/node-fetch/issues/1657

Could be related to this: node-fetch/node-fetch#1657. I don't think this issue has anything to do with the actual CDK code itself, just some dependencies/type related issue.

Agreed. Issue appears to be with upstream package. I have also expanded the GitHub Actions CI workflow to run npm run build so we only get updates which build.


feat: 🎸Integrate Livepeer Subgraph

What does this pull request do? Explain your changes. (required)
This pull request reduces the Explorer's load time and incorporates aggregate data (missed reward calls over last 30 rounds to start) using The Graph's self hosted data indexer (Graph Node).
Note: The Explorer currently points to a Graph Node running on a staging server set up on a Google Cloud instance. I'm working (with the help of The Graph team) on setting up a production grade hosting environment and will update this pull request to point to the new server when ready.
Specific updates (required)

Adds a brand new package to the mono repo called subgraph. This package contains the source code for the Livepeer Subgraph, a project that defines what Livepeer events we want our Graph Node to source and what transformations we'd like to make on the data in those events before storing it in a PostgreSQL database and serving it over a GraphQL endpoint. Instructions for running a Graph Node locally and deploying our subgraph are included in the README.
Combines multiple GraphQL APIs (the remote subgraph api and the local graph-sdk api) into a single GraphQL API endpoint using schema stitching inside the apollo package.

Before stitching the two schemas together, we first check the network conditions of our remote indexer. If it's down, we don't stitch, otherwise we do. This check ensures a resilient UI in the event that the indexer goes down.
The transcoders query in the subgraph api purposefully overrides the transcoders query in the graphql-sdk when stitched. This allows us to keep changes to the Explorer codebase to a minimum. For example, we don't have to conditionally check which GraphQL query to use based on network conditions.

Adds a column to the Explorer transcoder card for missed reward calls over last 30 rounds. (We're able to display this data because the indexer creates a unique "reward" entry in our PostgreSQL database for each eligible/active transcoder on every round. If a transcoder calls reward for a given round, we store its reward tokens inside this entry in a field called rewardTokens. In our GraphQL query, if rewardTokens returns null for a given transcoder and round then we know the transcoder failed to call reward.)
Makes a slight modification to the graph-sdk transcoders query. I had to rename one of the argument to match the query signature in our subgraph schema. This also required me to run the script that auto generates our graphql-sdk docs hence all the generated doc changes.
Adds an environment variable, REACT_APP_LIVEPEER_SUBGRAPH, for setting the subgraph's graphql endpoint. This currently points to the staging server I set up.

How did you test each of these updates (required)
Ran all tests to see how updates affected other areas of code. All tests pass.
Does this pull request close any open issues?
Fixes #199, #192
Checklist:

[x] I have read the CONTRIBUTING document.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.

@adamsoffer any updates?

@Randy1Burrell I pushed up some updates based on your feedback. You'll notice lots more comments! I'll have this production server provisioned along with instructions very soon. Collaborating w/ Eric on that.

@adamsoffer thanks man. I will review this soon. I'm excited to deploy this!!

Woohoo! Thanks @Randy1Burrell


dxguid & dxerr8 Prevent Cross-Compile on Linux Targeting Windows

I am cross compiling an app with Zig (v0.11.0) on Ubuntu 22.04 to be used on Windows. It appears the two libraries dxguid & dxerr8 cannot be found. Is there a way around this?
Here is the error:
/opt/hostedtoolcache/go/1.22.5/x64/pkg/tool/linux_amd64/link: running zig failed: exit status 1
error: unable to find Dynamic system library 'dxguid' using strategy 'paths_first'. searched paths:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>error: unable to find Dynamic system library 'dxerr8' using strategy 'paths_first'. searched paths:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>  /home/runner/go/pkg/mod/github.com/!allen!dang/cimgui-go@v0.0.0-2024071[105](https://github.com/alpine-client/pinnacle/actions/runs/10071331690/job/27841297791#step:12:106)5741-4b4d3ac1ee30/lib/windows/x64/libdxerr8.a

Here is the full log from the GitHub actions workflow where I've been trying to do this.
Here is the workflow file if that helps at all.
Notes: There is a chance this is caused by an issue fixed in Zig 0.12.0. However, when I try using Zig v0.12.0+ to compile my app, Zig can't see that cimgui.a, libglfw3.a, and libSDL2.a are located in /lib/windows/x64/.
(Maybe because of this issue?)
My C knowledge is very limited, so I can't say for certain. I was hoping you guys could look into this and let me know if this is an issue with Zig or cimgui-go.
Any help would be greatly appreciated.

I know nothing about zig so I might not be able to help here.

@xEricL its also my first time when I hear about zig.
I think you need to care about some type of PATH or INCLUDE_PATH in zig, however it works.
generally, in cimgui.go file is a list of C/C++/Linker flags that your compiler needs to read and use. You can try to copy-paste them in your command line and check - it may work...

The weird thing is that my cross-compiling worked when my app was using Giu v0.7 but when I upgraded to Giu v0.8 + cimgui it broke, so I was thinking it was because of a difference in cimgui (which is why I created the issue in this repo).
I will continue researching and keep you guys updated here. A lot of developers use Zig for cross compiling CGO, so I'm sure I won't be the last to run into this problem.

@gucio321 Is there a reason cimgui.a in /lib/windows/x64/ isn't following the naming convention for libraries? Like how glfw3 is called libglfw3.a and SDL2 is called libSDL2.a? I figured out how to get Zig to look in /lib folder but it keeps looking for libcimgui.a instead of cimgui.a.
| error: unable to find dynamic system library 'cimgui' using strategy 'paths_first'. searched paths:
|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
This is my build command
env GOARCH=amd64 GOOS=windows \
CC="zig cc -target x86_64-windows-gnu -Wl,--subsystem,windows -Wl,-s" \
CXX="zig c++ -target x86_64-windows-gnu -Wl,--subsystem,windows -Wl,-s" \
<EMAIL_ADDRESS>\
<EMAIL_ADDRESS>-lcimgui -lglfw3 -lSDL2" \
go build -trimpath -p 4 -ldflags="-H=windowsgui" -o bin/pinnacle-windows-amd64.exe .

But it isn't /lib its $PKGDIR/lib
These paths are set in cimgui.go file

Btw I undersrand meaning convention note and yeah, if this is a standard we should change to libcimgui

Are you sure you need to specify these LDFLAGS in your zig command?

Are you sure you need to specify these LDFLAGS in your zig command?

Are you sure you need to specify these LDFLAGS in your zig command?

I'm not 100% sure. Honestly I've just been trying everything to get it to work. The biggest issue is that dxguid and dxerr8 can't be found (like in the original post). Do you know where it should be finding that library?

@gucio321 I just need to be able to cross-compile for Windows on Linux. Have you guys been able to do that after giu migrated to cimgui-go in v0.8?

dx* sounds like direcx library. What requires that ? I think cimgui-go uses glfw with opengl

in case of cross-compilation for windows: I used to do that with mingw:
CC="x86_64-w64-mingw32-gcc" CXX="x86_64-w64-mingw32-g++" GOOS="windows" GOARCH="amd64" CGO_ENABLED="1" go build -o widgets.exe .

This method iirc is described somewhere in giu docs.

here is a workflow using that method: https://github.com/gucio321/tic-tac-go/blob/master/.github/workflows/build.yml

dx* sounds like direcx library. What requires that ? I think cimgui-go uses glfw with opengl

The only place it is mentioned is here.
https://github.com/AllenDang/cimgui-go/blob/4b4d3ac1ee3063ee5ecdc89044b664b6d53dc5e1/sdl_backend.go#L6

@xEricL can we close this? I think that the original issue was fixed (mingw compilation works)?


Caching-support + InMemory CacheStore (#102)

wip

wip

Fixes

Updated config.yaml

Fixes

wip

Updated caching to support caching paginated and limited results

Added caching with in memory cache store

fix

Added CachingStartupExtensionsGenerator tests

Updated implementation to use strings for cacheKeys, and added construction of cache keys to DataStoreService

A little cleanup

more cleanup

fixes

Codecov Report

Merging #103 (c1a8480) into main (b440906) will decrease coverage by 1.12%.
The diff coverage is 5.55%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main     #103      +/-   ##
==========================================
- Coverage   49.48%   48.36%   -1.12%
==========================================
  Files          60       62       +2
  Lines        1166     1193      +27
==========================================
  Hits          577      577
- Misses        589      616      +27

Impacted Files
Coverage Δ

...onfiguration/Extensions/ConfigurationExtensions.cs
77.77% <0.00%> (-6.23%)
:arrow_down:

...ation/Options/CacheStore/RedisCacheStoreOptions.cs
0.00% <0.00%> (ø)

...oduct/Features/Caching/CachingStartupExtensions.cs
0.00% <0.00%> (ø)

...ures/Caching/Services/InMemoryCacheStoreService.cs
0.00% <ø> (ø)

...eatures/Caching/Services/RedisCacheStoreService.cs
0.00% <0.00%> (ø)

...ct/Features/DataStore/Services/DataStoreService.cs
0.00% <0.00%> (ø)

...talGenerators/CachingStartupExtensionsGenerator.cs
90.00% <50.00%> (-4.74%)
:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Fix rfc2616 Section 4.2 compliance:

basic_headers no longer combines fields with the same name by appending a comma and concatenating the two values together. This was breaking certain header fields which expect each value to be distinct, such as the "Set-Cookie" header.
Now the container behaves more like a multi set with respect to insertion of multiple values with the same field name. Additional member functions are provided to provide extra functionality.
Reviewers: @HowardHinnant @scottschurr @willwray

[Current coverage][cc-pull] is 98.04%

Merging [#37][cc-pull] into [master][cc-base-branch] will decrease coverage by 0.05%

@@             master        #37   diff @@
==========================================
  Files            72         72
  Lines          5073       5075     +2
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits           4977       4976     -1
- Misses           96         99     +3
  Partials          0          0

Powered by Codecov. Last updated by [999e2fa...db58cf1][cc-compare]
[cc-base-branch]: https://codecov.io/gh/vinniefalco/Beast/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/vinniefalco/Beast/compare/999e2fa0318b5982736d3ea01a418770ea802671...db58cf1c0f4f215c9bc15450e096e41e2c169748
[cc-pull]: https://codecov.io/gh/vinniefalco/Beast/pull/37?src=pr

:+1: Looks good to me.

👍 Looks good here.  Passes tests on macOS.


Monthly ESMValtool meeting September

The next monthly ESMValTool meeting will be in September.  The result of the poll is that it will be on
Thursday September 10 at 10:00 CEST (9:00 BST).
The duration will be one hour. Join the zoom meeting.
@ESMValGroup/esmvaltool-developmentteam
Format and agenda

We start the meeting by doing a round where everyone gets one minute to introduce themselves and tell what they will be working on in the next month or what they would like help with.
Using the input from the introduction round, we can decide to talk about specific topics with the entire group or make smaller breakout groups to continue the discussion there.
If you already know that you would like to bring up a particular topic in the meeting, you can link the GitHub issue to this issue so it's easy to find for other people in the meeting and we can already start discussing there before the meeting. Just create a new GitHub issue if there is no open issue yet.
GitHub issues are our preferred way to communicate outside the meeting and keep track of work, so any feature requests/action items/etc discussed in the meeting should ideally result in a GitHub issue.

Notes from the previous meeting can be found here: https://github.com/ESMValGroup/ESMValTool/issues/1761#issuecomment-668042231

anytime's good for me, bud as long as you tell me the BST time :grin:

@ESMValGroup/esmvaltool-developmentteam The meeting will take place on Thursday September 10 at 10:00 CEST (9:00 BST).

Release strategy discussion #1809

Multimodel masking: ESMValGroup/ESMValCore#487, ESMValGroup/ESMValCore#767
Preprocessor ordering (extract_time): ESMValGroup/ESMValCore#762

Thanks for attending @ESMValGroup/esmvaltool-developmentteam!
Here is a summary with some news:

We have a proposal for when and how we will make the releases in the coming year #1809, please comment if you have an opinion
We may have another (online) workshop this autumn. @axel-lauer will look into organizing this, starting with creating an issue to collect topics we could work on
@axel-lauer Will look into compiling a list of persons who would be able to do scientific pull request reviews, @bouweandela will do the same for technical pull request reviews
We talked about why there are two different CMORization procedures: CMORization through a script in ESMValTool is easier to implement, while CMORization by fixes in ESMValCore is a bit more reliable because it has unit tests and the only viable option for large datasets.
@remi-kazeroni Joined the ESMValTool community at DLR!
@stefsmeets Joined the ESMValTool community at Netherlands eScience Center!
@sloosvel Announced that one of the PRIMAVERA diagnostics will soon be ready for publication, it would be good to have the associated pull request merged so it can be included in the next release and thus have a DOI that can go into the paper
@bettina-gier Announced she will soon be making a pull request for a diagnostic in a new paper
@rswamina Encountered some cases of slow performance and will make issues so others can try to help
@SarahAlidoost Announced that an ESMValTool tutorial  will be given a the Netherlands eScience Center next Friday
@jvegasbsc Is working on improving the (observational) data download and CMORization experience in #1657
@nielsdrost Announced that the eWaterCycle project is planning to add support for more observational datasets
@valeriupredoi Asked for input on the issue with poor performance when regridding irregular grids
@bouweandela Announced that regridding rectangular grids will be lazy in iris 3.0
@schlunma Asked for input on a new preprocessor function for multi model masking ESMValGroup/ESMValCore#487, ESMValGroup/ESMValCore#767
We talked shortly about how to use the extract_time preprocessor twice ESMValGroup/ESMValCore#762
The @ESMValGroup/userengagementteam will be involved in collecting requirements for potential future versions of ESMValTool or a related tool through @rswamina

The list of updates above has been compiled from memory, so please forgive me if I forgot anything and feel free to add it yourself.
If discussion in issues feels slow and unproductive to you, please remember that you do not need to wait for the next monthly meeting if you would like to discuss an issue or pull request in person. Just ask the relevant people if they would like to make a short call with you, e.g. using the open-source video meeting service https://meet.jit.si/ or simply WhatApp.


Potential windows bug

examples/viewers_dicom2D/index.html three.min.js:631 THREE.WebGLRenderer 75 models.stack.js:235 do not know how to order the frames... – Hemanth Anakapalle 10 hours ago
three.min.js:704 THREE.WebGLProgram: shader error: 0 gl.VALIDATE_STATUS false gl.getProgramInfoLog C:\fakepath(167,37-79): error X4002: indefinite logarithm Warning: D3D shader compilation failed with default flags. (ps_3_0) Retrying with avoid flow control C:\fakepath(167,37-79): error X4002: indefinite logarithm Warning: D3D shader compilation failed with avoid flow control flags. (ps_3_0) Retrying with prefer flow control C:\fakepath(167,37-79): error X4002: indefinite logarithm – Hemanth Anakapalle 10 hours ago
Warning: D3D shader compilation failed with prefer flow control flags. (ps_3_0) Failed to create D3D shaders. – Hemanth Anakapalle 10 hours ago
WebGL: INVALID_OPERATION: useProgram: program not valid three.min.js:671 WebGL: INVALID_OPERATION: drawArrays: no valid shader program in use 254three.min.js:600 WebGL: INVALID_OPERATION: useProgram: program not valid three.min.js:600 WebGL: too many errors, no more errors will be reported to the console for this context. – Hemanth Anakapalle 10 hours ago
I am seeing all these errors over browser console. I am trying this over windows, Is it fine ? –

OS : Windows 7 Professional
System Type : 64-bit
Chrome Version : Version 50.0.2661.102 m
(Google Chrome is up to date.)
Attached snapshots of the issue, console logs and also rest available information (see if it helps to debug).
It is an critical issue, please prioritize this, Thanks.

Note : It is working fine on my friend's laptop. His OS is windows 10.
Chrome version is same on our both laptops.

Hmm I tried on a windows 7 VM and Internet Explorer and it worked fine.
Does the following work on your machine?
http://jsfiddle.net/gh/get/library/pure/fnndsc/ami/tree/master/lessons/03#run
Also, can you try it in an "incognito" tab to see if it could be a cache issue?
Thanks

Tried both Nicolas .... still same, not working.

OK - and in the console when it writes down the error, can you click on it somehow to see which line of the shaders is the problem?

Are your graphic card drivers up to date too?

Can you please try this one?
http://jsfiddle.net/w1r76rxm/

Drivers I will check.
Tried latest link too, still same.

OK - If you click on the link for (anonymous function) (index):159 in the error message, does it point you to the fragment shader code that can not be compiled?

Yes ... it is pointing to "animate();" statement.

OK :/
I would now make sure the drivers are up to date then.
Does this one work?
http://threejs.org/examples/#webgl_shader_lava

working ... (y)

what is the issue ?

hmmm no idea yet, for some reason your machine can not compile the shaders properly.
Can you try in internet explorer?

http://threejs.org/examples/#webgl_shader_lava
This is working Nicolas.

yes sorry, I meant try the ami fiddle:
http://jsfiddle.net/w1r76rxm/

Tried on all browsers ... it is not working.

It would be really useful if we could know what part of the shader it is referring to:

this one points to the THREE.JS code, is there any way to see the AMI JS shader code that triggered this error?

Can you also try this one: (threejs 76)
http://jsfiddle.net/w1r76rxm/

Can you try this one in an incognito tab?
http://jsfiddle.net/xf12mc2y/
There was an illegal shader declaration which could have created the issue...

Last link is working in incognito tab. But not in normal tab.

Awesome, the issue was the following line of code:
int issigned = int(pow(-1., float(bytemeR[0])));

that I replaced by:
int issigned = 1 - 2 * bytemeR[0];

https://github.com/FNNDSC/ami/commit/8d2616647372c6944b2377abdfcb6541ae94f436
I cut a new release and you should be good to go now.
If it only works in "incognito mode", it is because an older version of the library is still in the cache. Reload the page while pressing "shift" and you should be good to go.
Please try to load the dataset you couldn't to make sure it all works now:
https://fnndsc.github.io/ami/examples/viewers_dicom2D/
Thanks!

It works, Good.
Thanks.

Great! thanks for helping to track it down!


UCP internal variable "page.ucp_version" needs to be bumped to 3.2.4

File: ee/ucp/admin/install/upgrade.md
This page references 3.2.3 thru an internal variable:
This page helps you upgrade Docker Universal Control Plane (UCP) to version {{ page.ucp_version }}.
The internal variable needs to be bumped to 3.2.4 for the November 2019 release.

@cemckenzie @lena-larionova Can you see the updated version? When I view the page, it references 3.2.4.

Yep, that shows the right version for me now.

OK, I will close this issue for now.


Refactor(exporter-variables-scss): Setup exporter as ES module

Description

For testing purposes
Trying to set up an exporter for Supernova.io as an EcmaScript module to test the support of the ESM syntax in exporters.
Additional context

Issue reference

https://supernovafriends.slack.com/archives/C020NGTNA0H/p1723720559286119

coverage: 74.222% (-4.2%) from 78.435%
when pulling 0dc12dc00cb4009c0cfe3b3b481c6a9bb7d9533d on test/variables-exporter-module
into 7ba942ca24b4e85ff1eeb447f94150aa97dfc7f3 on main.

This will probably not happen from Supernova.io side. Closing.


Delete Test Artifact link seems to be wrong.  Takes you to the same page, is it supposed to do that?

Delete Test Artifact link seems to be wrong.  Takes you to the same page, is it supposed to do that?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0e9f02cb-f094-d6bc-ca7b-973db9de0082
Version Independent ID: 1af6a7f5-9e99-ec99-081d-c0bf9e7ac5f5
Content: Move, change, delete, or restore user stories, bugs, tasks, and other work items - Azure Boards
Content Source: docs/boards/backlogs/remove-delete-work-items.md
Product: devops
GitHub Login: @KathrynEE
Microsoft Alias: kaelli

@gperrego - thank you for the feedback. You are correct, and I'm updating the links in this article today.


SAN-24-Fixing-Date-and-Amount updating ConfirmationController and ConfirmationControllerTest

updating ConfirmationController and ConfirmationControllerTest

CI: Security warnings found!

CI: Security warnings found!


lab 2 review

Hello Matteo, I'm reviewing your implementation of the evolutionary strategy for creating an agent which is able to play nim.
The notes are very clear and helps in understanding your thought-flow when implementing the strategy.
I really liked how you set some conditions to choose what do to and weighted them, you can surely tune some of them to allow more precise moves, since we know how to trick nim games. Anyway, I think this is a really good implementation and by "copying" the mechanism of neurons you can achieve great results. Also, with that sigma you can achieve a good balance between exploration and exploitation, and if we just want to speed-up the process, an higher value for the sigma will be needed at the beginning.
Overall, the code is well structured and the plus strategy adopted helps in maintaining a sort of steady state, so not losing good information for future generations.

Hi Francesco, thanks for you review!