Initial implementation

WIP.  Tracking #5.

Why the move from cobra to cli?

I like where this is going but will hold off on reviewing until you let us know it's ready for one

TBH I just wanted to try out cli.  I think it's marginally better in that it saves you from storing flags as global variables.
And yeah, I will let you guys know when this is ready for review.

@derekchiang what's the status here?

I've been taking a break in China therefore the inactivity.  Before that I had been meaning to add more tests and clean up the code a bit but I guess that shouldn't block code reviews.  So yeah, please give the code a review!  @jonboulle @klizhentas
Currently the tool is very much usable, although its image discovery mechanism is limited to file system.  Eventually we will probably want to borrow the image discovery code from rkt.

k, the first comment: can you please split this into two commits: the code itself and then vendored part. Right now it's quite hard to review the whole thing.

I seem to have done something peculiar and now Github won't let me reopen this PR.  I've opened a new PR in #10.  Please look there instead.  As Sasha suggested, I've broken done the PR into 2 commits.


Bad autocomplete in WebUI

Bad autocomplete in WebUI
In Chrome Version 112.0.5615.138 autocompleting login to search area.

This is fixed in v4


Allow recursive mkdir to install path

Fixes some action builds, such as with https://github.com/ange-yaghi/engine-sim

could you make a new release containing this commit?

There you go: https://github.com/MarkusJx/install-boost/releases/tag/v2.3.1


lidar_link and base_link   coordinate System issue

Run in Ubuntu 22.04 ROS2(humble)
When I run the dataset with "campus_small_dataset", and the config parmas is correct, I can get the total pointcloud, but the base_link is error (the Z axis is orientate  downward ), the picture show. Thank you, how can I solve the problem.


Master not building

Currently unable to build skydns due to the "transport" errors, can replicate by trying to build etcd master using official steps.
# git clone https://github.com/coreos/etcd.git
Cloning into 'etcd'...
remote: Counting objects: 61749, done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 61749 (delta 12), reused 0 (delta 0), pack-reused 61711
Receiving objects: 100% (61749/61749), 30.13 MiB | 206.00 KiB/s, done.
Resolving deltas: 100% (38453/38453), done.
~ # cd etcd/
~/etcd # ./build
# github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport
gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/listener.go:180: cfg.GetClientCertificate undefined (type *tls.Config has no field or method GetClientCertificate)
gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/listener.go:180: undefined: tls.CertificateRequestInfo
gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/listener_tls.go:151: undefined: net.DefaultResolver in net.DefaultResolver.LookupHost
# github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/netutil
gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/netutil/netutil.go:45: undefined: net.DefaultResolver in net.DefaultResolver.LookupPort
gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/netutil/netutil.go:55: undefined: net.DefaultResolver in net.DefaultResolver.LookupIPAddr
# github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/debugutil
gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/debugutil/pprof.go:28: undefined: runtime.SetMutexProfileFraction
gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/debugutil/pprof.go:30: undefined: runtime.SetMutexProfileFraction

@braddeicide is this with go1.8?

go version go1.7.3 linux/amd64
Is 1.8 required?

@braddeicide go 1.8 is needed to build master / 3.2, yes. The 3.1 branch will build with 1.7.

Yep, that's what i get for skimming :)
"For those wanting to try the very latest version, build etcd from the master branch. Go version 1.8+ is required to build the latest version of etcd"
Thanks.


#618:  Improvements for UrlUpdaterReportTest

This PR supports #618 by adding an improvement for the UrlUpdaterReportTest

We could create a new issue to improve the test even more? Right now, we use hard coded numbers of total urls. Maybe, we could extend the UrlUpdaterMock with some attributes for total number of urls or create a new Mocking Class for the UrlUpdaterReport test


Can't call std.variant.visit from a pure function

snarwin+bugzilla (@pbackus) reported this on 2016-11-05T03:07:05Z
Transfered from https://issues.dlang.org/show_bug.cgi?id=16662
CC List

andrei (@andralex)
ilyayaroshenko

Description
Under dmd 2.072, calling std.variant.visit from inside a pure function is forbidden, even if all of the functions used to instantiate visit are themselves pure.

Example code:

import std.variant;

alias IntOrString = Algebraic!(int, string);

IntOrString twice(IntOrString v) pure {
	return v.visit!(
		(int n) pure => IntOrString(2*n),
		(string s) pure => IntOrString(s ~ s)
	)();
}

...and the compiler error:

Error: pure function 'test2.twice' cannot call impure function 'std.variant.visit!(function (int n) => VariantN(& handler, cast(ubyte)0u, ).this(2 * n), function (string s) => VariantN(& handler, cast(ubyte)0u, ).this(s ~ s)).visit!(VariantN!(16LU, int, string)).visit'

snarwin+bugzilla commented on 2018-03-20T20:39:51Z
I've made some progress on this, but am unable to make VariantN.peek pure, because this if statement

    if (type != typeid(T))
        return null;

...calls object.opEquals, which is not pure. So issue #13933 needs to be fixed before we can solve this.

ilyayaroshenko commented on 2017-12-16T16:05:22Z
Variant's "get" method is not pure too. It was a big surprise. No I need to figure out how to deal with it, the whole big project is pure.

snarwin+bugzilla commented on 2017-06-04T22:28:07Z
Seems like the culprit is VariantN.peek, which calls the property function "type", which calls (through "fptr") an instance of the internal template function "handler", which isn't pure.

Almost all of VariantN's operations use "handler" internally, so making it pure would likely be difficult. On the other hand, modifying "fptr" to point to different instances of "handler" is how VariantN keeps track of the type of the stored object, so avoiding the use of "handler" would require substantial changes to VariantN's implementation.

Here's the trail of breadcrumbs:

---

// test.d
import std.variant;

alias Example = Algebraic!(int, double);

void main() {
	Example x;
	pragma(msg, typeof(&x.peek!int));
	pragma(msg, typeof(&x.type));
}

---

$ dmd test.d
inout(int)* delegate() inout @property @system
TypeInfo delegate() const nothrow @property @trusted

$ dmd --version
DMD64 D Compiler v2.074.0


Add .github/PULL_REQUEST_TEMPLATE.md

Add .github/PULL_REQUEST_TEMPLATE.md
Signed-off-by: Jiangnan Jia<EMAIL_ADDRESS>

Thanks for contributing!


Manually import / require describe it etc?

Is there anyway to manually import / require functions like describe and it?
I'd like to use mocha without the commandline interface, in a regular browser. (I'm using JSPM as dependency manager)
I couldn't find documentation on this on the Mocha website.

@peteruithoven  see here for using mocha in browser: https://mochajs.org/#running-mocha-in-the-browser

Ah thanks, the extra examples you added where the docs I was missing.
I'm not able to get your JSPM(SystemJS) example working though, are you sure you've included all the details? Is there more info elsewhere?
I've created a very minimal example of what I tried:
https://github.com/peteruithoven/jspm-mocha

Anyone?

see #956 for further discussion


Feature request: turn off echo

Hi Nicole!
Can we turn off / disable the echoing of the cypher script contents during the script executions, e.g. as shown in my comment here? :
https://github.com/neo4j/neo4j/issues/5901#issuecomment-379013851
That issue also raises the subject of the commenting of cypher statements in cypher scripts, which throws errors in Cycli (but not cypher-shell).  I believe that is a Neo4j issue (hence, my post there)?
Thanks again for all your work; e.g. https://github.com/nicolewhite/cycli/pull/60 -- I am thrilled that you continue to support Cycli, which is the best cypher CLI, IMO.  :-D

This PR #63 should add echo toggling (when combined with -f) - but it's not yet ready for merging with master!


[krbkeys] Should we look up account UUIDs in the Auth service?

Currently an account section which requests account generation stores the generated account UUID in an annotation. This is usual practice for a Kubernetes operator but it means that if the edge cluster is torn down and recreated we lose this information.
It might be better to treat the account mapping in the Auth service as authoritative, if it exists.

This will need to keep the separate account_uuid step, and it's probably easiest to keep the annotation as well. The only changes should be:

Move the principal mapping into account_uuid.
Look up the principal in the Auth service if we have no annotation. Only create a new account as a fallback.
Possibly we should gensym our own UUID rather than letting the ConfigDB do it? Then the auth principal mapping creation is atomic. Otherwise we still risk crashing between account creation and principal mapping; this will leave an orphaned account object which will never be cleaned up.


Make client.write_parquet_dataset available for export

This commit adds write_parquet_dataset to the imports from .client in
init.py
Previously, from json2parquet import write_parquet_dataset would
result in an error: ImportError: cannot import name 'write_parquet_dataset' from 'json2parquet'

Thanks for the PR for this!


Live score handling support

We can display multiplied score in gameplay scene when needed.
For this, we need to get the score from the API or whatever, and use a function / multiplier to handle it.

A draft solution of this issue has being pushed onto branch derived/tboard_ntcsAppend in commit 19509a7.
Need review on this.

Tested on this commit: osu!(lazer) tourney will crash when stable tourney is using for spectating, and score is being transferred into lazer while adjusting that sliding bar.

Although I'll look into this shortly, you can look through the 'runtime' log file for suspicious exceptions.

After some black-box testing here are some obvious behaviours:

The program will crash when multiplier are set in certain value range. (observed 0.7-0.9, probably 1.1 - 1.3)

Apart from this range, live change on multiplier value won't cause any crash.

logs

Stack overflow.
Repeat 1763 times:
at osu.Framework.Bindables.BindableNumber1[[System.Double, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].setValue(Double)    at osu.Framework.Bindables.Bindable1[[System.Double, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TriggerValueChange(Double, osu.Framework.Bindables.Bindable1<Double>, Boolean, Boolean)    at osu.Framework.Bindables.Bindable1[[System.Double, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].set_Value(Double)
at osu.Framework.Bindables.Bindable1[[System.Double, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TriggerValueChange(Double, osu.Framework.Bindables.Bindable1, Boolean, Boolean)
at osu.Framework.Bindables.Bindable1[[System.Double, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TriggerValueChange(Double, osu.Framework.Bindables.Bindable1, Boolean, Boolean)
at osu.Framework.Bindables.BindableNumber1[[System.Double, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].setValue(Double)    at osu.Framework.Bindables.Bindable1[[System.Int64, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TriggerValueChange(Int64, osu.Framework.Bindables.Bindable1<Int64>, Boolean, Boolean)    at osu.Framework.Bindables.BindableNumber1[[System.Int64, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].setValue(Int64)
at osu.Game.Tournament.IPC.FileBasedIPC.b__28_0()
at osu.Framework.Threading.ScheduledDelegate.RunTaskInternal()
at osu.Framework.Threading.Scheduler.Update()
at osu.Framework.Graphics.Drawable.UpdateSubTree()
at osu.Framework.Graphics.Containers.CompositeDrawable.UpdateSubTree()
at osu.Framework.Platform.GameHost.UpdateFrame()
at osu.Framework.Threading.GameThread.processFrame()
at osu.Framework.Threading.GameThread.g__runWork|70_0()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
程序“[22660] dotnet.exe”已退出，代码为 -1073741819 (0xc0000005)。

The score calculation won't work when value set to 0, the score will be displayed as there is no multiplier.
Any value between 0.1-0.6 (inclusive) will always causing score displayed as 0.

In TournamentMatchScoreDisplay, there seems to be a infinite loop:

Assuming we've just changed the redMultiplier;
adjustedTeam1Score will be changed;
Team1Score will be updated in displayTeam1Score;
adjustedTeam1Score will be changed again, then we are stuck, causing stack overflow.

We should change the logic a little bit.

Gameplay screen implemented by gosu/tosu have the ability to calculate score multiplier per contestant, which met the requirement of the OFFC tournament, as there will be FreeMod multiplier for different combinations of mods. For instance:

EZ Multiplier
HDEZ Multiplier

x2.4
x3

Live score calculation per team can be left in the code as an extra feature (handy for 1v1 tournament or don't want to setup gosu/tosu), however we have to think a way to implement score gathering per contestant using gosu/tosu into lazer tourney. Further work on this can be tracked in another issue.

Adding priority:3 to this issue now since:

Live score calculation will not being used in OFFC tournament, as they have decided to use gosu/tosumemory for gameplay instead. Another staff will be able to handle this.
To make this feature really comes in handy for tournament, we have to change live score calculation per team to live score calculation per contestant, which would be a huge work to do.

Considering these factors, this issue can be treated as a "sidequest" for future development instead. Soon I will revoming score multiplier slider in Gameplay screen, as they are still bugged.

Since we have no plans to support this feature in the code at this stage, closing this issue now.


Migrate from plain manifests to helm charts for meshery installation

Signed-off-by: Darren Lau<EMAIL_ADDRESS>Description
This PR fixes #https://github.com/meshery/meshery/issues/4103
Notes for Reviewers
Signed commits

[x] Yes, I signed my commits.

@panyuenlau Thank you for taking this up.
Heads up, there are some lint errors. We use golangci-lint.
Couple of things:

We should be downloading the charts to system when we system start.
The downloaded charts should ideally be "cached" in the sense that if there is no need to update, we should not try to fetch the latest Helm chart. (We could discuss the implementation of this)
system start with Meshery running throws an error. Ideally system start should be idempotent. We might need to update the helper function in MeshKit to take this into consideration. I'm not quite sure to the extent in which Helm supports this cleanly. See: helm/helm#3134

Same goes for system update. It fails to execute because of the same reason.

Hey @navendu-pottekkat, thank you for your review

The ApplyHelmChart() actually downloads the helm chart to the /tmp folder by calling the fetchHelmChart() function( reference: https://github.com/panyuenlau/meshkit/blob/f0a31a1b873effe10f500c5cb23d08177cbb977c/utils/kubernetes/apply-helm-chart.go#L307), which is different from where we have the manifests downloaded: ~/.meshery
The fetchHelmChart() checks if the required helm chart is already present in the user's file system, reference: https://github.com/panyuenlau/meshkit/blob/f0a31a1b873effe10f500c5cb23d08177cbb977c/utils/kubernetes/apply-helm-chart.go#L330-L332
helm install is indeed not idempotent
For system update, the meshkit helper function (ApplyHelmChart) needs to be updated to have the option to do helm upgrade instead of helm install

@navendu-pottekkat The system update command now works with this PR (https://github.com/layer5io/meshkit/pull/118) in the meshkit, the PR will also resolve linters issues identified by the CI

Codecov Report

Merging #4389 (56314a4) into master (d943ac3) will decrease coverage by 7.56%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #4389      +/-   ##
==========================================
- Coverage   31.85%   24.29%   -7.57%
==========================================
  Files          59       59
  Lines        5063     5034      -29
==========================================
- Hits         1613     1223     -390
- Misses       2989     3472     +483
+ Partials      461      339     -122

Flag
Coverage Δ

gointegrationtests
?

unittests
24.29% <0.00%> (+0.13%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

mesheryctl/internal/cli/root/system/start.go
1.09% <0.00%> (-53.87%)
:arrow_down:

mesheryctl/internal/cli/root/system/stop.go
0.86% <0.00%> (-36.75%)
:arrow_down:

mesheryctl/internal/cli/root/system/update.go
1.04% <0.00%> (-34.75%)
:arrow_down:

mesheryctl/pkg/utils/helpers.go
60.99% <ø> (ø)

mesheryctl/internal/cli/root/system/status.go
0.00% <0.00%> (-62.83%)
:arrow_down:

mesheryctl/internal/cli/root/system/restart.go
2.50% <0.00%> (-47.50%)
:arrow_down:

mesheryctl/internal/cli/root/system/check.go
1.39% <0.00%> (-22.38%)
:arrow_down:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0264776...56314a4. Read the comment docs.

@panyuenlau I'm arriving late to the party sweat_smile Seems like we still have the support for meshconfig (~/.meshery/config.yaml) under works. This was the one I used:
contexts:
  local:
    endpoint: http://<IP_ADDRESS>:9081
    token: Default
    platform: kubernetes
    adapters:
    - meshery-istio
    channel: edge
    version: latest
current-context: local
tokens:
- location: auth.json
  name: Default

We might wanna have control over which adapters to install. Right now, it installs all of them regardless the ones specified.

The implementation now only installs the adapters that are specified in the context

@navendu-pottekkat right we don't need to delete all pods and re-deploy them, helm upgrade detects and restarts the pods that need an update. It's fixed now

Tested it with the latest release of MeshKit. Everything looks and works good.

One thing to add here is some logs when mesheryctl system update is run. It shows only logs from resetting as shown in the screenshot above.
Related to that, do we need to delete and redeploy Meshery on system update? update should fetch updated Helm charts (if skipResetFlag is not set) and pull the latest images from docker hub. Is that functionality achieved without having to restart?

@navendu-pottekkat Thank you for helping to test that!
And right we don't need to delete all pods and re-deploy them, helm upgrade detects and restarts the pods that need an update. It's fixed now


set uaO From domain as in rfc3261 <IP_ADDRESS> From

одной опции SipConf.my_address недостаточно для инвайта, как вы думаете?

Hi, sorry for the delay. Slightly different, but more flexible patch has been pushed to add this feature. So, you can set the From domain by doing:
uaO = UA(...)
uaO.from_domain = 'sip.mydomain.net'
uaO.recvEvent(event)
Please let us know if you think that this needs further improvements.


Mobile Devices, User Checkouts & Rate Limiting

Main Points:

Mobile Devices are now considered from Jamf - appropriate switches have been added to the argsparser
User Checkouts have been added, with the addition of three new functions: checkout_snipe_asset, checkin_snipe_asset and get_snipe_user_id
Rate Limiting has been implemented with the use of hooks from the requests module. All requests now contain a hook that references back to the new function request_handler

Mobile Devices:
Adding this functionality required either running two different for loops that were similar but contained a lot of repeated code, or by iterating through a dictionary which tells the for loop what kind of device the array holds.
This was needed due to the subtle differences in the models that Jamf provides. A lot of the code can be shared, but the code that can't has been wrapped in if statements.
User Checkouts:
All the functionality is tied to the -u switches. Created an argsparse group to stop it being possible to specify multiple switches. Was unsure of what to have as the default functionality for this switch, but since Jamf doesn't clear the location username field, I thought that it checkout out items that have an old username in them would be quite annoying. I don't see this being too useful apart from when you want to first import all the computers into Snipe. Might help some people.
The functions use the Snipe API endpoints to search and checkout (and check in if already checked out to another user) devices to users.
The settings file also now contains a section that specifies what you want to search snipe with. For example, your usernames might not match against the two systems, but your email addresses might. Just wanted to put that in there for more flexibility.
Rate Limiting:
After searching for quite a while, I found that hooks are the way to log every request sent to a host. The request_handler function gets a copy of the response and counts how many Snipe IT requests have been made in the time since the first request to Snipe. If it goes over 1.95 requests per second, it brings it down by 0.5 seconds plus the time that our average is over 1.95 by.
If we're still rate limited even with this, we try again after two seconds. If we don't have rate limiting on, we quit the application and ask the user to turn on rate limiting.
This function requires an argument args, which conflicted with the argsparser argument. The other variable we had also called args has been renamed to user_args

I'm not ignoring this! I just haven't had time to take a look at it yet. =)

No worries! Take your time :)

Ok, this looks good. I was able to test and it seems like everything is working.
A few things I think we might need to change (that'd I'd like to put in writing as notes for myself as a reminder) and they're not big enough to be a blocker:

There are lots of messages like: INFO:root:No custom configuration found in settings.conf for asset tag name upon asset creation. INFO:root:Going over snipe rate limit of 120/minute (1.9610873473914496/minute), sleeping for 0.5110873473914497 Which I find a tit bit noisy for almost every entry. I have a few ideas on how to make it a little quieter (or perhaps indicate the asset tag created on the previous line)
Documentation needs to be updated to specifically mention that users are not created if they don't already exist.
It might be better to use the /api/v1/users to get the authoritative list rather than looking up each one, but we can do that later.
settings.conf needs a little more explicit documentation as that's what trips people up the most when trying to use this and most of the examples there probably won't work for them.

All in all, thanks for the great job. =)

My pleasure!

You're right, that probably needs to be run only once instead of being checked upon every single Jamf item! Something like No custom configuration found in settings.conf for asset tag name upon asset creation. Using jamf-{id} for new assets in the checking section of the script would be better Then a simple boolean check when iterating through the devices, and not displaying any output at that stage should suffice. Rate limiting messages should probably go under logging.debug?
Agreed.
Good idea, didn't really think about that! Would be a lot faster to do it that way.
Was thinking about this myself. Will see what I can do.

I'll make some amends next week some time and create another pull request.


Source files aren't being linked despite everything seeming to be OK

I've been working on trying to integrate SourceLink into some projects, but I've been running into a problem where the resulting PDB doesn't seem to have anything linked. For reference here's one of the PRs: https://github.com/Particular/NServiceBus.RabbitMQ/pull/387
I'm using the SourceLink.Create.GitHub package and dotnet-sourcelink-git tool. I've also been using the dotnet-sourcelink tool to test my PDB.
When I build the project locally (by setting CI=true), I see in the MSBuild output that SourceLink has to change the line endings on the files to LF, and I see that the sourcelink,json file has been created in the obj folder. Looking at the CoreCompile target, I see the sourcelink option being passed correctly to csc.exe
However, despite all of that, when I test the PDB with dotnet sourcelink test, it says that there are documents without URLs for all of the source files in the project, and dotnet sourcelink print-json says "Source Link JSON not found in file".
When I run sourcelink print-documents, the hashes it shows there seem to match the versions of the files I can download directly from GitHub using the URL from the sourcelink.json file that's in the obj folder.
At first I was thinking maybe the files were being embedded instead of linked, but looking at the dotnet-sourcelink-git code, I can see that since I see "fixed line endings" entries in my build log, the embedded fallback isn't happening.
Is there anything else I can check to see why this isn't working?

Hi @bording, dotnet sourcelink print-json should print the json file. If it doesn't, the other commands will not work. Can you also try SourceLink.Create.CommandLine, just to try it out?

Hi @bording, dotnet sourcelink print-json should print the json file. If it doesn't, the other commands will not work. Can you also try SourceLink.Create.CommandLine, just to try it out?

@ctaggart SourceLink.Create.CommandLine doesn't work either. I see its output in the MSBuild output, and a sourcelink,json file is created, but the resulting PDB doesn't have any sourcelink info inside it.

@bording Any chance you can reproduce on AppVeyor? The would be the first step, I'd take, would be to branch your PR and create an AppVeyor build. I'm lacking time to help this week.

@ctaggart I have finally figured out what is going on. The project I've been testing this on is also using Fody. Even though Fody has been updated to work with portable PDBs, it looks like Fody is wiping out the sourcelink info.
So it looks like I'll need to follow up with Fody, and likely Mono.Cecil, to see what can be done here.

Awesome! Good to know.


[FEATURE]: Add IF NOT EXISTS when adding new values to existing POSTGRES custom types

Describe what you want
When we add new values to existing POSTGRES custom type DrizzleORM generates SQL like below:
ALTER TYPE "custom_type" ADD VALUE "type3"
But it will be safer if generates SQL like below:
ALTER TYPE "custom_type" ADD VALUE IF NOT EXISTS "type3"

What are the cases, when this value will exists in this type?

It can happen any time. In development, we added a type and its values, later we reverted migration that the type values exist, after days, we decided to add new values to existing type and we try to add an existing value of type in database without knowing and because of that our migration failed.
If IF NOT EXISTS exists in generated SQL, migration won't fail. Because of that this is a need.

Also needed for prototyping with drizzle-kit push.
Related: #1865

Hi I can work on fixing this. Can you assign to me?

What are the cases, when this value will exists in this type?

In addition to https://github.com/drizzle-team/drizzle-orm/issues/1854#issuecomment-1926530161, i happen to manually delete some tables in sqlite because migrate:sqlite and push:sqlite failed after i made some changes to some freshly created tables(empty).
So i had to manually add  IF NOT EXISTS in every CREATE statements inside my migration files

Chiming in on this one as I was digging around trying to figure this out for some hours and also had to manually edit the SQL.
I also noticed that you are adding the UNIQUE constraint to any keys after the table is generated. This is also incredibly error prone. Even if you edit the CREATE statement to include IF NOT EXISTS the SQL driver will run CREATE UNIQUE INDEX table_key_unique ON table (key); after this and error out due to the fact the key exists already and or is already unique.
The create column should ideally have the constraints listed next to each key e.g. key text UNIQUE NOT NULL COLLATE NO CASE etc.
Would be great to get a fix, its a great library but this feels like quite an oversight.


Allows users to set a dependency mirror

Implements dependency-mirrors RFC
A new release of each buildpack will need to be cut with the updated version of packit containing this change.
Reason for this change:
While there is already support for dependency mappings, this adds support for dependency mirrors. The difference is the user would not need to specify a mapping for each individual dependency. Dependency mirrors are more efficient because the user can set one mirror for all or multiple dependencies.
The mirror can be set via an environment variable or a binding of type dependency-mirror. In addition, multiple mirror URLs can be set, a default mirror with extra specific hostname mirrors. For example, if a user wants to download all dependencies from the same place except for dependencies that originated in github (see below). The default mirror can contain a placeholder, {originalHost}, to support mirrors that host many different repositories, which in some cases include the original hostname in the path.  If only specific hostname mirrors are set, the appropriate dependencies will be download via those mirrors while everything else will be downloaded through the original URLs in the buildpack's metadata.
Example mirrors via environment variables:
BP_DEPENDENCY_MIRROR              https://mirror.example.org/{originalHost}
BP_DEPENDENCY_MIRROR_GITHUB_COM   https://mirror.example.org/public-github
BP_DEPENDENCY_MIRROR_NODEJS_ORG   https://mirror.example.org/node-dist

Example of mirrors via binding
/platform
    └── bindings
        └── dependency-mirror
            ├── default                https://mirror.example.org/{originalHost}
            ├── github.com             https://mirror.example.org/public-github
            ├── nodejs.org             https://mirror.example.org/node-dist
            └── type                   dependency-mirror

If a dependency mapping and a dependency mirror is set, an error will be generated. Dependency mappings and mirrors cannot be used simultaneously.
If a binding is used to set dependency mirrors, there can only be one binding of type dependency-mirror, containing multiple Entries for each mirror URL.
If both methods, environment variable(s) and binding, are used to set a dependency mirror, the mirror(s) from the environment variable(s) will be chosen, as specified in the above RFC.

@sophiewigmore I've updated the description. Let me know if you have any questions.

Do we need to add anything to deal with the username/password parts of the URI? and what about the file scheme (https or file). Can the postal.Deliver method deal with both of these things already? Might need tests for these scenarios too if we need to change anything

Do we need to add anything to deal with the username/password parts of the URI?

If a username and password are included in the url, it's captured by url.Parse which is a go-builtin function.

Can the postal.Deliver method deal with both of these things already?

@sophiewigmore  I'll double check and get back to you.

@sophiewigmore cargo.Transport handles both https and file.
I did some adhoc/integration testing with the nginx buildpack, inserting user:pass@ before the beginning of the uri and after the uri scheme. For example,
<EMAIL_ADDRESS>
Usernames and passwords are handled by cargo.Transport implicitly because it does not tamper with the uri before passing it directly to the http request.  (The username and password are put in the Authorization header of the request)
I've updated the PR to address all the above suggestions.


Increase response stream buffer (highwatermark)

Hello everyone!
Today I started developing a streaming feature for my website (using Express on back-end) and I faced a problem: the speed of data transmission is incredibly low. It takes around 10 seconds to send 30MB of data, even with my file stored in a SSD and I'm on localhost.
This is how I am sending data to response:
const stream = fs.createReadStream(path.resolve('..', '..', 'storage', 'file.mp4'), {
  start,
  end
});
stream.pipe(response);

So, I started to researching what could be causing it and found that it's because of a property of the streams named highwatermark, which defines the amouont of RAM used to store data in streaming. The higher the highwatermark value, the fast the data streaming, but also the higher the cost of memory.
I can't just change the highwatermark value for an existing stream instance (like the response object), so I am trying to find a way to do it. I need a higher highwatermark value for streaming this file, what can I do?

Hi @emiliosanches ! I assume you are referring to the response stream's highwatermark, is that correct? That particular stream is not created by Express, instead it is actually created by the underlying Node.js HTTP server (https://nodejs.org/dist/latest-v15.x/docs/api/http.html#http_class_http_server). As far as I know, the Node.js HTTP server does not provide an API to change this value, so Express wouldn't have a method to do so without that being added to Node.js first.

Hi @emiliosanches ! I assume you are referring to the response stream's highwatermark, is that correct? That particular stream is not created by Express, instead it is actually created by the underlying Node.js HTTP server (https://nodejs.org/dist/latest-v15.x/docs/api/http.html#http_class_http_server). As far as I know, the Node.js HTTP server does not provide an API to change this value, so Express wouldn't have a method to do so without that being added to Node.js first.

Thank you @dougwilson. So do you know any alternative to make this streaming faster? Any lib, framework, or another implementation using express?

Hi @emiliosanches I would guess not, as if Node.js itself as a platform does not allow you to alter the highwatermark for the HTTP server, then none of the web frameworks built on Node.js (like Express) would have a way to do this. It would have to get added to Node.js first so the frameworks on Node.js can then utilize it.

@emiliosanches  there aint anyway to solve your issues unless Node itself adds their support.
well thanks for your query.
m also thinking there will be a to increase upload speed without altering the watermark.
i'll try to make my research for you👍👍👍

which defines the amouont of RAM used to store data in streaming

That's not what watermarks mean :) Watermarks are to control "backpressure", it's marks that tell streams when to request more data and how much to buffer before waiting for the consumer to read more data.
For reference streaming 30mb locally on my computer takes ±150ms, so I am really not sure why it takes 10 seconds on yours it's quite surprising. Does it happen without express as well?

See https://nodejs.org/api/stream.html#stream_buffering

const stream = fs.createReadStream(path.resolve('..', '..', 'storage', 'file.mp4'), {
start,
end,
highWaterMark: 1024
});
stream.pipe(response);

Closing as never heard back from OP and unclear what action to take in Express.


POCO Json property attributes not honored on insert Newtonsoft.Json 10.0.2

Maybe everyone else knows this except me.  Seems like it would have been mentioned before now.
My simple POCO classes have attributes such as [JsonProperty("id")] to make sure that case is correct in CosmosDb.
After updating to Newtonsoft.Json 10.0.2 from 9.0.1 these are being ignored during this call:
await _client.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(_database.Id, _collectionId), entity, ro).ConfigureAwait(false);
All POCO properties starting with uppercase are written into the collection with uppercase property names, despite having attributes specifying otherwise.
Workaround is to drop back to 9.0.1 of Newtonsoft.Json.  Works great then.
Other facts:  I am using a repository class to make my Cosmos calls and not using bindings.  I am creating my own generic client, etc. so I can use strongly typed documents.
HTH.

This is expected behavior when using versions of JSON.NET that are newer than the current built in version. In the future we hope that support for binding redirects (tracked under https://github.com/Azure/azure-webjobs-sdk-script/issues/992) will help resolve this.

That makes perfect sense.  It just had never happened to me.  Thanks for responding...

Is there a way to make it work with newtonsoft 12.0.1 version???

@rprajeshpatel Another potential workaround is to use DataMemberAttribute instead of JsonPropertyAttribute

@rprajeshpatel This is an old issue, please file a new issue (and make sure you fill out the details that our issue template asks for)


Hi, I can't check the client's work.

Hi, I can't check the client's work.
Server: test depmax64 High Five
http://www.depmax64.com/forum/index.php?threads/publichnye-testovye-servera.1928/
  Username: "test254",
  Password: "test",
  Ip: "<IP_ADDRESS>",
  ServerId: 1,
  CharSlotIndex: 0

Server: https://www.rpg-club.com/
  Username: "zzgromov",
  Password: "etumonfn",
  Ip: "<IP_ADDRESS>",
  ServerId: 1,
  CharSlotIndex: 0

Please tell me maybe I'm doing something wrong ?
My goal is to get ServerList after authorization

@gromov-io most probably those servers are GG protected.

@gromov-io most probably those servers are GG protected.


Sort tests files for testinfra by filename

With sorted, tests files for testinfra are now executed in the same order, since os.walk return a list in an arbitrary maner.

Bugfix #2845

I'm looking into the py36/37 errors separately. Those don't look like they're part of your PR. But the lint errors definitely are. Can you please run "tox -e lint" locally and then push the resulting changes. It's failing the "black" hook, which should auto-reformat the offending files if you run it locally, commit, and push the changed files.


Maps of builders

Given the following schema:
Panel: {
  title: string
}

Dashboard: {
  panels: {
    [string]: Panel
  }
}

One would expect a DashboardBuilder to be generated with the following option:
func (builder *DashboardBuilder) Panels(panels map[string]cog.Builder[Panel]) {
  // …
}

Without this PR, the following option is generated:
func (builder *DashboardBuilder) Panels(panels map[string]Panel) { // note: map of panels, not panel builders
  // …
}

ToDo:

[x] ~java~ → https://github.com/grafana/cog/issues/684
[x] update go converters
[x] update PHP converters

The diff is somewhat expected.
To avoid breaking BC in the Foundation SDK, we can define a veneer to not generate a builder for the ValueMappingResult object (which doesn't need one anyway as it's a fairly small type)


heroku buildpacks:set should advertise

To improve discoverability, heroku buildpacks:set should write a message to explain how you can use multiple buildpacks:
Did you know you can add multiple buildpacks? Read more...

Doesn't quite feel right to be advertising at that point. And at any rate, discoverability != advertising. Perhaps the help should be clearer? Starting with changing the overall description from singular to plural, or at least buildpack(s):
buildpacks      #  manage the buildpack for an app


Creating a method to make it work for minutes/ seconds only

I am trying to make this work wiht minutes and seconds only, but it doesn't seemt to be laying out correctly - can you give some guidance on where to make changes if I post up my work so far.

I had a go at doing the work myself - I found the easiest approach was just to hide the hours instead of recalculating everyying to makeit flexible and quick enough to use.
https://github.com/ludvigeriksson/LETimeIntervalPicker/pulls
@ludvigeriksson  -  would be great to have your feedback.


we need more compelling motivations/use cases

What's the current stage of this proposal? Or better asking, what do we need to move this to the next step?
I would love to have this in the spec.

I recently read an article about "why async-await is bad", one of the reasons is that: if you want to do the asynchronous tasks efficiently and correctly, you always have to deal with the underlying Promise itself. The author thinks it's some kind of abstraction leaking.
I think this proposal might help reduce the abstraction leaking because when you deal with multiple tasks, you don't have to Promise.all, but await.all. Maybe that's a possible motivation for this proposal.

That's part of my motivation, I think, just more clearly stated.

(If you're interested, the article I read is https://uniqname.medium.com/why-i-avoid-async-await-7be98014b73e )

I have updated the motivation of this proposal


Outlining how you plan on handling highlighting going forward?

So, in the section called: "Automatic Syntax Highlighting", you outline how to setup the syntax highlighting for Cpp.  Previously, all of this was contained in ScintillaNET, was it not?  Are you now saying that all the time we need to define these "style sets" ourselves? Also, there is no longer support for SQL?
I'm sorry, I'm just confused about how things are going to be going forward and what I will be expected to do versus what Scintilla will be handling for me.  Can you please clarify the future of ScintillaNET in this regard?  Thanks.

I've been adding lexer support on an as-needed-basis and you are the first to request SQL. I should be able to get the appropriate constants added to the codebase shortly.
Prior versions of ScintillaNET used a home-grown XML configuration for configuring lexer and style definitions. In some cases (like for SQL) those XML configurations were embedded in the ScintillaNET DLL itself. Internally we would read the XML configuration file and do the work for you of setting the style definitions, lexer enumeration, and keywords as you see documented in the "Automatic Syntax Highlighting".
In my many years of moderating the forum discussions, I found this approach to be a great source of confusion for developers. The XML configuration that was embedded didn't allow for customization, so it was fine if you wanted the colors and fonts we picked out for you, but if you wanted to customize it you would need to create your own XML configuration file and there was no documentation on how to do that. The embedded XML files would also frequently get out of date when new language keywords were added or the lexers themselves were enhanced with new style definitions.
My approach going forward has been to hide less of Scintilla's functionality from the users and instead provide better documentation so they understand the mechanics of what's happening. I think this approach, while seemingly extra work at first glance, will pay dividends later when developers need to troubleshoot or want to explore Scintilla's API deeper.
Thus, I haven't yet, and not sure I plan to, add a single switch for SQL syntax highlighting.
That being said, I'm always interested in developer feedback. If you think there is value in creating some kind of "load style definitions from XML" method, I would be happy to listen. Although, more often than not this tends to be application-specific and developers want these XML files to be handled differently, formatted differently, etc....

Totally understand, just was curious about some clarification on that.  I think it makes sense to allow it to be open-ended.  It can create a lot of different solutions to the same "problem" but allows each developer to tackle it for their purposes.
So, then why do we need the Lexer.Cpp, or Lexer.Python at all if those don't even do anything out of the box (at least they don't appear to on the surface) and we still need to define the keywords and the colors to even get them to work?  What is scintilla.Lexer = Lexer.Cpp accomplishing if not the highlighting?  Is that strictly for specifying the options available in Style.Cpp?

Specifying Scintilla.Lexer = Lexer.Cpp is the instruction to Scintilla to actually 'lex' the document, i.e. break it down into its component parts: numbers, operators, braces, keywords, comments, strings, etc... However, without various style definitions, you wouldn't likely see the difference. If you were to poke around the document using GetStyleAt you would see various styles assigned to spans of text but since style 1, 2, 3, etc... all have the same appearance, you wouldn't visually see the difference. Scintilla intentionally separates the lexing (classification) of a document, from the appearance (display) of a document.
As for the keywords, it's typical of most lexers shipped with Scintilla to separate-out keywords from the actual lexing logic. This is smart because often the syntax of a language doesn't change, but new keywords are added such as the query keywords (select, where, from, orderby, etc...) introduced in C# 3.5. If they were baked-in to the lexer, the Scintilla team (and ScintillaNET) would need to release new versions each time there is a change. The keyword lists are not just for language keywords, but also for user-defined keywords. For example, you might want to use one of the keyword lists in the SQL lexer for user-defined functions. So setting them independently of the lexer makes sense.

Perhaps it would still be worthwhile to add a set of default styles that can be applied for each language? A LexerConfiguration and StyleConfiguration for instance. This would also make it easier for end-user applications to deal with the serialization of, say, an XML file to define them.


[EC-51] EVM System Operations - SELFDESTRUCT

Depends on https://github.com/input-output-hk/etc-client/pull/64 .
Minor changes:

Set log level in tests to WARN

@whysoserious have you addressed the subtleties in YP section 7.1? If not, please remember to add a task to investigate this.

have you addressed the subtleties in YP section 7.1? If not, please remember to add a task to investigate this.

Yes.

LGTM but it is outdated

@adamsmo. @rtkaczyk updated


HDFS-15680. disable broken azure test-units

NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  6s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
36m 35s

trunk passed

+1 :green_heart:
compile
0m 38s

trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
0m 30s

trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

+1 :green_heart:
mvnsite
0m 33s

trunk passed

+1 :green_heart:
shadedclient
55m 56s

branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 34s

trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 26s

trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 32s

the patch passed

+1 :green_heart:
compile
0m 33s

the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
0m 33s

the patch passed

+1 :green_heart:
compile
0m 24s

the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

+1 :green_heart:
javac
0m 24s

the patch passed

+1 :green_heart:
mvnsite
0m 28s

the patch passed

+1 :green_heart:
whitespace
0m  0s

The patch has no whitespace issues.

+1 :green_heart:
xml
0m  2s

The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
18m 53s

patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 30s

the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 29s

the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

_ Other Tests _

+1 :green_heart:
unit
1m 25s

hadoop-azure in the patch passed.

+1 :green_heart:
asflicense
0m 38s

The patch does not generate ASF License warnings.

84m 52s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2457/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2457

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml

uname
Linux 06a00ae68cd1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 375900049cc

Default Java
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2457/1/testReport/

Max. process+thread count
627 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2457/1/console

versions
git=2.17.1 maven=3.6.0

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.


bioconda

need to add click as prerequisite

same, tried uninstalling and installing click using conda, still didn't work
Erics-Mac-Pro:~ cryomics$ atlas -h
Traceback (most recent call last):
File "/usr/local/bin/atlas", line 7, in
from atlas.atlas import cli
File "/usr/local/lib/python3.6/site-packages/atlas/atlas.py", line 1, in
import click
ModuleNotFoundError: No module named 'click'

I tried installing in linux and didn't have this problem. If ATLAS is not compatible with macos you should put that up front

The only portion not compatible with macOS is genomic binning and is due to maxbin2 compatibility issues. We're considering other binning approaches to get around this issue. click is absolutely compatible with macOS, so it's likely an issue with your PATH.


fix(web-server): ignore future socket errors

Karma sometimes reports uncaught exceptions (ECONNRESET) which causes it to exit the process with a non-zero exit code.
Fixes #3295

:white_check_mark: Build karma 2460 completed (commit https://github.com/karma-runner/karma/commit/6b00be646c by @mgroenhoff)

@googlebot I signed it!

:white_check_mark: Build karma 2783 completed (commit https://github.com/karma-runner/karma/commit/8f39f90852 by @mgroenhoff)

Any update on this? Is anyone who knows the code willing to do the required review?

@mgroenhoff do we still need to merge this CL?


resource-manager: Ignore the ToBeDeletedByClusterAutoscaler taint in the system-components-config webhook

How to categorize this PR?

/area quality ops-productivity
/kind bug
What this PR does / why we need it:
See https://github.com/gardener/gardener/issues/8154
Which issue(s) this PR fixes:
Fixes https://github.com/gardener/gardener/issues/8154
Special notes for your reviewer:
Release note:

`gardener-resource-manager`'s `system-components-config` webhook no longer adds the toleration for the `ToBeDeletedByClusterAutoscaler` taint to system components in shoot clusters. The `ToBeDeletedByClusterAutoscaler` taint is maintained by the `cluster-autoscaler`. This was breaking `cluster-autoscaler`'s drain mechanism when scaling down an under-utilized node. It was causing just evicted system components from to be deleted node to be scheduled again on the to be deleted node.

I need to add test.

/approve

/retest

/cherry-pick release-v1.74


[BUG] Memory Leak | JavaScript heap out of memory

We had a memory leak on two different services running ts.ed framework on node 12.
After some investigating, we discovered the same error logs on both of the services:
FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

[18:0x39f9c20] 429514310 ms: Scavenge 912.9 (989.3) -> 899.1 (989.3) MB, 4.8 / 0.1 ms  (average mu = 0.999, current mu = 0.999) allocation failure
[18:0x39f9c20] 429516411 ms: Scavenge 914.1 (989.3) -> 900.3 (989.3) MB, 4.8 / 0.1 ms  (average mu = 0.999, current mu = 0.999) allocation failure
[18:0x39f9c20] 429518553 ms: Scavenge 915.4 (989.3) -> 901.5 (989.3) MB, 4.8 / 0.1 ms  (average mu = 0.999, current mu = 0.999) allocation failure

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x13725d9]
Security context: 0x3be9081808a1 <JSObject>
    1: /* anonymous */ [0x20e9a7407ac9] [/usr/src/app/node_modules/@tsed/common/lib/server/utils/contextMiddleware.js:~13] [pc=0x391e14c502e9](this=0x0e2eeab4fc79 <Object map = 0x2e9f2ac42191>)
    2: next [0x3be9081a3651](this=0x20e9a7407ea9 <JSGenerator>)
    3: /* anonymous */(aka /* anonymous */) [0xff65d1e9041]<EMAIL_ADDRESS>

Version: 5.28.0 and 5.33.0
Packages:

"@tsed/ajv": "5.33.0",
"@tsed/common": "5.33.0",
"@tsed/core": "5.33.0",
"@tsed/di": "5.33.0",
"@tsed/swagger": "5.33.0",

and:
"@tsed/common": "5.28.0",
"@tsed/core": "5.28.0",
"@tsed/di": "5.28.0",
"@tsed/swagger": "5.28.0",

We are checking if this memory leak was fixed on version v5.41.2.

Hello @NitzanHardonVonage
Have some questions to investigate correctly your issue:

Did you use request.ctx.logger in your code ?
Have you changed the logger settings?

@ServerSettings({
   logger: {
       requestFields: ["reqId", "method", "url", "headers", "body", "query","params", "duration"]
   }
})

how I can reproduce it?

if it's possible, can you give me an access to your project? It'll be faster for me to understand and solve your problem ;)?
See you
Romain

@Romakita
I created a simple ts.ed API and I reproduced the memory leak.  I ran this API on an AWS Fargate service and I sent ~340 requests per second using locust (load test tool).
container settings:
memory (MiB)512
CPU (unit)256

https://github.com/NitzanHardonVonage/tsed-mem-leak

Thanks for the repo example :).
I published a fix too reduce the potential memory leaks on the createContext middleware. Can test it pls (latest version).
I’ll checkout your example and investigate with the same tool :). I’ll come back to you when I found the reason.
See you

Adding my locust code to save you some time :)
import os
import random
from locust import HttpLocust, TaskSet, runners, task, web
from locust_check_results import run_validation_check

def get_id():
    return random.randint(1,100001)

names = ['string', 'name', 'nitzan', 'hardon', 'memory', 'leak' 'aaaa', 'bbbb', 'cccc', 'asdfasfsa', 'iy', 'nihfq', 'aga', 'jasdfh', 'csscs', 'dddd']
def get_name():
    return random.choice(names)

class LocustSet(TaskSet):

    def on_start(self):
        pass

    def on_stop(self):
        pass

    @task(1)
    def get_command(self):
        id = get_id()
        self.client.get("/ctrl/%i" % id, name="/ctrl/[id]")

    @task(1)
    def post_command(self):
        i = get_id()
        body = {'name': get_name()}

        self.client.post("/ctrl/%i" % i, name="/ctrl/[id]", json=body)

    @task(1)
    def put_command(self):
        id = get_id()
        name = get_name()
        self.client.put("/ctrl/%i?name=%s" % (id, name), name="/ctrl/[id]?name=[name]")

    @task(1)
    def delete_command(self):
        id = get_id()
        self.client.delete("/ctrl/%i" % id, name="/ctrl/[id]")

class MemLeakLocust(HttpLocust):
    host = "http://localhost:3000"
    task_set = globals()[os.getenv('TASK_SET', 'LocustSet')]
    min_wait = 0
    max_wait = 0

def get_common_headers(method, path):
    common_headers = {
        'Accept': 'application/json',
        'Content-Type': 'application/json'
    }

    return common_headers

As a response to your PR on the dummy project repository,
It looks like there is still a memory leak, even though I removed all the fields from the required fields array.
I ran it once again and I took a snapshot a few minutes after I started the locust test.
Adding the snapshots' results:

In your opinion, is there any chance that the callback function's closure of the process variable in HandlerBuilders class is never released from the memory by the GC? - Same thing for the anonymous function incontextMiddleware.

Hello @NitzanHardonVonage
I found the problem. The fix is almost done ;)
For the rxjs, I used this to transform an Observable returned by the endpoint to a Promise.
I used rxjs also to handle params. I see where is the problem. I try avoid this.

@Romakita great I'm glad to hear that!
Keep me updated on this :)

Hello @NitzanHardonVonage
I continue to investigate on the HandlerBuilder. The problem seems to come from the code using rxjs (as you figured out ;) ). Problem is to find the reason and how I can fix that without change all the code :(
See you

Other important point:
This code create a huge memory leak:
if (this.httpServer) {
      this.httpServer.keepAliveTimeout = 600000;
    }
    if (this.httpsServer) {
      this.httpsServer.keepAliveTimeout = 600000;
    }

I reproduced the memomy leak with a pure express.js server:
const express = require("express");
const fs = require("fs");
const chalk = require("chalk");
const onFinished = require("on-finished");
const bodyParser = require("body-parser");
const compression = require("compression");
const cookieParser = require("cookie-parser");
const cors = require("cors");
const methodOverride = require("method-override");
const uuidv4 = require("uuid/v4");
const http = require("http");

const app = express();
const httpServer = http.createServer(app);

httpServer.keepAliveTimeout = 60000;

app.use(bodyParser.json());
app.use(cookieParser());
app.use(compression({}));
app.use(cors());
app.use(methodOverride());
app.use(bodyParser.urlencoded({
  extended: true
}));

function listen(http, port, address) {
  const promise = new Promise((resolve, reject) => {
    http.on("listening", resolve).on("error", reject);
  }).then(() => {
    const port = http.address().port;
    console.log("==>Port:", port);
  });

  http.listen(port, address);

  return promise;
}

const emitter = {
  async $emit(event, request, response) {
    // console.log("Event", event, request.id);
  }
};

app.use((request, response, next) => {
  const id = uuidv4().replace(/-/gi, "");
  request.id = id;
  request.ctx = {
    id
  };

  onFinished(response, (err, response) => {
    emitter.$emit("onResponse", request, response);
  });

  emitter.$emit("onRequest", request, response);

  next();
});
//
// app.get("/ctrl/:id", (req, res) => {
//   const {params: {id}} = req;
//
//   res.json({
//     id,
//     name: `${id} name`
//   });
// });
//
// app.post("/ctrl/:id", (req, res) => {
//   const {params: {id}, body: {name}} = req;
//
//   res.json({
//     id,
//     name
//   });
// });
//
// app.put("/ctrl/:id", (req, res) => {
//   const {params: {id}, body: {name}} = req;
//
//   res.json({
//     id,
//     name
//   });
// });
//
// app.delete("/ctrl/:id", (req, res) => {
//   const {params: {id}} = req;
//
//   res.json({
//     id,
//     name: `${id} name`
//   });
// });

listen(httpServer, 3000);

//------------------------------
const stats = [];

const write = () => {
  const data = JSON.stringify(stats);
  fs.writeFile(__dirname + "/../../stats.json", data, (err) => {
    if (err) {
      console.log(err);
    } else {
      console.log("\nSaved stats to stats.json");
    }
    process.exit();
  });
};

let last = 0;
const onTick = () => {
  try {
    global.gc();
  } catch (e) {
    console.log("You must run program with 'node --expose-gc index.js' or 'npm start'");
    process.exit();
  }
  const heapUsed = process.memoryUsage().heapUsed;
  stats.push(heapUsed);
  if (last) {
    console.log("Memory heap => ", heapUsed, heapUsed - last > 0 ? chalk.red("+") : chalk.green("-"));
  }
  last = heapUsed;
};
setInterval(onTick, 500);
setTimeout(write, 20000);

Now here the graph with contextMiddleware + HandlerBuilder (without keepalive):

We can see a memory leaks but isn't huge.
Now without handlerBuilder:

I'll check what you said about the keepAliveTimeout and see it' effect on the memory.
I'm trying to understand if you still think that there is another problem coming from the code that using rxjs, or the only problem is with the keep-alive timeout?

'm testing it with a new value but this value is greater than 60 seconds. The reason for it is that we are running our services as ECS tasks and we're using AWS load balancer. The load balancer requires us to set the keep-alive timeout to a value greater than 60 seconds to avoid 504 error responses: https://aws.amazon.com/premiumsupport/knowledge-center/504-error-classic/

I understand the usage of the keepAliveTimeout option.
We have two problem :

one with the keepAliveTimeout which increase the memory and cpu usage. I see this topics (https://shuheikagawa.com/blog/2019/04/25/keep-alive-timeout/). Maybe adding
The rxjs usage in handler buildler create multiple Subject and Observable per endpoint. I consume a huge memory for nothing. I have to optimize this part. This is my main point for this week :)

Hey @Romakita
is there any new update about the memory-leak?

Yes. I removed the rxjs on handler builder. I’m on the unit test.
Remove rxjs and use a native approach take a long time ^^
I hope my work will finish today or tommorow

Branch is ready here https://github.com/TypedProject/tsed/pull/770

I runned load testing with the new implementation:

The consumed memory is better but the leak is always here... (I'm not sure...)

I'll merge the current work to let you run your own load test. Maybe the actual version is better in your case :)

V5.44.0 is released :)

@Romakita I ran the same load test, still got the same exception

Is a bad news. I removed rxjs and refactored the code to avoid all potential memory leak. I clean after each request all data.
Honestly I need help to a find where is the problem and Understand chrome dev tools result isn’t easy (it don’t help me to fix the problem).
The code responsible to build handlers is here:
https://github.com/TypedProject/tsed/tree/production/packages/common/src/platform
https://github.com/TypedProject/tsed/blob/production/packages/common/src/platform/services/PlatformHandler.ts#L60
Can you review the code maybe you’ll see something.

Yes of course, I'll dig into it tomorrow, hopefully I could find something.


How create dataset format FUNSD ?

example format:
        {
        "form": [
        {
            "id": 0,
            "text": "Registration No.",
            "box": [94,169,191,186],
            "linking": [
                [0,1]
            ],
            "label": "question",
            "words": [
                {
                    "text": "Registration",
                    "box": [94,169,168,186]
                },
                {
                    "text": "No.",
                    "box": [170,169,191,183]
                }
            ]
        },
        {
            "id": 1,
            "text": "533",
            "box": [209,169,236,182],
            "label": "answer",
            "words": [
                {
                    "box": [209,169,236,182
                    ],
                    "text": "533"
                }
            ],
            "linking": [
                [0,1]
            ]
        }
    ]
    }

You need to construct a labeling config first, then it will be much easier to understand what you need.

It looks like standard OCR task, check this labeling config please, will it work for you?
<View>
  <Image name="image" value="$ocr"/>

  <Labels name="label" toName="image">
    <Label value="Text" background="green"/>
    <Label value="Handwriting" background="blue"/>
  </Labels>

  <Rectangle name="bbox" toName="image" strokeWidth="3"/>
  <Polygon name="poly" toName="image" strokeWidth="3"/>

  <TextArea name="transcription" toName="image"
            editable="true"
            perRegion="true"
            required="true"
            maxSubmissions="1"
            rows="5"
            placeholder="Recognized Text"
            displayMode="region-list"
            />
</View>

It looks like a standard OCR task, check this labeling config please, will it work for you?

It doesn't seem to solve the link issue.
this part dataset
            "linking": [
                [0,1]
            ]

It this link between bboxes?

It this link between bboxes?

yes
this example

@DimIsaev did you solve this in the end? Or you used another tool?

@DimIsaev did you solve this in the end? Or you used another tool?

project  paused
I'll be back later

As workaround you can follow this way:

you can go to console where you run LS and call there:

label-studio shell

it will run LS shell (Django shell_plus) and you will be able to execute Django ORM commands

In shell plus:

from tasks.serializers import TaskWithAnnotationsSerializer

tasks = Task.objects.filter(project=<ID>)
export_json_data = TaskWithAnnotationsSerializer(tasks, many=True).data

Optionally:

import json
with open('export.json', 'w') as f:
  json.dump(export_json_data, f)

I am also having to label something like this.
The labeling tool should use auto OCR so that it becomes easy to work for these cases.

I am also having to label something like this.
The labeling tool should use auto OCR so that it becomes easy to work for these cases.

They added a new feature for this https://github.com/heartexlabs/label-studio-ml-backend/tree/master/label_studio_ml/examples/tesseract

@makseq can we connect on slack? I have an urgent query to be solved. Related to the same issue regarding output format for OCR into funsd dataset.

Check this PR: https://github.com/heartexlabs/label-studio-converter/pull/127
Unfortunately we can't make 100% compatible conversion to FUNSD format, because it has root bboxes and words bboxes and LS doesn't build root bboxes automatically. So, this converter creates one root bbox with one word inside of it.

I am also having to label something like this.
The labeling tool should use auto OCR so that it becomes easy to work for these cases.

They added a new feature for this https://github.com/heartexlabs/label-studio-ml-backend/tree/master/label_studio_ml/examples/tesseract

Hi, @WaterKnight1998, I had a try by following readme. In step 4, setup Tesseract ML backend:

pip install -r label_studio_ml/examples/tesseract/requirements.txt
label-studio-ml init my-ml-backend --from label_studio_ml/examples/tesseract/ner_ml_backend.py --force
label-studio-ml start my-ml-backend -d -p=9090 --debug

But I cannot find the file label_studio_ml/examples/tesseract/ner_ml_backend.py.

The ner_ml_backend.py is now the tesseract.py file boss
On Tue, 27 Sep 2022 at 8:58 AM, wujushan @.***> wrote:

I am also having to label something like this.
The labeling tool should use auto OCR so that it becomes easy to work for
these cases.
They added a new feature for this
https://github.com/heartexlabs/label-studio-ml-backend/tree/master/label_studio_ml/examples/tesseract
Hi, @WaterKnight1998 https://github.com/WaterKnight1998, I had a try by
following readme. In step 4, setup Tesseract ML backend:
pip install -r label_studio_ml/examples/tesseract/requirements.txt
label-studio-ml init my-ml-backend --from
label_studio_ml/examples/tesseract/ner_ml_backend.py --force
label-studio-ml start my-ml-backend -d -p=9090 --debug
But I cannot find the file
label_studio_ml/examples/tesseract/ner_ml_backend.py.
—
Reply to this email directly, view it on GitHub
https://github.com/heartexlabs/label-studio/issues/2634#issuecomment-1258920597,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALUQ7EAZYTOWAWVMANFQRPLWAJSU3ANCNFSM53BYNWUQ
.
You are receiving this because you commented.Message ID:
@.***>

@DimIsaev, @wujushan, I wanted to suggest another possibility here too. I'm working on an open source tool that can export directly in FUNSD. It's called OpenContracts. Check it out. Feedback welcome!

Check this PR: HumanSignal/label-studio-converter#127. Now you can export to FUNSD using this script.
Unfortunately we can't make 100% compatible conversion to FUNSD format, because it has root bboxes and words bboxes and LS doesn't build root bboxes automatically. So, this converter creates one root bbox with one word inside of it.

this script doesn't support the linking[] how to do it?


rank-one matrix unary operator

-(::RankOneMatrix) was left undefined, which did cast it to a dense matrix by default

Codecov Report

Merging #221 (f82cef4) into master (6ef4507) will decrease coverage by 0.05%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #221      +/-   ##
==========================================
- Coverage   68.79%   68.74%   -0.06%
==========================================
  Files          14       13       -1
  Lines        1689     1686       -3
==========================================
- Hits         1162     1159       -3
  Misses        527      527

Impacted Files
Coverage Δ

src/types.jl
65.59% <100.00%> (+0.37%)
:arrow_up:

src/utils.jl
53.37% <0.00%> (-0.28%)
:arrow_down:

src/oracles.jl
86.59% <0.00%> (-0.14%)
:arrow_down:

src/FrankWolfe.jl

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ef4507...f82cef4. Read the comment docs.


Unescaped line and paragraph sep in older lang versions with UTF-8

Previously, if you specify UTF-8 and have unicode line or paragraph separators in strings, those characters would be escaped.
Here is an example of the previous behavior:
$ java -jar target/closure-compiler-1.0-SNAPSHOT.jar --charset UTF-8 --language_out ECMASCRIPT5
The compiler is waiting for input via stdin.
alert("\u0061\u2028\u2029\u0062");
<^D pressed here>
alert("a\u2028\u2029b");

With https://github.com/google/closure-compiler/commit/a6aeaab3ed2cd5f4a28344dedb3d346b8f99cbc1 these two characters are allowed to be emitted unescaped in the output JavaScript.  Here is an example with that commit:
$ java -jar target/closure-compiler-1.0-SNAPSHOT.jar --charset UTF-8 --language_out ECMASCRIPT5
The compiler is waiting for input via stdin.
alert("\u0061\u2028\u2029\u0062");
<^D pressed here>
alert("a<U+2028><U+2029>b");

Note that, in the above <U+2028> and <U+2029> are the actual unescaped characters (I chose to write them this way in this issue so that they are visible.)
It seem that the intent of https://github.com/google/closure-compiler/commit/a6aeaab3ed2cd5f4a28344dedb3d346b8f99cbc1 is to allow this unescaping when language out is ES_NEXT, but the example above illustrates that the unescaped versions of these two characters appears even for earlier language out versions.

Created internal issue http://b/127294275

Will take a look!


Add Tests

Functions Requiring Tests in main.go

[ ] updateFileTags
[ ] searchFiles
[ ] getAvif
[x] loadConfig
[x] loadFileMetadata
[x] writeFileMetadata
[x] buildLink
[x] saveFile
[x] listFiles

Functions Requiring Tests in resizer.go

[x] writeNewImg
[x] createAltSizes

Reopening issue to track tests that have yet to be implemented.


Allow suppression of auto-open behaviour for bottom panel

I'd love to see a setting to suppress any and all auto-open behaviour of this panel:

Rationale
This comes from years of using code as it has been evolving. My "clue" for needing this is that I find myself closing or going after every possible way to close this part of the UI. Especially after using keyboard shortcuts that won't necessarily require me to digest any of the information presented.
The two most common scenarios for me are when code is automatically started as my git commit message editor (from WSL2) and when I do a build (ctrl-shift-b).  Especially in the WSL2 scenario where the panel actually steals focus, requiring me to click in the editor window before authoring my commit message!
While in the case of failed builds, I will want to see information, after a successful build, having nominal "this will stay open" information presented to me takes away my attention and forces me to take an action to close the panel to get back to my regular "focused/unobstructed" layout.
As for focus-stealing, I think these need to be chased down.  Unless the open of the panel was as a direct result of user intention (ctrl-`, etc), this panel should never steal focus.
To make all of this conceptually easier, I do not believe code should care about whether tasks or other events that pop this panel up are successful or not.  Just give me a way to say "never pop this open, I will open it myself when I need it".

It is a fair feature request to have this setting, thus leaving this open.
However apart from that we should still find all places that un-necesserily open the bottom panel and file issues against tham. @atrauzzi can you please file independet issue for when panel is opened and it should not be in your opinion so that owners can tackle those independently?
Thanks

Hi.
I totally agree. A "Don't automatically open the bottom panel" setting is needed imho. I personally see this behavior exactly like an unwanted popup thrown to my face. If we can't fine tune it, at least I'd love to be able to nuke it.

That is such a terrible way to prioritize work.


how to remove element in a scaledbloomfilter?

I want to remove some elements, so next time add(them) will return false.
is there inner function to do this?

According to my quick search result, a basic Bloom Filter doesn't support remove operation because adding an element is irreversible in Bloom filter.
For further information, see 1) the wiki page explaining how Bloom Filter works and 2) a StackExchange page discussing how to improve Bloom Filter to make it support delete operation, like using a second bloom filter to store deleted elements, or using Counting Bloom Filter or 3) search some new data structure like "Cuckoo Filter"


question: Persistent Subscription Docs vs. Code

// PersistentAllSubscriptionOptions options for most of the persistent subscription requests.
type PersistentAllSubscriptionOptions struct {
	...
	// Requires the request to be performed by the leader of the cluster.
	RequiresLeader bool
}

I am just curious, why is there an option for RequiresLeader when the docs state the following:

Persistent subscriptions only run on the Leader node. This means that more pressure will be put on the Leader node, and there is no option to balance the load to a follower like with a Catch-up subscription.

Is there a use case for this option or should I default to setting it to true?

This is a fair question. As the documentation states it, it's not possible to run persistent subscription on anything than the leader node. All write operations and the subscribe don't need that option indeed. We should remove it from the code to avoid confusion.

I would be happy to try and make this change myself if you’re interested in
having more contributors.
On Thu, Jun 6, 2024 at 12:48 AM Yo Eight @.***> wrote:

This is a fair question. As the documentation states it, it's not possible
to run persistent subscription on anything than the leader node. All write
operations and the subscribe don't need that option indeed. We should
remove it from the code to avoid confusion.
—
Reply to this email directly, view it on GitHub
https://github.com/EventStore/EventStore-Client-Go/issues/176#issuecomment-2150401059,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BD3VQPC4KCYYE6LIM7ZXMX3ZF4XNVAVCNFSM6AAAAABIFDMLKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJQGQYDCMBVHE
.
You are receiving this because you authored the thread.Message ID:
@.***>

Please have a go if you want. Will be glad to approve your PR!

Cool, I will give it a shot this weekend.

Hey @seanppayne

I do not have permission to push code to a branch here if I clone the repo. Should I fork instead?

Yes, you need to fork the repo first then open a pull request on the branch  you used to write your patch

I get the following error in make test. Do I need to adjust some env vars or do I need some credentials to access this image repository?

You shouldn't have to anything more. Could it be you need to run docker while being root on your machine? I personally always run my docker containers rootless. In any case, don't stress over it, just write your patch, the CI will take care of the testing part.
Thanks for your contribution!


Unnecessary code tagged strings within blockquote tags

Access MultiDimensional Arrays With Indexes and Understand String Immutability have text in code tags within a blockquote already. The code tagged strings need to be removed. You can see them because they are highlighted again within the code block.
Access MultiDimensional Arrays with Indexes:

Understand String Immutability:

Agreed, these just just be in the block quote.


Add an unless aborting? guard to BROADCAST-LOCK

My interpretation is that we never used more than one BROADCAST-LOCK in a script, and so never ran into this.


feat(RHTAPWATCH-760): Smee client on DS cluster

Install Gosmee in client mode on the DS staging cluster.

/approve
/lgtm


Add a full wg peers resync, default every hour

Seems redundant. If anything we need to understand more the wathcer's resync period and maybe loosen it a bit or just work with defaults: https://github.com/kubernetes-sigs/controller-runtime/issues/521#issuecomment-518457302


Give up on openssl, use gnupg

I couldn't sort out how to make openssl continue working. Using the suggested -pbkdf2 and -iter options to encrypt didn't produce a file that I could decrypt. Even on the system that did the encryption. ¯_(ツ)_/¯

Again, there's nothing to be learned from running the PR, so don't bother.


Add Github Action for testing on Windows

See https://github.com/Senzing/template-go/issues/90

Pull requests:

https://github.com/Senzing/go-queueing/pull/41


Does not work with React 16.x.x - PropTypes

Some of your libs i.e Chart.js have PropTypes taken out of react library - unfortunately they've been removed from React 16 and you need to add them as an additional package.
Snippet from Chart.js file /
var _react = require("react");
...
Chart.contextTypes = {
chartConfig: _react.PropTypes.array,
subscribe: _react.PropTypes.func.isRequired,
unsubscribe: _react.PropTypes.func.isRequired
};
^ This throws an error saying that PropTypes is undefined
https://reactjs.org/docs/typechecking-with-proptypes.html

0.7 beta uses the prop-types package https://github.com/rrag/react-stockcharts/blob/master/src/lib/Chart.js#L3
If you are using 0.6 then you might have to monkey patch the proptypes. Upgrade to 0.7 beta if you can. The only reason it is still in beta is the interactive indicators are changing a lot.