Compiled Lambda Expressions Hurt Performance on UWP

Hi,
I've been struggling with (what I consider to be) unsatisfactory performance with Json.Net in UWP projects. After much research and testing, and some discussions with a developer on the .Net Native team, it would appear the problem is the use of compiled lambda expressions within Json.Net. This is most notable on slower ARM based devices, such as a Windows Phone, but can be seen on any .Net Native build.
On the desktop frameworks (and possibly others), the compiled expressions for setting properties/fields and creating type instances etc. do provide a significant performance improvement barring the cost of the overhead of creating/compiling the expressions in the first place.
However, in UWP/.Net Native projects it appears there are two issues that mean compiled lambdas actually run slower than the equivalent reflection code (i.e propertyInfo.SetValue vs. using the compiled setter expression).

A lot of internal caching of reflection types has been removed within the .Net Native/UWP runtimes, so reflection is more expensive there than on desktop. This can largely be avoiding by implementing your own caching of propertyinfo/constructorinfo and similar objects, which Json.Net already does.
Apparently, most of the stuff System.Linq.Expressions is not well supported, and in fact expression.Compile is basically a no-op (so I'm told by the Microsoft dev). It was suggested to me that the .Net Native team actually considered making these throw 'platformnotsupportedexceptions' but eventually decided slow was better than no support at all. The types in the Expressions namespace don't implement their own caching either, so using the compiled expressions actually results in worse performance than the normal reflection calls when you already have your own caching in place.

I realise it would be a non-trivial amount of work, and it probably needs testing to prove the performance benefits under UWP, but is there any chance at all there could be a UWP specific binary that avoided the compiled expressions for improved perf?
I know that is phrased as a question, but it isn't really suitable for Stackoverflow, and while I'm asking about a potential fix I'm really reporting the problem that an existing optimisation hurts on a different platform. My apologies if this is still the wrong place to post this, but it seemed like it met the criteria for 'bug report' to me.
Thanks.

Json.NET already supports switching between methods of doing reflection. The simplest solution would be a flag somewhere on the .NET Core runtime that could be tested to see whether dynamic compilation (IL emit) is available. If it is true then use IL emit or System.Linq.Expressions for reflection, if it isn't then use traditional MemberInfo reflection.

Thanks for the excellent and fast response. I do not know the answer to your question myself, but I will ask my Microsoft contacts and see if they can suggest something.

Could Json.NET have a "build" step option where the compiled expressions are emitted to an assembly? Then that assembly would work with UWP's AOT. Just an idea...

Not sure how that would work. There is no "emit" method in the UWP framework, and the expressions aren't known until runtime when the code executes against types (either type objects or generic type arguments passed to methods). Maybe I'm missing something or don't understand the suggestion? Seems quite a complicated solution too.

Re the referenced corefx issue... I am unsure if Xamarin.iOS has the same problem. Yes it uses AoT so the expressions probably aren't compiled because they just can't, but part of the UWP issue is that internal caching of some reflection objects was removed to reduce memory footprint. The Xamarin team may not have made the same decision, and I have not been able to test on that platform to compare relative performance. It's possible the Xamarin.iOS implementation is comparable to just using reflection anyway, depending on how it was implemented.

You'd need to know the types upfront at build time (though I'd assume you do as you'd be serialising and deserilising to them). Though would be more of a extension to Json.NET e.g. output strongly typed serialiser and deserialiser for types X, W, Z to assembly as part of a pre-build step. (Like bundling and minification is).

Could work, but as you say is probably an extension rather than an update to the library.
Re the existing possible solutions, since we are yet to hear back from MS about how to determine if expressions are actually compiled, could we consider one of the following;

A property on the SerializerSettings class that allows the consumer of the library to explicitly specify whether to use reflection/expressions/emit? Could default to an 'auto' value for current behaviour. The benefit here is that as other frameworks develop in the future, or the frameworks potentially change perf characteristics between versions, the consumer of the library could make the best choice for themselves even if Json.Net temporarily doesn't know the best option itself?

If we can't detect if expressions are compiled easily, would we settle for detecting the platform we're running on and making an assumption about the behaviour? i.e if (and I do not yet know how) we could detect we're running in a .Net Native/Xamarin.iOS/Unity environment/CLR we avoid expressions, and for all others we use the existing behaviour? Not as clean/accurate as knowing whether the expressions are compiled but if the logic is wrapped up in a function it would be easy to change to a better flag later if one becomes available.

I'm not doing a build step. I simply want a way to determine whether a platform is AOT or not so the optimal way of doing reflection is used.

I have been cced on an email between .Net Native team members suggesting they are going to use your corefx issue to track adding a suitable flag for this, not sure if/when that might happen.
Are either of the two suggestions I made viable as alternate solutions?

The problem with a UWP only binary is it only improves UWP. There would need to be one for Xamarin and Unity as well. And eventually compiling to native in .NET Core will be a thing and a platform specific binary won't help at all in that case. I want a solution that works everywhere.

Hi James,
I think you misunderstood what I tried to say. I agree a UWP specific binary is uncool.
One suggestion is just to allow the client to explicitly state which mechanism to use. Not ideal, but does give the consumer ultimate control (with the default being the current 'automatic' behaviour).
The other suggestion is not a specific binary, but to attempt to detect the runtime environment (on the basis that there probably isn't a current flag that determines the behaviour of expression.compile). For example, I was hoping we could use Environment.Version or something similar to detect that we were running on a .Net Native build. Unfortunately that doesn't seem possible and I haven't yet found something that does work (am still looking).

The second option is what I want. I feel there must be some way, somewhere to find out if the app is native.

Haven't tested it, but try something like:
internal static class NativeUtils
{
    private class NativeTest<T> { }

    public static bool IsRunningOnNative()
    {
            return null == Type.GetType("Newtonsoft.Json.Utilities.NativeUtils+NativeTest`1", false);
    }
}

This is based on assumption that metadata for open generic types is not included in the .NET Native binaries unless explicitly specified.

I develop a music player app for the Windows Store and used Newtonsoft.Json for the serialization of playlists. The difference in performance between my Release build which has .Net Native enabled and my Debug build is huge. So here are my results from testing with a large playlist with 16000 songs:
Serializing with Newtonsoft.Json:
DEBUG: 0,760678 seconds
RELEASE: 7,9427199 seconds
Deserializing with Newtonsoft.Json:
DEBUG: 1,7782815 seconds
RELEASE: 10,6551608 seconds
As a comparison I tested the same playlist with ServiceStack.Text Json Serializer:
Serializing with ServiceStack.Text Json:
DEBUG: 0,9823664 seconds
RELEASE: 9.0322984 seconds
Deserializing with ServiceStack.Text Json:
DEBUG: 0,9636399 seconds
RELEASE: 1.1934371 seconds
Conclusion: Please do something about this issue because as of today Newtonsoft.Json is not usable in my app.

For those struggling with these perf issues, writing your own code that deserialises using a jsonreader instance is generally the way to get best perf anyway and eliminates this issue (http://www.newtonsoft.com/json/help/html/Performance.htm). If you have a lot of types and can't be bothered with that, a custom converter that uses cached reflection objects can be 25-50% faster (in my completely unscientific testing). A sample (not production code) here;
https://gist.github.com/Yortw/e568b363405577d55d1b68cad8279851
Note, this will be slower in debug builds, but is faster in release builds on everything I've tested. YMMV.


Update not working?

I'm trying your new update, using the tunnel broker. I have no idea what that is, but I did as you say in the readme and after installation everythinig seems fine on the server.
But on the netflix site, I get "Internet Connection Problem", which is good, because it doesn't say " Proxy error" anymore.
I devtools, I see a lot of calls to
https://ipv6_1-lagg0-c068.1.iad001.ix.nflxvideo.net/range/0-18101?..
timing out.
Also, you should probably set something like this up as some kind of paid service :)

Try rebuilding on another VPS (destroy your current one).
Otherwise, see if you can troubleshoot from the command line first..
dig ipv6_1-lagg0-c068.1.iad001.ix.nflxvideo.net @<your-vps-ip>
curl -v -H 'host: ipv6_1-lagg0-c068.1.iad001.ix.nflxvideo.net' <your-vps-ip>

I'm having the same issues here too @nixxle – everything seems to check out, and both of those two tests seem to return positive results @ab77. It's working on DO though it seems? Must be a configuration issue of some kind then... Maybe I should spin up an instance there and see if that makes any difference. Just liking my current VPS provider, is all, but not much use if the actual end result isn't positive.

What provider are you using? I am happy to test it out..

same issue here on vultr, tried debian/ubuntu hosts

Just confirming – we should be configuring VPS to be without IPv6 support now, or still with support just no local addresses configured? If I build without IPv6, and the build.sh script says IPV6=Disabled, the system seems to work and the error disappears... but Netflix comes back with proxy detected instead.

Just built from scratch with a new DO droplet – same error. Seems like it might be an issue with the code or perhaps the hurricane tunnel? I'm leaning towards the tunnel, as everything was working before that change was required. Perhaps we're missing something at that end? (Or perhaps they've made a move to block this traffic?)

I just tested on Vultr and it works fine for me.
If you get IPv6=Disabled, then your tunnel isn't working. After you modify your interfaces files and before you try to deploy the proxy, just run ifup he-ipv6, followed by ping6 google.com.
If you get no reply, don't go any further until you've got the tunnel working. Don't build your server with Vultr IPv6 option, make sure it is IPv4 only.

How strange... problem, at least for me, is that it appears to work...
ping6 google.com
PING google.com(lax17s03-in-x0e.1e100.net) 56 data bytes
64 bytes from lax17s03-in-x0e.1e100.net: icmp_seq=1 ttl=58 time=1.01 ms
64 bytes from lax17s03-in-x0e.1e100.net: icmp_seq=2 ttl=58 time=1.15 ms
64 bytes from lax17s03-in-x0e.1e100.net: icmp_seq=3 ttl=58 time=1.07 ms
64 bytes from lax17s03-in-x0e.1e100.net: icmp_seq=4 ttl=58 time=1.07 ms

Everything, that is, except for actual streaming of content.

Try DO maybe to rule out an issue with your local network setup?

Yep, did already. Same issue as the original post and my current non-DO setup. Netflix just reported "internet connection problem".

Sorry, no idea. Give me access to your server and I'll have a look to see if I can work it out..

I'll spin that DO droplet back up and you can jump in!

I am using digitalocean and facing the following issues:
sudo iptables -I FRIENDS -s x.x.x.x/32 -j ACCEPT iptables-save > /etc/iptables/rules.v4 || iptables-save > /etc/iptables.rules
it adds 3 lines ip addresses of my VPS
And it doesn't delete ip addresses when I run
sudo iptables -D FRIENDS -s x.x.x.x/32 -j ACCEPT iptables-save > /etc/iptables/rules.v4 || iptables-save > /etc/iptables.rules
I also receive Bad Argument 'Iptables-save'

@abukabhs: Start again: https://github.com/ab77/netflix-proxy
You are using some pretty old instructions and the commands you are using are also completely wrong.

I am getting US netflix working though!

@abukabhs: good for you.
try:
sudo iptables -D FRIENDS -s x.x.x.x/32 -j ACCEPT && \
  (sudo iptables-save > /etc/iptables/rules.v4 || sudo iptables-save > /etc/iptables.rules)

OK, I fixed the major issue, which was affecting most people. Basically, IPv6 was broken in most cases.
Please rebuild with the new version and it should be good to go
(no need to destroy the tunnel, use the current configuration - jut rebuild the VPS).

(also, re-read the instructions, there is an extra step without which it will fail)

Yes! It works! :) Thanks, I donated a beer through Paypal, cheers.

Authorizing Additional IPs
how to reset the password?
which file has the credentials?

Ugh... I'm still struggling for some reason. Same issue, even with the new -s option for build.sh... New droplets and everything. Just not having any luck! At least @nixxle proved that the system works again!

Netflix was working fine with the old instructions.
With the new one I created the tunnel and I ran the command:
ifup he-ipv6 && git clone https://github.com/ab77/netflix-proxy /opt/netflix-proxy && cd /opt/netflix-proxy && ./build.sh -s 2001:470:1f07:8a6::
I received the following:
; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> +time=10 netflix.com @<IP_ADDRESS>
;; global options: +cmd
;; connection timed out; no servers could be reached
Failure! Retrying in 160..
You've failed me for the last time! (/usr/bin/dig +time=10 netflix.com @<IP_ADDRESS>
; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> +time=10 netflix.com @<IP_ADDRESS>
;; global options: +cmd
;; connection timed out; no servers could be reached

Build command wrong - you are missing the subnet prefix (e.g. ./build.sh -s 2001:470:1f07:8a6::/64).

To change the password, see https://github.com/ab77/netflix-proxy/wiki/Changing-Admin-Password-For-Auth-Version.
You can work everything else out from that and this: https://github.com/ab77/netflix-proxy/issues/114#issuecomment-201742433

@scottaki: did you get it working?

@ab77... no, unfortunately no luck for me... everything seems to build and test OK, but actually trying to play a title on Netflix just results in "internet connection problem", with lots of error 420s when looking at the web inspector. Everything else loads OK. Ping6 and Traceroute6 give the right results. Weird...
Any chance you could take a quick look for me? Same login details as last time – just added your key back in.

I have rebuilt my droplet several times already and I put the command with
/64 and it didn't work.  I will try one more time. I will also try my luck
with vultr.
Thanks
On Sat, Mar 26, 2016 at 12:13 PM, Anton Belodedenko <
<EMAIL_ADDRESS>wrote:

Build command wrong - you are missing the subnet prefix (e.g. ./build.sh
-s 2001:470:1f07:8a6::/64).
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/ab77/netflix-proxy/issues/112#issuecomment-201742520

I've noted that the android app will throw a network error if blocking google/hardcoded DNS - be sure to redirect rather than drop DNS.

@machcnz: very interesting, thanks for pointing this out.

@scottaki: something is wrong with the IPv6 path between your VPS and Netflix CDNs. Try blowing that one away entirely and deploy a new one, in a different DC (doesn't really matter which one). After you build the proxy, first try:
docker run -it --rm ubuntu bash -c "ifconfig && ping6 -c 10 ipv6_1-lagg0-c102.1.iad001.ix.nflxvideo.net"
If you get a reply, then it is working and you should proceed with the build, otherwise there is a communication issue between HE and Netflix, which could be caused by weird MTU mis-matches. You can try playing around with MTU size under the Advanced tab of the HE tunnel setting, but it is easier just to find a DC with a path to Netflix over IPv6 which works..

I changed from DigitalOcean to Vultr and Netflix is working now.
However http://:8080/ didn't accept my user name and password.
I tried the procedure:
https://github.com/ab77/netflix-proxy/wiki/Changing-Admin-Password-For-Auth-Version.
but I get
Error: near "sqlite": syntax error

@abukabhs: Google is your friend here and also #netflix-proxy IRC channel :)
Good luck!

Thanks @ab77. I'll keep working on it and see how things go. Appreciate the help!

I managed to change the password!
Do you have plan to automate ip address addition or deletion?
I need to run a cron script at my dd-wrt or open-wrt router to automatically change my ip address whenever my isp change my ip address.

@abukabhs: automate away! :grimacing:  (and if you find a neat solution, perhaps you can every write a Wiki page?)

It works!! @ab77 you genius! Seems like it was something to do with both of the DCs I was using. Tried somewhere else and it now works perfectly! Amazing!!

Good stuff, I am going to close this before the thread gets hijacked :grimacing:

Seeing the complexity of what you have developed so far, I would think that
automation should be quite easy for you.
Please keep us updated with your future fixes and improvements.
Thanks a lot for your support.
On Sat, Mar 26, 2016 at 2:30 PM, Anton Belodedenko<EMAIL_ADDRESS>
wrote:

@abukabhs https://github.com/abukabhs: automate away! [image:
:grimacing:] (and if you find a neat solution, perhaps you can every
write a Wiki page?)
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/ab77/netflix-proxy/issues/112#issuecomment-201774021

Hello
Would you consider supporting the use of no-ip to spare users from manually
updating their ips?
Thanks
On Sat, Mar 26, 2016 at 7:11 PM, Hussain Abukabbos<EMAIL_ADDRESS>wrote:

Seeing the complexity of what you have developed so far, I would think
that automation should be quite easy for you.
Please keep us updated with your future fixes and improvements.
Thanks a lot for your support.
On Sat, Mar 26, 2016 at 2:30 PM, Anton Belodedenko <
<EMAIL_ADDRESS>wrote:

@abukabhs https://github.com/abukabhs: automate away! [image:
:grimacing:] (and if you find a neat solution, perhaps you can every
write a Wiki page?)
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/ab77/netflix-proxy/issues/112#issuecomment-201774021


Fix Flow nits in DraftEditorProps

In one case it seems the return type was just incorrect. We also
clarified the comments for 'keyBindingFn' in order to avoid confusion -
previously due to some Flow issues we had almost changed this flow type
to be incorrect. See https://github.com/facebook/draft-js/pull/894
Also fixes a hanging comma lint.

Going to merge this in order to unblock releasing 0.10.0 soon.


NodeFilesystemSpaceFillingUp is triggered before kubelet gc could start

What happened?
The alert NodeFilesystemSpaceFillingUp is triggered because the filesystem is used more than 81%.
In my case, filesystem space is occupied mostly by Docker images.
Did you expect to see some different?
I'd expect to see in action the kubelet GC but the default values are https://github.com/kubernetes/kubernetes/blob/e1fdaa177f60e280669aef94074a21efd0d928ea/pkg/kubelet/apis/config/v1beta1/defaults_linux.go
So this means that this alarm triggers in many clusters out there except for those who tune these settings.
Which is your opinion here? Would you tune kubelet GC to be more aggressive and kick in before the alarm would be triggered? So for example set imagefs.available": "45%
How to reproduce it (as minimally and precisely as possible):
Environment

Prometheus Operator version:

v0.33.0

Kubernetes version information:

v1.14.6

Kubernetes cluster kind:

kops
Anything else we need to know?:

Is the warning alert the one that fires for you or the critical one? These alerts come from the general node mixin, which nominally has nothing to do with kubernetes, so if anything then we need to make things configurable there to fit the kubernetes case.

It is the warning one. yep, that's true. these alerts rely on the node-exporter metrics and don't have any context about kubernetes operations.
What do you propose here @brancz?

The alert should probably be more configurable in the node mixin.

Yeah, make sense. Then I would open an issue and possibly a PR here https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin.

Closing this. Thank you @brancz

I'd be ok with leaving this open here as we would probably want a specific setting in kube-prometheus for the node mixin.

Oh make sense. Reopning then.

Any updates on this? We're experiencing the same behaviour and we're unsure on how to tackle it. Should we decrease the GC threshold to something like 79%, or should we fine-tune the alert so it only fires when the available disk space drops below 15%?
Also, @mazzy89 did you end up opening an issue on the node_exporter repo? If so, could you link it? I couldn't find it.

For the moment we are just living with the warnings that disappear after a while because the kubelet GC kicks in and clean the node.
This kind of alerts requires to be tuned according the kubelet GC, I believe that each cluster has different settings so I'm not sure how to make the configuration tunable and work for all the cases.
The issue anyway is not needed to be opened under node_exporter but under kube-prometheus.

As said above this first needs to be made configurable in our dependency that we use for node monitoring. Then we can fix it here.

I'm willing to contribute on that regard, but I first wanted to validate the approach.
So you'd rather adjust the alert thresholds than adjust the kubelet GC settings, right? If so, the place to make these alerts configurable is the node_exporter repo, correct?

Correct! :)

I've created a PR on the node_exporter project: https://github.com/prometheus/node_exporter/pull/1624
I think it should cover this use case, right?

Ok, the PR in the node_exporter project is merged. Now, what would be the next steps on this repo?
Considering the kubelet triggers GC at 85% by default, I'd set the critical alert threshold at 85% or 86%. What do you think?

Sounds good!

Here you go: https://github.com/coreos/kube-prometheus/pull/432

Closing as related PR is merged.

I have a brand new install of the helm chart (https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) and I'm seeing these alerts
Labels
alertname = NodeFilesystemSpaceFillingUp
container = node-exporter
device = /dev/sda1
job = node-exporter
mountpoint = /
namespace = monitoring
pod = prometheus-operator-prometheus-node-exporter-pjh84
Annotations
description = Filesystem on /dev/sda1 at xxx has only 19.68% available space left and is filling up.
However, I dont understand why the threshold is 20% and not 15% (i.e if kubelet allows "imageGCHighThresholdPercent": 85, then it means 15% should be available)
Am I misunderstanding something here ?
Should I open another issue ?

@christopheblin There are 2 thresholds(for warning & critical alert) related to NodeFilesystemSpaceFillingUp. fsSpaceFillingUpCriticalThreshold was set to 15% to align with kubelet imageGCHighThresholdPercent: 85 and fsSpaceFillingUpWarningThreshold was set to 20% to align with imageGCLowThresholdPercent: 80.

@arajkumar thanks for your answer !
Ok so I'm hitting the warning alert and not the critical ... and this is considered "normal"
As a newcomer to k8s monitoring and kube-prometheus, I'd argue that it is kind of "weird/strange" that I'm required to silence an alert from the monitoring stack day 1 :)
unless there is something I can do somewhere and that I've missed :)

Ok so I'm hitting the warning alert and not the critical ... and this is considered "normal"

@hristopheblin, I'm not saying this is "normal" :), as the default kubelet GC suppose to happen on every 2m. Probably there might be other contributors?

@arajkumar thank you for your precisions and sorry if I misunderstood you
I should not have the warning because the GC should happen every 2m and so hitting lowThresholdPercent should NOT happen -> is that what you meant ?
if yes, I still do not see what is causing this issue on the pod prometheus-operator-prometheus-node-exporter-pjh84 directly created from the kube-primetheus helm chart ...
When I search for "NodeFilesystemSpaceFillingUp node-exporter", this issue is among the top answers and the single one that is providing some pointers
Would you have any other clue / pointers on why this is happening ? is there something I can look at ?

@arajkumar thanks again for the answer !
the thing is that this pod is directly created by https://github.com/prometheus-community/helm-charts, I do not even know what there is exactly inside this pod ...
unfortunately, I cannot find time to diagnose more in depth so, at the moment, I ignored the alert in the alertmanager when severity=warning :(
I'll try to open another issue once I will be able to investigate
thanks again for the support, at least I am now confident that this will not blow up inadvertently :)

the thing is that this pod is directly created by https://github.com/prometheus-community/helm-charts, I do not even know what there is exactly inside this pod ...

You don't need to worry about the things inside the pod, In this case node_exporter pod mounts the host/node file-system to expose metrics related to fs mounts.
So the quick thing to do is to get into the node where the node_exporter pod runs(it will run all k8s nodes, as it a daemonset) and see how much space the root fs still has(du ..).


Not compatible ie11

ENG Translate
SCRIPT5007: Cannot get property '__handlers' that is undefined or null reference.
I'm using the versions of the list below.
next.js 11.1.2
swr: 1.0.0
I checked similar issues and tried all the settings at the bottom of the issue, but the same error continues to occur.
ISSUE : https://github.com/vercel/swr/issues/278
I would appreciate it if you could let me know which part to check or try.

I'm also seeing an error in IE11 on a page that only uses SWRConfig from swr:

Invalid calling object

Help would be greatly appreciated!

@philipseo I've added https://github.com/vercel/swr/pull/1463 which solves the bug that I was seeing.
I couldn't reproduce your bug though. I'm currently using SWRConfig, useSWR and useSWRImmutable in my code base. These APIs seem to work in IE11 with the fix from the PR.


✨Configuration UI for adding/updating a server

Closes #24
Closes #56
User can view server options, edit, and add.
Note: for a production ready system (aka non-prototype) probably would be better to store server options in a DB instead of the Redux store - the Redux store only persists for a browser session.

I think it might be useful to display on the Login screen which server is being logged into. You could display the server name or the IP. Probably the IP?
Related question - is there a unique constraint on the server configs - for example can someone put in two server configs with the same IP?

@znatty22 I removed the REACT_APP_KF_PW and REACT_APP_KF_USER variables like we talked about, and also logged the user out between servers.
I think that having the server selection be the first step into the app might be good - there are a view ways a user can get "stuck" with the current flow I think. So when thinking about the dashboard UX overhaul I'll think about that too.

@abgeorge7 If you want I could help you get the FHIR server up locally? Should be pretty easy since its just a docker compose stack. Then you can test locally with the Localhost server config that I've been testing with.

@znatty22 if theres a token in the browser session it will go directly to the homepage. I can get rid of this session storage if it's confusing.
The variable for auth should be REACT_APP_FHIR_API_AUTH_REQUIRED instead of REACT_APP_FHIR_AUTH_REQUIRED (there was a typo in the previous README)


text background extended on the next line

SerialPort Version
10.4.0
Node Version
16.15.1
Electron Version
No response
Platform
No response
Architecture
No response
Hardware or chipset of serialport
VSCode bash terminal + ST Link V3 serial port
What steps will reproduce the bug?
#define B_RED_BG(msg) "\e[1;101;30m" msg "\e[0m"
printf(B_RED_BG("line 1") "\n");
printf(B_RED_BG("line 2\n")); // here \n line terminator precedes the \e[0m reset sequence
printf(B_RED_BG("line 3") "\n");
printf(B_RED_BG("line 4") "\n");

What happens?
If on previous line terminal settings were reset to default (\e[0m) after line terminator (\n), background of the next line gets extended to the end of the line.

What should have happened?
I guess, whether or not \n line terminator precedes the \e[0m reset sequence should not matter, but even if it should, then the current line should be affected, not the next one.
Additional information
No response

Hi Thomasz,
The output looks exactly like what I would expect.  What makes you believe that an escape sequence issued after a newline affects the previous line?
First, I think order matters. Second, if so, then the terminating escape has no effect since it is immediately followed be the opening escape sequence of the following printf() statement.
Try
#define B_RED_BG(msg) "\e[1;101;30m" msg "\e[0m"
printf(B_RED_BG("line 1") "\n");
printf("\e[1;101;30m“ „line2\n“);
printf(B_RED_BG("line 3") "\n");
printf(B_RED_BG("line 4") "\n");
The output should be the same as before.
As a second variant, try this:
#define B_RED_BG(msg) "\e[1;101;30m" msg "\e[0m"
printf(B_RED_BG("line 1") "\n");
printf(B_RED_BG("line 3") "\n");
printf(B_RED_BG("line 4") "\n");
printf(B_RED_BG("line 2\n"));
The last printed line (with “line 2”) should be completely red, and then be follow by an empty black line.
Kind regards,
Goetz
Von: T
omasz Jastrzębski @.>
Gesendet: Sonntag, 26. März 2023 14:27
An: serialport/node-serialport @.>
Cc: Subscribed @.***>
Betreff: [serialport/node-serialport] text background extended with auto-scroll (Issue #2614)
SerialPort Version
10.4.0
Node Version
16.15.1
Electron Version
No response
Platform
No response
Architecture
No response
Hardware or chipset of serialport
VSCode bash terminal + ST Link V3 serial port
What steps will reproduce the bug?
#define B_RED_BG(msg) "\e[1;101;30m" msg "\e[0m"
printf(B_RED_BG("line 1") "\n");
printf(B_RED_BG("line 2\n"));
printf(B_RED_BG("line 3") "\n");
printf(B_RED_BG("line 4") "\n");
What happens?
If on previous line terminal settings were reset to default (\e[0m) after line terminator (\n), background of the next line gets extended to the end of the line.
https://user-images.githubusercontent.com/7325277/227775007-98742925-61e7-4ab1-8e6b-d0b6abdd5f16.png
What should have happened?
I guess, whether or not \n line terminator precedes the \e[0m reset sequence should not matter, but if it should, then the current line should be affected, not the next one.
Additional information
No response
—
Reply to this email directly, view it on GitHub https://github.com/serialport/node-serialport/issues/2614 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG2K5C6CM42YVPXRZXZMNTW6A72PANCNFSM6AAAAAAWIE2TJY .
You are receiving this because you are subscribed to this thread.  https://github.com/notifications/beacon/AAG2K5G45E2WLRZLGR3DQLDW6A72PA5CNFSM6AAAAAAWIE2TJ2WGG33NNVSW45C7OR4XAZNFJFZXG5LFVJRW63LNMVXHIX3JMTHGDTP62Y.gif Message ID: @.*** @.***> >

By the way, this is not node programming, but shell programming. And in your code, the escape sequence is not applied to the input line which contains it but to the output. If you send a newline, then a new line is started. If the newline is followed by an escape sequence, then, of course, this will be applied to the current line which is the new one. What was the current line before sending a newline is not current any more after sending it!
Kind regards,
Goetz
Von: Tomasz Jastrzębski @.>
Gesendet: Sonntag, 26. März 2023 14:27
An: serialport/node-serialport @.>
Cc: Subscribed @.***>
Betreff: [serialport/node-serialport] text background extended with auto-scroll (Issue #2614)
SerialPort Version
10.4.0
Node Version
16.15.1
Electron Version
No response
Platform
No response
Architecture
No response
Hardware or chipset of serialport
VSCode bash terminal + ST Link V3 serial port
What steps will reproduce the bug?
#define B_RED_BG(msg) "\e[1;101;30m" msg "\e[0m"
printf(B_RED_BG("line 1") "\n");
printf(B_RED_BG("line 2\n"));
printf(B_RED_BG("line 3") "\n");
printf(B_RED_BG("line 4") "\n");
What happens?
If on previous line terminal settings were reset to default (\e[0m) after line terminator (\n), background of the next line gets extended to the end of the line.
https://user-images.githubusercontent.com/7325277/227775007-98742925-61e7-4ab1-8e6b-d0b6abdd5f16.png
What should have happened?
I guess, whether or not \n line terminator precedes the \e[0m reset sequence should not matter, but if it should, then the current line should be affected, not the next one.
Additional information
No response
—
Reply to this email directly, view it on GitHub https://github.com/serialport/node-serialport/issues/2614 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG2K5C6CM42YVPXRZXZMNTW6A72PANCNFSM6AAAAAAWIE2TJY .
You are receiving this because you are subscribed to this thread.  https://github.com/notifications/beacon/AAG2K5G45E2WLRZLGR3DQLDW6A72PA5CNFSM6AAAAAAWIE2TJ2WGG33NNVSW45C7OR4XAZNFJFZXG5LFVJRW63LNMVXHIX3JMTHGDTP62Y.gif Message ID: @.*** @.***> >

@hellerim I see what happens. If "carriage" moves to the new line while background is set, the entire new line background is immediately painted with the current selected background color, even if background is reset to default immediately after and before any subsequent character is sent.
I would say, it is a strange behavior, but if it is expected and according to specs then it is not a bug.

I see the same effect also when the reset sequence \e[0m is followed by new line '\n' and this new line causes scroll.
This makes results a bit nondeterministic, but that is another issue.

HI Thomasz,
if this is so, then you have an issue with the bash shell but not with Node.js. Please, check the bash release docs. If you don’t find anything about this behavior, send them a bug report.
Kind regards,
Goetz
Von: Tomasz Jastrzębski @.>
Gesendet: Sonntag, 26. März 2023 17:31
An: serialport/node-serialport @.>
Cc: hellerim @.>; Mention @.>
Betreff: Re: [serialport/node-serialport] text background extended on the next line (Issue #2614)
I see the same effect also when the reset sequence \e[0m is followed by new line '\n' and this new line causes scroll.
This makes results a bit nondeterministic, but that is another issue.
—
Reply to this email directly, view it on GitHub https://github.com/serialport/node-serialport/issues/2614#issuecomment-1484132261 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG2K5GIS66XTAUSKQTFBJLW6BVNDANCNFSM6AAAAAAWIE2TJY .
You are receiving this because you were mentioned.  https://github.com/notifications/beacon/AAG2K5FNVVYE5YRXTYN4JP3W6BVNDA5CNFSM6AAAAAAWIE2TJ2WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSYOYH2K.gif Message ID: @.*** @.***> >


Does not work with new Firefox (48+) browsers

Now it does not work for SampleUITest for Firefox.
There is need to switch to geckodriver:
http://stackoverflow.com/questions/38916650/what-are-the-benefits-of-using-marionette-firefoxdriver-instead-of-the-old-selen/38917100#38917100
I have tried but had not luck, and my time run out.
@cezarypiatek btw which version of FF (x86, x64) are you using? There are different drivers and I think we should stick to the one that you have.

So far I've been using  drivers x86.  We have to migrate Tellurium to SeleniumHQ 3 in order to use gecko driver.

@kmorcinek Currently I'm unable to migrate Tellurium to Selenium 3.X because geckodriver has no option of taking full page screenshots. Problem is described in this issue https://github.com/mozilla/geckodriver/issues/570 Without full page screenshots VisualAssertions seems to be partially useless. You can still run tests on Firefox with Tellurium and Selenium 2.53, simply install firefox 47 and block auto-updates.  There is also an option for creating BrowserAdapter from existing driver so there is a chance that it could work with selenium 3.X (try install Tellurium, update Selenium to 3.X and  add appropriate binding redirect in app.config)

I was able to successfully migrate to Selenium 3.6 with support for legacy firefox 47.0.1 as well for the newest one with geckodriver.


Add support for logging to local syslog

Meilisearch HTTP currently logs all messages to the stderr output. When Meilisearch is run as a system service, users usually want to aggregate logs from all system services and reroute them as needed (to different files, remote log servers etc.). Unix systems typically use syslog for this.

Hello @MixusMinimax!
Thanks again for this suggestion, I discussed it with our PM, and we think it could be indeed useful. However, this impacts the user interface, and the product team needs more time to really think about the suggestion. Unfortunately, this feature is not the one we want to prioritize right now, we have other features to work on before this one, so I cannot give you any precise ETA.
So I will close this issue in favor of this discussion in the product repo https://github.com/meilisearch/product/discussions/543
The product repository is where we discuss the product improvements before choosing to implement them in the code base
However, I let the PR open: if we decide to go with your solution, we will use your work. If you're not available at this moment, and if we need to apply changes, we will merge your branch into a temporary branch, we will continue the work, and then merge this temporary branch into main.
Thanks again for the suggestion, it brings an interesting point to our product table! 😄

Some users seem to ask Meilisearch to return the information into a specified file (if required).

Not to a specific file, but to syslog, the standard for message logging on unix systems.


Use full stack traces

Having truncated stack traces is really sad. cc @malcolmredheron

Coverage remained the same at 100.0% when pulling 72c1bca41f894a927e712f60aa8b398ced87c6cb on stack-traces into 06664c227b31421405081d57072819a7e47936e3 on master.

lgtm


Move to tabs in mesh elements

Description
We've discussed having items:

overview
policies
gateways
dataplanes
services

just be tabs instead of elements in the sidebar.
This is related to: https://github.com/kumahq/kuma-gui/issues/892

Ah thanks @lahabana 👍


Release 0.7.6

[x] write release notes using the Release Notes Helper
[x] bump version in meson.build to an even minor release number
[x] bump SIMDE_VERSION_{MAJOR,MINOR,MICRO} in simde/simde-common.h to match
[ ] make release candidate (make sure to mark it as a "pre-release" in GitHub)
[x] test on Debian against all official and unofficial architectures using all available gcc and clang versions
[ ] Rebuild all packages in Debian that use SIMDe in experimental to see if there are any regressions.
[x] fix or ignore any errors found
[ ] make another release candidate (repeating the above as necessary)
[x] make final release
[x] upload an amalgamated build of the headers
[x] create a matching tag for the simde-no-tests repo
[x] blog post https://simd-everywhere.github.io/blog/2023/05/19/0.7.6-release.html
[ ] email the mailing list
[x] ping vcpkg to update: https://github.com/microsoft/vcpkg/issues/31465
[x] post-release: bump version in meson.build to an odd minor release number
[x] post-release: bump SIMDE_VERSION_{MAJOR,MINOR,MICRO} in simde/simde-common.h to match
[ ] copy this checklist to the wiki for next time

On Debian, all tests are passing on all official architectures using SIMDe v0.7.6 + https://github.com/simd-everywhere/simde/commit/4de999a0ad2c15b64efc6f51b15f21435f87ba2e & https://github.com/simd-everywhere/simde/commit/96d6e5392eb3baa72b8934594925d7c6754a729b


Profile name

Add a command to get/set default profile.
command
1.Get default profile name
nem2-cli profile current

It will ask you to fill in the profile name, you just press Enter to skip.
2.Set default profile name
nem2-cli profile current -d My_Profile

I can't use the default keyword, because it conflicts with  the default setting.

Pull Request Test Coverage Report for Build 195

0 of 0   changed or added relevant lines in 0 files are covered.
19 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-13.2%) to 79.429%

Files with Coverage Reduction
New Missed Lines
%

build/src/service/profile.service.js
2
62.5%

build/src/respository/profile.repository.js
17
41.51%

Totals

Change from base Build 187:
-13.2%

Covered Lines:
101

Relevant Lines:
129

💛  - Coveralls

@NemTechCN This branch has conflicts that must be resolved, could you resubmit the PR fixing the conflicts? Thanks!


Question -- what would it take to get controller buttons working?

Out of curiosity, what would it take? Reverse engineering, trying to bind things properly, trial/error, or something else?

Mostly it would take knowing how SteamVR expects drivers to give input data to SteamVR. At the time I was looking at it, there was no example code and documentation was not enough for me to get it working, but I could give it another try.
Also there is this branch that makes the Vive controllers work as 3dof controllers, which would give me input data from a real world device which would make it more interesting to develop. https://github.com/lubosz/OpenHMD/commits/wip/vive-controller
So I may take another look soon, but I'd also be happy for anyone to contribute some code that does it. :)

#12 would be the starting point.
I think first I need to write a demo application that uses the SteamVR Input System instead of the legacy input so we can properly test it.

I could test on the Rift CV1

because if this worked fully i'd be able to switch to linux fully :D

I could test on the Rift CV1

Have you managed to get your CV1 touch controllers working with OpenHMD alone? At the time I created the pull request, I couldn't get the CV1 touch controllers working, but I tested it with @ChristophHaag's simulator and almost all the buttons I tried to implement worked.

For my own reference: hydra driver fork, a branch with the interesting name steamvr_input_new: https://github.com/betavr/steamvr_driver_hydra/tree/r/steamvr_input_new
There has been a complaint that it doesn't work anymore though https://github.com/ValveSoftware/driver_hydra/issues/9

The BetaVR driver problem was in the unique ID (razer_hydra), the games did not understand it and therefore nothing worked. It was necessary to change the ID to "vive_controller" and bind the buttons, after which everything worked fine.

I have some of the controller buttons working in the https://github.com/ChristophHaag/SteamVR-OpenHMD/tree/controller-input branch, but controller rotation / positioning was quite messed up when I tried it last.

I also have problems with rotation, in my other project (DIY VR), one Euler axis uses 180 degrees, and the other two use 360 degrees, and after converting to a quaternion, problems start.

I need to try it again and remind myself what was wrong when I looked at it. I just remember the the controllers acted in a way I didn't expect.

Hey, I decided to try out the controller-input branch on my rift s yesterday and again today. From what I can tell so far the triggers, B and Y are working...is that to be expected? in that case I could try to figure out what to do about the other buttons, but I have yet to understand what part steamvr-openhmd and openhmd itself are playing. Maybe someone could shed some light on that for me?

OpenHMD reports events for all the buttons, triggers and grips - I don't think there's anything to do in OpenHMD right now, and in the controller-input branch they all get reported to SteamVR too. What's missing is knowledge and glue in getting the input bindings set up right so that the controls actually work.
TBH - I'm surprised you got trigger, B & Y working! What was running in SteamVR then? SteamVR Home, or something else?

I forgot to mention the menu and oculus button worked too :-D I had them working in SteamVR Home and Skyrim VR.
I didn't even configure anything yet, they just worked out of the box...maybe they picked up some other controller profile. The only thing I did was to create a
.ohmd_config.txt
which gave me the rotations of the controllers. One thing I noticed was, that SkyrimVR was showing renders of a different VR controller that I didn't recognize...so the controllers might also have been picked up as a different controller due to my config maybe?
I'll see if I can get the rest of the controller to work next.

It's also possible that someone magical changed in SteamVR since I last tested :)

Turns out ~/.ohmd_config.txt wasn't necessary...it seems to work just fine without.
I have exported the controller profile from windows and tried to copy some of the bindings from there into a profile on linux under ~/.local/share/Steam/userdata/someuserid/250820/remote/binding_config.json, however without much success.
Going into the settings->controller->test controller did however yield some information at least:  clicking all buttons, trigger and grip are being recognized by SteamVR. (touching the buttons doesn't do anything though). Then I tried calibrating the thumbsticks and they appear to only be recognized within the top right quadrant. (I guess that's got something to do with radians/degree conversion).
So that's where I am at currently, not sure what to look at and try next yet :-/

Touching the buttons doesn't work because OpenHMD doesn't have report types for capsense yet. The Rift driver can see the touch information, but can't report it to the SteamVR-OpenHMD layer.
I thought the thumbstick reports were working - but I guess not. It's been a while since I've tested that.

I think the missing piece here is anyone who understands how the input profile bindings stuff works. The buttons/grips/trigger reports are there... but SteamVR doesn't know how to map them to the application inputs.

Touching the buttons doesn't work because OpenHMD doesn't have report types for capsense yet. The Rift driver can see the touch information, but can't report it to the SteamVR-OpenHMD layer.

Ah, makes sense, just thought I should note anything I am not sure about.

I thought the thumbstick reports were working - but I guess not. It's been a while since I've tested that.

I downloaded your rift-s-experiments repo today to see what is actually coming out of the device, and the data is certainly there.
Also, I managed to bind teleporting around to the X key, which at least gives me some mobility withing SteamVR Home! I'll proceed in creating some bindings and try to stay reasonably close to the windows ones. I think I should be able to come up with something useful either later tonight or tomorrow.

Hi, I wanted to give a short update und this, since I have been monkeying around with these bindings. As far as I currently know, there's two ways to get bindings set up:

Manually Inputting them through the bindings menu in VR
Downloading them from the workshop
Since there currently are no bindings on the workshop I have been configuring these by hand, mostly copying the bindings from windows. This has gotten me some working bindings for SteamVR Home and the SteamVR Dashboard so far, SkyrimVR apparently counts as a legacy application, and for that I have yet to get the thumbsticks configured (judging from the windows config that should be axis0, but that doesn't exist as an option O.o). I'll try and publish the working configs for SteamVR Home and Dashboard to the workshop, in case someone would like to try them out :)

That sounds great! It should also be possible to find the bindings files steam has created in its directories somewhere and to put them directly into the upstream repo.

Hm, I've been grepping through ~/.local/share/steam and  the only results are binaries, some workshop files and my personal bindings. Which matches what I found online - in order to export anything of this sort one has to create a personal binding which then goes into "~/.local/share/steam/userdata/$userid/$gameid". And also steam apparently really doesn't like one manually copying files into there :-D At least I wasn't able to load them afterwards.
The files in the directories mentioned above could of course go into the upstream repo.

So I have been trying out mappings for skyrim vr the past week or so, and I got some mixed results with this mapping: legacy_game.json.txt
But as far as I can tell the mapping is perfectly fine for rift s controllers, Skyrim VR just wrongly assumes them to be HTC Vive controllers, which is an issue on the game's side. I'm currently looking for another game to try this out with ;-)
I would consider this issue resolved, since it only depends on controller bindings within Steam VR now, everything else appears to work just fine :-)

Awesome! I'll look at incorporating that as the default legacy bindings on the weekend and see where it leads

If you get a chance, can you try the changes I just pushed? I incorporated your bindings - the legacy bindings and the dashboard ones from the steam workshop. I removed the references to non-existent touch reports for the joystick, trigger and grip.
Let me know if it's any better/worse. I had quite inconsistent results here - I need a more stable test machine for Linux/SteamVR.

I just pulled it and tried it, seems to work just as expected for me! So, everything works in SteamVR Home and the Dashboard, and is still kinda broken in SkyrimVR because the game expects me to be holding HTC Vive controllers instead of Rift S ones. While looking for information on this I found a possible fix for this on the games' side, that I will have to try out still.
How this works for other headsets and controllers I don't know.

Thanks for the quick testing!
How does it break in SkyrimVR? Is it just that it's showing the wrong controller models, or is there something more going on?
I think fixing the models would be possible - but to really do it right, OpenHMD needs to provide model information for the SteamVR-OpenHMD wrapper to publish.

The Mail reached me, when I was in front of my PC and had free time, therefore I could test it almost immediately.
The thumbsticks directions are not being recognized in SkyrimVR, when I use teleporting motion (like in SteamVR Home), it's bound to left stick click, which I didn't put there. I believe this is related to a similar issue with Index controllers:
https://www.nexusmods.com/fallout4/mods/41931
Which curiously enough gets fixed by making them act as oculus controllers on windows, apparently.
 In that game the thumbsticks are working perfectly fine, therefore I'm fairly certain that my previous issues with SkyrimVR are game specific and the bindings themselves are okay. Haven't solved those issues yet though...

I merged the controller branch back to master last night - that'll make it easier for more people to test


Add Showwcase website

Added Showwcase site.

You were supposed to add it to the database, but you created a separate file for it. @Ayaan49

You were supposed to add it to the database, but you created a separate file for it. @Ayaan49

Sorry, I thought we needed to create a seperate file to add the data. I will do it again.


[Spark] Add Scala clone API

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

Resolves #3391
Add a Scala API for cloning a Delta Table. Simply creates a CloneTableStatement like the SQL parser does and lets the analyzer handle the rest. I tried to mimic what limited info I could find on the current Databricks implementation: https://docs.databricks.com/en/delta/clone.html#language-scala.
How was this patch tested?

New suite extending CloneTableSuiteBase
Does this PR introduce any user-facing changes?

New Scala API on DeltaTable for cloning

@Kimahriman Hey Adam, are you still working on this? I am interesting in adding the Scala Clone APIs as well and have some time these weeks, shall I pick up this issue for you?

@Kimahriman Hey Adam, are you still working on this? I am interesting in adding the Scala Clone APIs as well and have some time these weeks, shall I pick up this issue for you?

I'm just waiting for feedback and an answer on the hadoopConf thing

@Kimahriman Could you give me the permission to push to your branch?
This works when I run it locally.
Remove clone* from ignoreMethods and move them to testedMethods, you can keep .filter(!_.isSynthetic)
  test("clone - with filesystem options") {
    withTempDir { dir =>
      val baseDir = fakeFileSystemPath(dir)

      val srcDir = new File(baseDir, "source").getCanonicalPath
      val dstDir = new File(baseDir, "destination").getCanonicalPath

      spark.range(10).write.options(fakeFileSystemOptions).format("delta").save(srcDir)

      val srcTable =
        io.delta.tables.DeltaTable.forPath(spark, srcDir, fakeFileSystemOptions)
      srcTable.clone(dstDir)

      val srcLog = DeltaLog.forTable(spark, new Path(srcDir), fakeFileSystemOptions)
      val dstLog = DeltaLog.forTable(spark, new Path(dstDir), fakeFileSystemOptions)

      checkAnswer(
        spark.baseRelationToDataFrame(srcLog.createRelation()),
        spark.baseRelationToDataFrame(dstLog.createRelation())
      )
    }
  }

  test("cloneAtVersion/timestamp - with filesystem options") {
    Seq(true, false).foreach { cloneWithVersion =>
      withTempDir { dir =>
        val baseDir = fakeFileSystemPath(dir)
        val fsOptions = fakeFileSystemOptions

        val srcDir = new File(baseDir, "source").getCanonicalPath
        val dstDir = new File(baseDir, "destination").getCanonicalPath

        val df1 = Seq(1, 2, 3).toDF("id")
        val df2 = Seq(4, 5).toDF("id")
        val df3 = Seq(6, 7).toDF("id")

        // version 0.
        df1.write.format("delta").options(fsOptions).save(srcDir)

        // version 1.
        df2.write.format("delta").options(fsOptions).mode("append").save(srcDir)

        // version 2.
        df3.write.format("delta").options(fsOptions).mode("append").save(srcDir)

        val srcTable =
          io.delta.tables.DeltaTable.forPath(spark, srcDir, fakeFileSystemOptions)

        if (cloneWithVersion) {
          srcTable.cloneAtVersion(0, dstDir)
        } else {
          // clone with timestamp.
          //
          // set the time to first file with a early time and verify the delta table can be
          // restored to it.
          val desiredTime = "1983-01-01"
          val format = new java.text.SimpleDateFormat("yyyy-MM-dd")
          val time = format.parse(desiredTime).getTime

          val logPath = new Path(srcDir, "_delta_log")
          val file = new File(FileNames.unsafeDeltaFile(logPath, 0).toString)
          assert(file.setLastModified(time))
          srcTable.cloneAtTimestamp(desiredTime, dstDir)
        }

        val dstLog = DeltaLog.forTable(spark, new Path(dstDir), fakeFileSystemOptions)

        checkAnswer(
          df1,
          spark.baseRelationToDataFrame(dstLog.createRelation())
        )
      }
    }
  }

Added you as a collaborator.
I have no idea why those tests passed and I'm very confused that that actually works, I guess I should have tried to write the tests. I especially don't understand how the fakeFileSystemOptions get propagated to writing target table.

@Kimahriman Awesome work! Thank you so much for resolving some of my review comments, let's resolve the outstanding ones and I think we should be getting there soon :)

Just a couple comments on the doc of the clone APIs, then I think the PR looks good to me.

Hey @allisonport-db, could you please take a look at this APIs addition for our DeltaTable? If all looks good, could we have it merged please?


Should environment's name appear in KV name?

🐛 Bug Report
Environment

operating system:
Mac OS
output of rustc -V:
output of node -v:
v10.16.2
output of wrangler -V:
👷 ✨  wrangler 1.3.1

Steps to reproduce
Mini issue that I don't know if this is a bug, or expected, or even which approach is better.
I tried the new environment commands to interact with KV via the CLI. Initially with 1.3.0 and then due a publish failure with that (due to a since-removed file as you speedily released 1.3.1) upgraded to 1.3.1. And I tried creating a KV for dev and prod, the two environments I have set. Both worked, nice, but I noticed a slight change: now when I make a prod one, 'prod' appears in the KV name. Previously it did not.
Not having it in the KV name seemed more consistent with the worker name Since I'm using the approach of only declaring the name once at the top of wrangler.toml. And then letting wrangler append the env, as it desires. So it does for dev, and I assume it doesn't for prod. Haven't tried publishing a production one.
(I tried making a KV called 'config' initially and then one called 'kv')
1.3.0
$ wrangler kv:namespace create "config" --env prod
🌀  Creating namespace with title "worker-config"
✨  Success: WorkersKvNamespace {
id: "...",
title: "worker-config",
}
✨  Add the following to your wrangler.toml under [env.prod]:
kv-namespaces = [
{ binding = "config", id = "..." }
]
$ wrangler kv:namespace create "config" --env dev
🌀  Creating namespace with title "worker-dev-config"
✨  Success: WorkersKvNamespace {
id: "...",
title: "worker-dev-config",
}
✨  Add the following to your wrangler.toml under [env.dev]:
kv-namespaces = [
{ binding = "config", id = "..." }
]
1.3.1
$ wrangler kv:namespace create "kv" --env dev
🌀  Creating namespace with title "worker-dev-kv"
✨  Success: WorkersKvNamespace {
id: "...",
title: "worker-dev-kv",
}
✨  Add the following to your wrangler.toml under [env.dev]:
kv-namespaces = [
{ binding = "kv", id = "..." }
]
$ wrangler kv:namespace create "kv" --env prod
🌀  Creating namespace with title "worker-prod-kv"
✨  Success: WorkersKvNamespace {
id: "...",
title: "worker-prod-kv",
}
✨  Add the following to your wrangler.toml under [env.prod]:
kv-namespaces = [
{ binding = "kv", id = "..." }
]
What did you expect to see?
Both to match, either 'prod' appears in the KV's name for both, or neither
What did you see instead?
Now 'prod' appears in the KV name. Now, this isn't a big deal because the KV name is not seen of course, but I wondered if it would have implications if the code looked for a KV with 'prod', and doesn't find one, or vice versa. So in other words, code that works for 'dev' (where the behaviour is consistent) would break when published to 'prod'. Which wouldn't be good. Though perhaps it just uses the id internally and so the name doesn't matter.

hey @gregmsanderson !
This is great feedback. The TL;DR is that ultimately, the namespace title is relatively inconsequential, except that the API asserts that they be unique within a given account. Your Worker only references the namespace by the binding property, and Wrangler uses the id property to establish that binding for you.
So in order to keep uniqueness, but try to reinforce the relationship between the title, your worker script, and your binding, we structured the titles to be <worker-name>-<binding>

@ashleymichal Hmm ... I think I had the [env.dev] and [env.prod] already set up in the toml before running the command. But now you ask, I'm only like 95% sure. So perhaps that was it.
I then wondered if the order matters since I noticed I ran the prod and dev commands in reverse. Can't think why it would result in a different title system. I didn't try and duplicate that.
Out of interest I just deleted the prod KV, checked the toml had a [env.dev] and [env.prod] set, and re-ran the command to make it:
$ wrangler kv:namespace create "kv" --env prod
🌀  Creating namespace with title "worker-prod-kv"
✨  Success: WorkersKvNamespace {
id: "...",
title: "worker-prod-kv",
}
✨  Add the following to your wrangler.toml under [env.prod]:
kv-namespaces = [
{ binding = "kv", id = "..." }
]
... so it does continue to add 'prod' to the title of the KV. Which is fine and I'm reassured it doesn't matter since the id is the important bit in the binding. It sounds like the title is just for the humans.
If the KV name does or doesn't match the worker name (I guess 'worker-dev' for dev, and simply 'worker' for prod), well, no problem. It's clear which is the dev one.
Thanks.


Document how to load echarts/map/js/world.js

Version
4.6.0
Steps to reproduce
Hello,
First, this library is great! easy to use.
However, some more documentation would be helpfull.
Can someone help explain how to load the echarts/map/js/world.js? (in TypeScript), what should I import from it? is it a method call?
Looking for its source, not found, looking in the file gets me dizzy.
Best,
Yarden
What is expected?
Documetation on how to load echarts/map/js/world.js
What is actually happening?
Don't know how to load it

Ok, so I went through the code again, and found that it looks for the echarts require/import.
So I tried just importing the map import 'echarts/map/js/world'; and it worked.
Thank you,
Yarden


Update database.json

Add Force CMS

We have changed how we add modules. No longer is it a manual PR process. Go to https://deno.land/x and click the "add module button" - follow the instructions. Thanks!


Restful API used gin-swagger and oauth2，can not get the client_id,client_secret from the post request.

Describe the bug
Like the next picture,marked some question

To Reproduce

I defined the authorization type "OAuth2Application".

// @securitydefinitions.oauth2.application OAuth2Application
// @tokenUrl http://localhost:8080/oauth2/token
// @scope.write Grants write access
// @scope.admin Grants read and write access to administrative information

I defined the action with some paramater ,especially this '@Security'

// ListAccounts godoc
// @Summary Get all account as a list [auth token]
// @Description get accounts
// @Tags accounts
// @Accept  json
// @Produce  json
// @Param q query string false "name search by q" Format(email)
// @Success 200 {array} model.Account
// @Failure 400 {object} httputil.HTTPError
// @Failure 404 {object} httputil.HTTPError
// @Failure 500 {object} httputil.HTTPError
// @Security OAuth2Application **[admin]
// @Router /accounts [get]
func (c *Controller) ListAccounts(ctx *gin.Context) {

	q := ctx.Request.URL.Query().Get("q")
	accounts, err := model.AccountsAll(q)
	if err != nil {
		httputil.NewError(ctx, http.StatusNotFound, err)
		return
	}
	ctx.JSON(http.StatusOK, accounts)
}

cmd input 'swag init ',after that I get a Restful API server.
Browder my site http://localhost:8080/swagger/index.html,
when I click the 'Authorize' button,I get this bug.

there are my sourcecode repository.

I will look this issue asap.

I will look this issue asap.

Thank you very much.I am standing by for your reply.

Dear Eason,
I am waiting for your solution yet . Do you need some screenshot or code any more ?
I would like to answer you at all times.Thank you so much.

Eason,
Thank you , I corrected this issue, it is worked now.
Reference : https://github.com/swagger-api/swagger-ui/issues/4533.
export const authorizeApplication = ( auth ) => ( { authActions } ) => {   let { schema, scopes, name, clientId, clientSecret } = auth   let headers = {     Authorization: "Basic " + btoa(clientId + ":" + clientSecret)   }   let form = {     grant_type: "client_credentials",     scope: scopes.join(scopeSeparator),     client_id: clientId, //need line     client_secret: clientSecret //need line   }


v4.0.0 error building with MinGW-w64

When building with MinGW-w64 I encountered the following error:
[36/37] Linking CXX shared library libreflex.dll
FAILED: libreflex.dll libreflex.dll.a
C:\Windows\system32\cmd.exe /C "cd . && D:\Prog\winlibs-gcc13.2.0-posix-msvcrt-11.0.1\mingw64\bin\c++.exe -O3 -DNDEBUG   -shared -o libreflex.dll -Wl,--out-implib,libreflex.dll.a -Wl,--major-image-version,0,--minor-image-version,0 CMakeFiles/ReflexLib.dir/lib/convert.cpp.obj CMakeFiles/ReflexLib.dir/lib/debug.cpp.obj CMakeFiles/ReflexLib.dir/lib/error.cpp.obj CMakeFiles/ReflexLib.dir/lib/input.cpp.obj CMakeFiles/ReflexLib.dir/lib/matcher.cpp.obj CMakeFiles/ReflexLib.dir/lib/pattern.cpp.obj CMakeFiles/ReflexLib.dir/lib/posix.cpp.obj CMakeFiles/ReflexLib.dir/lib/simd_avx2.cpp.obj CMakeFiles/ReflexLib.dir/lib/simd_avx512bw.cpp.obj CMakeFiles/ReflexLib.dir/lib/unicode.cpp.obj CMakeFiles/ReflexLib.dir/lib/utf8.cpp.obj CMakeFiles/ReflexLib.dir/lib/matcher_avx2.cpp.obj CMakeFiles/ReflexLib.dir/lib/matcher_avx512bw.cpp.obj CMakeFiles/ReflexLib.dir/unicode/block_scripts.cpp.obj CMakeFiles/ReflexLib.dir/unicode/language_scripts.cpp.obj CMakeFiles/ReflexLib.dir/unicode/letter_scripts.cpp.obj  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
D:/Prog/winlibs-gcc13.2.0-posix-msvcrt-11.0.1/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/ReflexLib.dir/lib/unicode.cpp.obj:unicode.cpp:(.text+0x7ca): undefined reference to `reflex::Unicode::Tables::composer()'
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

The solution was add unicode/composer.cpp to the sources specified with set(lib_sources in CMakeLists.txt.

Ah, yes this is a new source addition added to the makefiles, except for the oversight to add it to CMakeLists.txt too.

Thanks.
No hurry from my side for a new release.

I've released 4.0.1 with the fix and I've also included some extra examples in this release.
Sigh... So as I am releasing these updates with the help of people who use RE/flex and contribute to it, apparently someone does not like that I bump up the RE/flex version information on the Wikipedia RE/flex page for every RE/flex release. He seems to think that I personally have "a conflict of interest" and that I am "promoting myself or the project" (see Talk page), when in fact I've made many minor updates (80%-90%) to bump the version info on Wikipedia. I am also not gaining anything in value for doing this work, nor do I do this for anyone one else who gains by this. RE/flex is free, because I like to work on this sort of stuff for decades. Genivia is my org where I can do that.
If someone volunteers to update the RE/flex version info on Wikipedia from now on, then please do!

Dr. van Engelen,

Sigh... So as I am releasing these updates with the help of people who use RE/flex and contribute to it, apparently someone does not like that I bump up the RE/flex version information on the Wikipedia RE/flex page for every RE/flex release. He seems to think that I personally have "a conflict of interest" and that I am "promoting myself or the project" (see Talk page), when in fact I've made many minor updates (80%-90%) to bump the version info on Wikipedia. I am also not gaining anything in value for doing this work, nor do I do this for anyone one else who gains by this. RE/flex is free, because I like to work on this sort of stuff for decades. Genivia is my org where I can do that.

The world is full of low-lives.  It's much like the old saw that theatre critics are like eunuchs in a harem. They can sit on the side and comment and criticize all they want about technique, but they can't actually perform. That's the case here too.
Keep up the good work!

My fault is that I am a perfectionist, taking time to update small things often.
Would really appreciate when you guys make a few updates and improvements the RE/flex page, like others have done in the past. Otherwise, it is going to be a lost wiki page. I no longer want to update this or comment on it (sunk-cost-fallacy).


Still signout - signin when offline

In my case after I login to google account I disconnect network.
Then
I signout -> OK success state
continue siginin -> OK success state
I don't know that is your concept or bug state or I need implement to prevent that case.

This is expected. Before cached OAuth tokens expire, we allow you to sign-in even if offline. You don't need to do any special handling there.


vm.overcommit_memory 学习笔记

关于 overcommit_memory 学习的一些记录，其中一些为网上前辈文章的整理记录
什么是 vm.overcommit
但在实际情况下，进程请求的内存可能比实际需要的内存少很多，而且很多进程请求的内存并不会全部使用。在这种情况下，如果系统采用严格的内存申请机制，将会导致系统的内存利用率低下，一些能够使用的内存被浪费。
因此，为了提高系统的内存利用率和灵活性，Linux 提供了 vm.overcommit_memory 机制。当这个参数的值为 0 或 1 时，内核允许进程请求超过系统实际可用内存的内存量。这种行为被称为 "overcommit"。
vm.overcommit 的三个值
#define OVERCOMMIT_GUESS        0 // default
#define OVERCOMMIT_ALWAYS       1  // redis
#define OVERCOMMIT_NEVER        2 // postgres

The Linux kernel supports the following overcommit handling modes

0	-	Heuristic overcommit handling. Obvious overcommits of
		address space are refused. Used for a typical system. It
		ensures a seriously wild allocation fails while allowing
		overcommit to reduce swap usage.  root is allowed to
		allocate slightly more memory in this mode. This is the
		default.

1	-	Always overcommit. Appropriate for some scientific
		applications. Classic example is code using sparse arrays
		and just relying on the virtual memory consisting almost
		entirely of zero pages.

2	-	Don't overcommit. The total address space commit
		for the system is not permitted to exceed swap + a
		configurable amount (default is 50%) of physical RAM.
		Depending on the amount you use, in most situations
		this means a process will not be killed while accessing
		pages but will receive errors on memory allocation as
		appropriate.

		Useful for applications that want to guarantee their
		memory allocations will be available in the future
		without having to initialize every page.

https://github.com/torvalds/linux/blob/v4.6/mm/util.c#L481
当 vm.overcommit_memory = 2 时
grep -i commit /proc/meminfo

/proc/meminfo 中的 Committed_AS 表示所有进程已经申请的内存总大小，（注意是已经申请的，不是已经分配的），如果 Committed_AS 超过 CommitLimit 就表示发生了 overcommit，超出越多表示 overcommit 越严重 -> 触发 OOM
那么这个 CommitLimit 是怎么算出来的呢？
int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin)
{
	long free, allowed, reserve;

	VM_WARN_ONCE(percpu_counter_read(&vm_committed_as) <
			-(s64)vm_committed_as_batch * num_online_cpus(),
			"memory commitment underflow");

	vm_acct_memory(pages);

	/*
	 * Sometimes we want to use more memory than we have
	 */
	if (sysctl_overcommit_memory == OVERCOMMIT_ALWAYS)
		return 0;

	if (sysctl_overcommit_memory == OVERCOMMIT_GUESS) {
		free = global_page_state(NR_FREE_PAGES);
		free += global_page_state(NR_FILE_PAGES);

		/*
		 * shmem pages shouldn't be counted as free in this
		 * case, they can't be purged, only swapped out, and
		 * that won't affect the overall amount of available
		 * memory in the system.
		 */
		free -= global_page_state(NR_SHMEM);

		free += get_nr_swap_pages();

		/*
		 * Any slabs which are created with the
		 * SLAB_RECLAIM_ACCOUNT flag claim to have contents
		 * which are reclaimable, under pressure.  The dentry
		 * cache and most inode caches should fall into this
		 */
		free += global_page_state(NR_SLAB_RECLAIMABLE);

		/*
		 * Leave reserved pages. The pages are not for anonymous pages.
		 */
		if (free <= totalreserve_pages)
			goto error;
		else
			free -= totalreserve_pages;

		/*
		 * Reserve some for root
		 */
		if (!cap_sys_admin)
			free -= sysctl_admin_reserve_kbytes >> (PAGE_SHIFT - 10);

		if (free > pages)
			return 0;

		goto error;
	}

	allowed = vm_commit_limit();
	/*
	 * Reserve some for root
	 */
	if (!cap_sys_admin)
		allowed -= sysctl_admin_reserve_kbytes >> (PAGE_SHIFT - 10);

	/*
	 * Don't let a single process grow so big a user can't recover
	 */
	if (mm) {
		reserve = sysctl_user_reserve_kbytes >> (PAGE_SHIFT - 10);
		allowed -= min_t(long, mm->total_vm / 32, reserve);
	}

	if (percpu_counter_read_positive(&vm_committed_as) < allowed)
		return 0;
error:
	vm_unacct_memory(pages);

	return -ENOMEM;
}

红帽犯过这个错误 ：https://access.redhat.com/solutions/665023

OOM-killer
OMM killer 机制：linux 会为每个进程算一个 score，当内存不足时候它会根据 score kill
score 越大表示越容易杀死
#!/bin/bash

while read -r pid comm
do
    printf '%d\t%d\t%s\n' "$pid" "$(cat /proc/$pid/oom_score)" "$comm"
done < <(ps -e -o pid= -o comm=) | sort -k2 -n

/proc/${pid}/oom_score_adj, 取值范围为-1000到1000， 如果将该值设置为-1000，则进程永远不会被杀死 --> pg
/proc/${pid}/oom_adj, 取值是-17到+15，取值越高，越容易被干掉。如果是-17，则表示不能被kill
/proc/${pid}/oom_score, 是系统综合进程的内存消耗量、CPU 时间(utime + stime)、存活时间(uptime - start time)和 oom_adj 计算出的，消耗内存越多分越高。
https://elixir.bootlin.com/linux/v5.4.58/source/mm/oom_kill.c
数据库如何设置，如何避免被 oom-killer

short answer for redis -> echo 1 > /proc/sys/vm/overcommit_memory :)
for PostgresSQL or Greenplum -> echo 2 > /proc/sys/vm/overcommit_memory :)

why
redis:

The Redis background saving schema relies on the copy-on-write semantic of the fork system call in modern operating systems: Redis forks (creates a child process) that is an exact copy of the parent. The child process dumps the DB on disk and finally exits. In theory the child should use as much memory as the parent being a copy, but actually thanks to the copy-on-write semantic implemented by most modern operating systems the parent and child process will share the common memory pages. A page will be duplicated only when it changes in the child or in the parent. Since in theory all the pages may change while the child process is saving, Linux can't tell in advance how much memory the child will take, so if the overcommit_memory setting is set to zero the fork will fail unless there is as much free RAM as required to really duplicate all the parent memory pages. If you have a Redis dataset of 3 GB and just 2 GB of free memory it will fail.

Setting overcommit_memory to 1 tells Linux to relax and perform the fork in a more optimistic allocation fashion, and this is indeed what you want for Redis.
PostgresSQL or Greenplum:

Why Use Strategy 2?
The issue with the default overcommits strategy, or using strategy 1 is the unpredictable nature of the failure. In either case we are not guaranteed that the memory is available at time of allocation and that results in unpredictable termination of processes. In particular, the nature of the failure with strategy 1 can result in corruption of either datafiles or transaction logs as a memory failure can occur mid-transaction resulting in the immediate shutdown of the database process without any cleanup.

那如果 redis 和 pg 跑在一台机器上有什么好的建议么？

红帽犯过这个错误 ：https://access.redhat.com/solutions/665023

比起截图，是不是 quote 文字会好一些？，可以全放在 > 里

@wey-gu 好滴。


Last point of map-matched output lies too far from actual route

Map matching of some .gpx files yields a good result, however, a point is very often added at the end, which lies very far from the actual route.
In this file four .gpx files are included.  output_bad.gpx, output_good.gpx are the files that are going to be map matched (should have changed the name). output_bad.gpx.res.gpx and output_good.gpx.res.gpx contain the map matched routes.
Run this with the two files output_bad.gpx, output_good.gpx in folder.
java -jar ~/graphhopper_new/map-matching/matching-web/target/graphhopper-map-matching-web-1.0-SNAPSHOT.jar match ~/folder/*.gpx
Both files yield a good result, however output_bad.gpx.res.gpx appends a point, which lies hundreds of kilometers away. This behavior is noticed very often.
Wrong output
output_bad.gpx is red (input for map matching)
output_bad.gpx.res.gpx is blue (output of map matching)

Expected ouput
output_good.gpx is red (input for map matching)
output_good.gpx.res.gpx is blue (output of map matching)

Running on Ubuntu 18.04

Ah, no, wait, I got it. Thanks.

Seems to have magically fixed itself -- I get it on the version you were working from, but don't get it on 1.0 / current master (as of today).
Can you confirm?

Sorry for not replying; will test in the following days and let you know if the problem persists


Define schemas in a separate package

Hello!
For the sake of sharing code between backend and frontend, I'd like to define my schema files outside of the package that uses ferry_generator.
Would it be possible to support in the build.yaml a schema that comes from a different package?

Hi!
Just for clarification:
You want to be able to do something like

have a separate package that contains the graphql schema
a separate fronted and backend package, which both depend on the schema package, but their own graphql operations

Is this correct?
I'll look into supporting this, should be quite easy, unless there is an unexpected issue with built_value serialization

That is correct, yes.

a draft PR is available at #606.
still needs a cleanup and tests, but you can already try it out by using

ferry_generator:
   git:
     url: https://github.com/gql-dart/ferry.git
     path: packages/ferry_generator
     ref: external_schema_import

this was released in ferry_generator 0.12.0


refactor to use hmm-lib 1.0.0

I just published a pre-release (snapshot) of hmm-lib 1.0.0.
Using this new version in Graphhopper map-matching would bring several improvements:

The API was redesigned to allow calling the Viterbi algorithm iteratively. This allows using single-source multi-target routing, which should result in a speed-up. Moreover, if an HMM break occurs, the GPS trace can be split at the break and both sub traces can be matched separately.
The paths between matched positions (the entire matched route) can now be retrieved after computing the most likely sequence using transition descriptors. Hence, there is no need anymore to store all paths between subsequent candidates.
Memory footprint is reduced from O(t*n²) to O(t) for typical GPS traces, where t is the number of  time steps and n is the number of candidates per time step.

The project offline-map-matching, branch-1.0.0 is now also using the hmm-lib 1.0.0-SNAPSHOT. In there, OfflineMapMatcherTest.java provides an example how to use the new hmm-lib API, which can be used as a template for Graphhopper map-matching.
Feedback is welcome!

Fixed via #66


Implement K-Means algorithm

An Unsupervised classification machine learning algorithm.
Totally blanked on kNN vs kMeans and implemented the wrong one. So rather than waste code, moved it over to a new branch and made a few of the comments from PR #48.
Opening pull request to get feedback on code.
Todos:

[x] Write README.md

Hey, more algorithms is better! :-D

@hollance Can I get a review of code and README for potential merge?

Some minor points:
I would call VectorND just Vector. I also wouldn't call the methods getData() and getLength(). That's what Java people do. ;-) Instead, you can declare the data and length properties as private(set) public. That way you don't even need the methods.
I prefer to use a separate random() function instead of using arc4random_uniform() directly, since it can be hard to figure out exactly what range arc4random_uniform() works on.
It's best not to use ++ since this is going away in Swift 3.0. += 1 makes the code future-compatible.
In the playground, when the code becomes large I usually add it as a separate source file. That keeps the actual playground cleaner.

I changed Vector into it's own source file. It could be expanded upon and maybe even moved to it's own directory as a Data Structure.
I'm still learning swift which is why I found this repo to be enjoyable. Refresh myself on algorithms and implement in new language. Still learning some of the nuances of swift over other OO languages.
Made change to use random() and modulus, as well as removed all '++'.
Changed code from a playground to a project with a few tests.

Great, I'll look into this soon. I agree that Vector deserves a topic of its own. I've been writing my own Matrix class for doing things like logistic regression. It uses the Accelerate framework to make things really fast. Maybe we could do something similar for Vector.

OK, I finally had a look at it (and edited the text and code a little). It would be great if there was a bit more explanation of how the code actually works. The illustrations are very useful already, but a step-by-step explanation of the logic would also be good to have. :-)

@hollance wrote Issue to track it.


SB: documentation updated

Description
Changes proposed in this pull request:

Status codes for DELETE in case of suspension/deprovisioning, non-existing instance
Status codes for GET in case of suspension, non-existing instance

Related issue(s)

The whole PUT request should also be updated. It still has this part accepts_incomplete=false&service_id={{service_id}}&plan_id={{plan_id}} which should be removed.

Maybe we should also mention in the PUT, that this request also returns metadata with the expires_at field.


add append change

able to also append data to an existing table

Thanks @dfernan! I'll take care of the append = True when I get a chance.

I didn't realize that the copy command already appended if the table existed. Thanks!

great thanks! really useful library.

It's update on PyPi now. You should be able to pip install it with your additional changes.

wow this was fast.  I got the new version running and working.. thanks so much!


Check user traffic volume usage

Hello,
How can we check the amount of traffic consumed by each user ?
Thanks.

You can use Anylink with Daloradius to do that. I'm undergraduate so I don't have much time.


Add installation hints for Windows

thanks, i'll add an issue to make the manual script work on Windows as well


Unable to load channels with 1.0 (34)

Summary
When switching channels, the channel intermittently fails to load
Environment Information

Device: iPhone8,1 (6S)
Device Name: iPhone
OS Version: 10.3.2
Mattermost App Version: 1.0 (34)
Mattermost Server Version: 3.8.0

Steps to reproduce

Switch between various open channels, unable to reproduce issue
Switch between various private channels, issue can sometimes be reproduced

I could reproduce it consistently yesterday, but today at time of writing, I'm not able to reproduce it at all.
Expected behavior
When switching between various channels, I expect to see the placeholder circle and rectangles and then have that fade away to the conversation at hand.
Observed behavior
When switching between various channels, I see the placeholder circle/rectangles, but it does not fade away to the conversation.
Possible fixes
I've suggested that server maintainer update to MatterMost 3.10.0.

Hi @WaLLy3K - thanks for the report!
We've had a couple other people seeing similar things - we think we may have fixed it as a side effect of this PR: https://github.com/mattermost/mattermost-mobile/pull/653
We'll be cutting a new build in the next day or two, it would be great if you could help test it once it's out and let us know if you see the issue anymore or not.

I can confirm I'm still seeing the issue, so I'll definitely hit that up when I see an update in TestFlight! Thanks @lfbrock

I think it's about InteractionManager of channel_drawer.
Sometimes, callbacks of runAfterInteractions won't be called after clicking a channel item, so posts of that channel won't be loaded.
This is because of the reassignment of openLeftHandle. When swiping right to open the channel list, handleDrawerOpenStart will be called twice at the same time in some cases. That is InteractionManager.createInteractionHandle will be called twice,  and this.openLeftHandle is assigned twice.
After drawer's open action end, handleDrawerOpen will be called, however, only the last handle can be cleared by calling InteractionManager.clearInteractionHandle.
Here is a screenshot. Notice that handler 27 will never be cleared, so that React Native think animation is not finished yet and callback of runAfterInteractions won't be called.

~ignore typo of the function names~
I don't know why this happens. I didn't make further test.
Possible Solutions
The first solution
Use if to avoid reassignment, and make handler null after clearing.
In handleDrawerOpenStart:
        if (!this.openLeftHandle) {
            this.openLeftHandle = InteractionManager.createInteractionHandle();
        }

in handleDrawerOpen
        setTimeout(() => {
            if (this.openLeftHandle) {
                InteractionManager.clearInteractionHandle(this.openLeftHandle);
                this.openLeftHandle = null;
            }
        });

The second solution
Remove codes about InteractionManager.
On my iPhone 6s, Mattermost works well. In fact, the first solution will cause some delay after clicking on channel item. The drawer started to close not immediately but about 300ms after clicking.  Didn't test on other devices, don't know is there any problem on other devices.

Thanks guys for the report and a possible solution, there is a PR with a fix here https://github.com/mattermost/mattermost-mobile/pull/665/files#diff-8adfe070b0e342ec9f0fb3f1b8f354ac
@Witcher42 please feel free to make a PR whenever you see it fit, it will be much appreciated ;)

Closing via #655


settings, local_settings

settings.py is used to centralize settings and local_settings.py uses clever override behavior to enable having environment specific settings on a server that does not get committed to source control (git). This means that you can have parameterized settings that a respective to laptops, dev server, UAT, QA, beta and prod. For example, dev keys or empty keys can be specified in local_settings in Jenkins jobs as well as in deployment scripts, whereas protected keys can live in secure areas of the filesystem or as Linux environment variables.
Investigate usage of local_settings.py, local_settings.template and local_settings.cutover across Jenkins and dev-hs-## environments, beta, prod and all deployment automated and manual (tribal/documented/undocumented) steps and also coordinate with RENCI to update/remove as necessary. Scott should probably approve any final changes, before going live, especially if the changes are outside source control and developers don't necessarily see them.

I no longer have a need for this. DevOps has their own methods for handling this across environments.

I no longer have a need for this. DevOps has their own methods for handling this across environments.


Push image to docker registry with s3 storage cause HTTP 500 error

Docker registry version: v2.6.2
When push images to a registry with s3 storage, it end with HTTP 500 error.
docker push /library/alpine:latest
The push refers to a repository [/library/alpine]
7bff100f35cb: Pushed
received unexpected HTTP status: 500 Internal Server Error
In the registry log we found that it failed to validate the digest.  the digest is an empty string.
But actually the digest is not empty:
HEAD /v2/library/alpine/blobs/sha256:cd784148e3483c2c86c50a48e535302ab0288bebd587accf40b714fffd0646b3
Registry log:
registry.log

@stonezdj - experienced the same, working fine with 2.7.1 though.

@dtucny Tried registry v2.7.1 still got HTTP 500 error

Can you please verify if this is still the case on the latest release.

Closing as outdated and lack of noise from the OG reporter.


zsh not initialized correctly (ssh logins)

Basic info
To make sure that we are on the same page:

Kernel: (run uname -a) Linux lapole 3.18.25 #1-NixOS SMP Thu Jan 1 00:00:01 UTC 1970 x86_64 GNU/Linux
System: (NixOS: nixos-version, Ubuntu/Fedora: lsb_release -a, ...) 15.09.952.55b0326 (Dingo)
Nix version: (run nix-env --version) nix-env (Nix) 1.10
Nixpkgs version: (run nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion) 16.03pre75806.77f8f35

Describe your issue here
I installed zsh-5.2 from nixpkgs channel to get away from a 5.0.8 bug. After I changed my login-shell to 5.2, ssh logins were no longer functional. Ie. PATH wasn't fully initialized.
Reverting to 5.0.8 fixed the problem.
I'm not sure why there's a difference between regular login shells and ssh though.
Running strings /nix/store/*zsh-5.2/bin/zsh yields:
/etc/profile
.profile`
/etc/suid_profile
$out/etc/zprofile
.zprofile
profile signal

For 5.0.8 I get the expected /nix/store/... path instead of the strange $out path.
The line with $out seems wrong.. Checking the commit log I find https://github.com/NixOS/nixpkgs/commit/97332d30f62b149aa81a63f798e7e53825e12663
Dropping into nix-shell for zsh-5.2, setting set -x and manually running configurePhaseI get:
+ ./configure --prefix=/nix/store/icd9hwav49fign863szpk4s3imn5dm2b-zsh-5.2 <snip>
--enable-multibyte '--enable-zprofile=$out/etc/zprofile' <snip>

So it seems to me that variables in configureFlags is not expanded leading to zprofile not being run. It could be something else of course (not everything seems to add up without more investigating), but the $out thing seems wrong either way.

cc @calbrecht

This is fixed by the merge of #14343. Cherry-picked to release-16.03.


Bug due to typo, can't create an author.

Hey there, super neat repo and a nice introduction to the topic of GraphQL. I was actually using this to introduce an intern to the concept. It's been very helpful to quickly put a presentation together. I did notice one small bug. Due to a typo, when adding an author, you're not actually getting the length from the authors array. Instead, you've got "author" which throws the error "Cannot access 'author' before initialization".
Here's the whole changelog to fix it:

Hi @matthew-powaschuk-t thank you so much for your message and report.
The reported bug has been solved.

Super neat repo and a nice introduction to the topic of GraphQL. I was actually using this to introduce an intern to the concept. It's been very helpful to quickly put a presentation together

It's awesome.


Duplicate content

This page has the same text as this one.

Good catch and thanks @panstav - fixed.

Great, that extra paragraph is extremely helpful.
Thanks


Updating Doc with required permissions for SSH and WinRM for SSH

Harness Developer Pull Request
Thanks for helping us make the Developer Hub better. The PR will be looked at
by the CODEOWNERS.
What Type of PR is This?

[ ] Issue
[ ] Feature
[ ] Maintenance/Chore

If tied to an Issue, list the Issue(s) here:

Issue(s)

House Keeping
Some items to keep track of. Screen shots of changes are optional but would help the maintainers review quicker.

[ ] Tested Locally
[ ] Optional Screen Shoot.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://657b3886085eee00744572c9--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://657b38f4654dfe007617c0a2--harness-developer.netlify.app


[Core] in batch prefix caching by delay scheduling

Motivation
Implement in-batch prefix caching.
If a new request has a short enough matching prefix, and if there are more than 1 such request that share the same prefix, we deprioritize them except one of them.
The feature benefits the workloads with many shared prefix prompts. Here's the extreme example we can be benefited from this feature.
    prompts = [
        *["Hello, my name is " * 50 + chr(i) for i in range(65, 87)],
        *[
            "The president of the United States is " * 50 + chr(i)
            for i in range(65, 87)
        ],
        *["capital of France is " * 50 + chr(i) for i in range(65, 87)],
        *["The future of AI is " * 50 + chr(i) for i in range(65, 87)],
        *["dkfjklwerkj lskjdfsa " * 50 + chr(i) for i in range(65, 87)],
        *["What time is it now? " * 50 + chr(i) for i in range(65, 87)],
        *["hello it's good to see you " * 50 + chr(i) for i in range(65, 87)],
        *["xai is the " * 50 + chr(i) for i in range(65, 87)],
        *["oops " * 50 + chr(i) for i in range(65, 87)],
        *["hello it's me " * 50 + chr(i) for i in range(65, 87)],
    ]

With this prompt, we can see 2X+ perf improvement
 after
e2e takes<PHONE_NUMBER>225211 ms
# before
e2e takes 1828.76543502789 ms

I also added bench_prefix.py script which is originally written from @Ying1123. With prefix_len=1024, gen_len=128, and sampling_size = 32, I could see 10+% performance improvement
# lpm-2
Throughput: 94.79 requests/s, 109553.50 tokens/s
# lpm
Throughput: 83.51 requests/s, 96514.85 tokens/s

The feature adds a slight more overhead to calc_priroity because it requires additional radix cache operation. I believe it will be negligible with overlap schedule. i also verified there's no regression from python -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10 just in case.
Modifications

Checklist

[ ] Format your code according to the Contributor Guide.
[ ] Add unit tests as outlined in the Contributor Guide.
[ ] Update documentation as needed, including docstrings or example tutorials.

Awesome enhancement! Can we use https://github.com/sgl-project/sglang/pull/1990 instead of adding a new bench script?

cc @MrAta i think this is useful for our case too

removed benchmark_prefix.py (use https://github.com/sgl-project/sglang/pull/1990 instead)
Support dfs-weight. For the same extreme prompts, I see 2x improvement

# dfs-weight after
e2e takes<PHONE_NUMBER>101852 ms
# dfs-weight before
e2e takes 1497.2664599772543 ms

Does #1990 also test the in-batch performance? We need a test to make sure this new algo is similar or better than using "prompt hint" - insert the shared part ahead of the batch.

This looks great!

@merrymercy The current policy is if we don't have more requests to schedule, it just schedule it together (instead of delaying it), but @merrymercy pointed out it should still schedule 1 request and delay others. We will revert this PR and merge again with this fix.


loss calculation

The loss calculation during training uses the probability density function loss of all targets in context and predict, rather than just predict. Why?

So, in all decoder-only models, the "predict" part actually makes no sense during training, as all the points are used for the loss. Note that this is for all decoder-only models such as GPT.
In the code, a context and predict are separately used to support inference for a certain specific prediction length, given a certain fixed length context.

So, in all decoder-only models, the "predict" part actually makes no sense during training, as all the points are used for the loss. Note that this is for all decoder-only models such as GPT.
In the code, a context and predict are separately used to support inference for a certain specific prediction length, given a certain fixed length context.

Thank you very much for your reply. I got it！


Night CI 2023-02-25: Failed, Get "https://<IP_ADDRESS>:37467/apis/spiderpool.spidernet.io/v1/spiderippools xxx": context deadline exceeded

action url: https://github.com/spidernet-io/spiderpool/actions/runs/4271666184

{"level":"ERROR","ts":"2023-02-25T20:35:04.903Z","logger":"spiderpool","caller":"cmd/command_add.go:127","msg":"[POST /ipam/ip][500] postIpamIpFailure  failed to allocate IP addresses in standard mode: failed to mark IP allocation: spiderendpoints.spiderpool.spidernet.io \"ds-3440-425830407-fx84k\" is forbidden: unable to create new content in namespace ns3438-326662282 because it is being terminated","Action":"Add","ContainerID":"b384a9b975d9a78ad95061bc06f93a3698f75ac482806bdd3b708c330fd47239","PodUID":"","PodName":"ds-3440-425830407-fx84k","PodNamespace":"ns3438-326662282","IfName":"eth0"}

Ignore this issue during code refactoring.