redis-cluster attempting to connect to a stale redis server IP and not recovering

Hi @bjosv,
We came across a case wherein hiredis-cluster seems to be trying to connect to a stale redis-server ip while issuing commands to redis server, this connection flapping kept on happening (approx 2hrs) until one more redeploy was done for the application image (which has redis-cluster integrated).
From the server-side logs, it looks like this ip was alive earlier for some period of time in the past, but not for the previous couple of hours atleast when the issue was seen. There was an image upgrade done on the redis server nodes which could have resulted in the change of ips.
As part of the callbacks registered by application with hiredis, the below prints were seen as part of the redis command send calls.
[2023-01-12T07:47:30.263102][disconnectCallback]Error: Server closed the connection

...connectCallback print for <IP_ADDRESS>

[2023-01-12T07:48:26.189608][disconnectCallback]Error: Timeout

...connectCallback print for <IP_ADDRESS>

[2023-01-12T07:48:28.191117][disconnectCallbackk]Error: Timeout

...connectCallback print for <IP_ADDRESS>

[2023-01-12T07:48:34.186333][disconnectCallback]Error: Connection reset by peer

...connectCallback print for <IP_ADDRESS>

[2023-01-12T09:12:53.184634][disconnectCallback][Disconnected from <IP_ADDRESS>:6379

kubectl get pods output was collected when the issue was seen, and at that point the list of ips are as captured below. 3 primary nodes and 2 replica per node (adds up to total 9 nodes) as shown below. Here, the ip which redis-cluster is trying to connect - <IP_ADDRESS> is not appearing.
  1 NAME                                READY   STATUS    RESTARTS   AGE    IP                NODE                                    NOMINATED NODE   READINESS GATES
  2 rediscluster-node-for-redis-9cj4g   2/2     Running   0          169m   <IP_ADDRESS>   cl-60-node-5-dcc808-4b2c09f7b029c460    <none>           <none>
  3 rediscluster-node-for-redis-9vvzf   2/2     Running   0          165m   <IP_ADDRESS>     cl-60-node-9-dcc808-4b2c09f7b029c460    <none>           <none>
  4 rediscluster-node-for-redis-h5lw6   2/2     Running   0          168m   <IP_ADDRESS>    cl-60-node-4-dcc808-4b2c09f7b029c460    <none>           <none>
  5 rediscluster-node-for-redis-mx8gs   2/2     Running   0          168m   <IP_ADDRESS>    cl-60-node-1-dcc808-4b2c09f7b029c460    <none>           <none>
  6 rediscluster-node-for-redis-qwrqg   2/2     Running   0          168m   <IP_ADDRESS>     cl-60-node-6-dcc808-4b2c09f7b029c460    <none>           <none>
  7 rediscluster-node-for-redis-slbj7   2/2     Running   0          166m   <IP_ADDRESS>    cl-60-node-3-dcc808-4b2c09f7b029c460    <none>           <none>
  8 rediscluster-node-for-redis-t92lx   2/2     Running   0          169m   <IP_ADDRESS>     cl-60-node-4-dcc808-4b2c09f7b029c460    <none>           <none>
  9 rediscluster-node-for-redis-wgbdq   2/2     Running   0          166m   <IP_ADDRESS>    cl-60-node-10-dcc808-4b2c09f7b029c460   <none>           <none>
 10 rediscluster-node-for-redis-z5dr7   2/2     Running   0          169m   <IP_ADDRESS>   cl-60-node-2-dcc808-4b2c09f7b029c460    <none>           <none>

Do you have some clue on what could be resulting in this behaviour?

Hi, can someone please take a look at this one? We are a bit blocked with this currently.

That sounds strange.
I guess you have checked the individual nodes by running the command CLUSTER NODES (or CLUSTER SLOTS) individually on them to make sure they are all in sync.
Do you set the max retry config to a specific value (using redisClusterSetOptionMaxRetry()) or is it default? (in async API that means 5 errors starts a timer to fetch topology).
Do you have/see any logs about failed command callbacks, which then would trigger a new topology fetch?

Hi @bjosv
Yes, cluster nodes command was separately run on the redis-server when issue happened and ouput(captured below) is matching with the kubectl o/p.
>>> Calling cluster nodes
redis-cluster.local:6379: 51c49c0ba2f1c33456dafc1654e4969c8ab25416 <IP_ADDRESS>:6379@16379 slave 6d68da34389cb97e70e4ee8075108f6394847446 0<PHONE_NUMBER>113 29 connected
6fde1a7b2f6ed7355066534e8a1656f0f9a0e988 <IP_ADDRESS>:6379@16379 slave 8035f1c9c0a7ade4a303381e492ff2b85dbe5f96 0<PHONE_NUMBER>598 35 connected
8035f1c9c0a7ade4a303381e492ff2b85dbe5f96 <IP_ADDRESS>:6379@16379 master - 0<PHONE_NUMBER>000 35 connected 10924-16383
7061695cb989db63cf08cecf5dcdcee7293a7c4e <IP_ADDRESS>:6379@16379 myself,master - 0<PHONE_NUMBER>000 34 connected 2-5463
3cb872f1153c39f9e1b5ce3f1ddf2136a2558bba <IP_ADDRESS>:6379@16379 slave 7061695cb989db63cf08cecf5dcdcee7293a7c4e 0<PHONE_NUMBER>088 34 connected
6d68da34389cb97e70e4ee8075108f6394847446 <IP_ADDRESS>:6379@16379 master - 0<PHONE_NUMBER>100 29 connected 0-1 5464-10923
b138088411b13d8db3b855ca454d3f7881065d39 <IP_ADDRESS>:6379@16379 slave 7061695cb989db63cf08cecf5dcdcee7293a7c4e 0<PHONE_NUMBER>089 34 connected
5e99ed9c02ae3daecabbef44d05563f502efc6eb <IP_ADDRESS>:6379@16379 slave 6d68da34389cb97e70e4ee8075108f6394847446 0<PHONE_NUMBER>000 29 connected
b2c8cdd82c52c27ecda5c44a121e21ebb5d2406a <IP_ADDRESS>:6379@16379 slave 8035f1c9c0a7ade4a303381e492ff2b85dbe5f96 0<PHONE_NUMBER>094 35 connected

We dont set redisClusterSetOptionMaxRetry() explicitly. So default value should be set i guess. I couldnt find any logs which suggests that we ran cluster nodes command, and i suppose if it were run it would self resolved.
One more point to add is, when hiredis calls back into application during tcp disconnect, application doesnt do anything with the redisAsyncContext* pointer that is sent back (I mean no delete of object or anything). I hope that is fine.

detailed log set from begining:
begining of connection with redis.

{"severity":"debug","debug-string":"[2023-01-12T07:38:41.584874][connectCallback][98]UDSF connectCallback invoked
"}
{"severity":"debug","debug-string":"[2023-01-12T07:38:41.584888][connectCallback][114]Connected to <IP_ADDRESS>:6379
"}

{"severity":"debug","debug-string":"[2023-01-12T07:45:45.904981][connectCallback][98]UDSF connectCallback invoked
"}
{"severity":"debug","debug-string":"[2023-01-12T07:45:45.905107][connectCallback][114]Connected to <IP_ADDRESS>:6379
"}

{"severity":"debug","debug-string":"[2023-01-12T07:47:30.263102][disconnectCallback][149]Error: Server closed the connection
"}

{"severity":"debug","debug-string":"[2023-01-12T07:48:24.185349][connectCallback][98]UDSF connectCallback invoked
"}
{"severity":"debug","debug-string":"[2023-01-12T07:48:24.185751][connectCallback][114]Connected to <IP_ADDRESS>:6379
"}

{"severity":"debug","debug-string":"[2023-01-12T07:48:26.186998][CommandCb][1480]Redis command timeout
"}

{"severity":"debug","debug-string":"[2023-01-12T07:48:26.189608][disconnectCallback][149]Error: Timeout
"}
{"severity":"debug","debug-string":"[2023-01-12T07:48:26.190408][connectCallback][98]UDSF connectCallback invoked
"}
{"severity":"debug","debug-string":"[2023-01-12T07:48:26.190434][connectCallback][114]Connected to <IP_ADDRESS>:6379

{"severity":"debug","debug-string":"[2023-01-12T07:48:28.191117][disconnectCallback][149]Error: Timeout
"}
{"severity":"debug","debug-string":"[2023-01-12T07:48:32.200001][connectCallback][98]UDSF connectCallback invoked
"}
{"severity":"debug","debug-string":"[2023-01-12T07:48:32.200195][connectCallback][114]Connected to <IP_ADDRESS>:6379
"}
{"severity":"debug","debug-string":"[2023-01-12T07:48:34.185965][CommandCb][1480] Redis command timeout
"}

{"severity":"debug","debug-string":"[2023-01-12T07:48:34.186333][[disconnectCallback][149]Error: Connection reset by peer
"}
{"severity":"debug","debug-string":"[2023-01-12T08:39:34.672284][[connectCallback][98]UDSF connectCallback invoked
"}
{"severity":"debug","debug-string":"[2023-01-12T08:39:34.672358][[connectCallback][114]Connected to <IP_ADDRESS>:6379
"}

{"severity":"debug","debug-string":"[2023-01-12T08:39:36.675527][disconnectCallback][149]Error: Timeout
"}
{"severity":"debug","debug-string":"[2023-01-12T08:53:21.596753][connectCallback][98]UDSF connectCallback invoked
"}
{"severity":"debug","debug-string":"[2023-01-12T08:53:21.596809][connectCallback][114]Connected to <IP_ADDRESS>:6379
"}

{"severity":"debug","debug-string":"[2023-01-12T08:53:23.599153][disconnectCallback][149]Error: Timeout
"}
{"severity":"debug","debug-string":"[2023-01-12T09:11:27.937986][connectCallback][98]UDSF connectCallback invoked
"}
{"severity":"debug","debug-string":"[2023-01-12T09:11:27.938091][connectCallback][114]Connected to <IP_ADDRESS>:6379
"}

there are application commands that are timing out in bwn

Hi @bjosv,  You have any thoughts on this?
One other thing which i wanted to understand is that while client is connected to redis-server, and there is a slot redistribution, how will redis-cluster get to know of that? I guess tcp connection won't be broken from server side for slot redistribution, right?

One other thing I saw while going through the code is that max_retry_count is checked for making cluster commands call is done in following methoods.
__redisClusterGetReplyFromNode()
redis_cluster_command_execute()
redisClusterAsyncRetryCallback()
But in this scenario mentioned in the ticket, I don't think we are going through any of these methods. So it looks like max_retry_count never gets compared against which could explain why we are remaining/stuck in the same state.

I only see 2 CommandCb logs, which if they are errors would increases the internal failure counter.
The async legacy code expects 5 send errors (default is 5) before it attempts to renew the slot distribution, so that might be a reason for the logs.
One remedy could be to use redisClusterSetOptionMaxRetry(1) to only allow one retry after a MOVED/ASK/error.
Normally the query from a client would get MOVED response, which will trigger the renewal of hiredis-clusters slots distribution. In this case in the async code 5 send errors are allowed before the renewal. Each time the client tries to send the command it first attempts to connect to the redis node.

I only see 2 CommandCb logs, which if they are errors would increases the internal failure counter. The async legacy code expects 5 send errors (default is 5) before it attempts to renew the slot distribution, so that might be a reason for the logs. One remedy could be to use redisClusterSetOptionMaxRetry(1) to only allow one retry after a MOVED/ASK/error.
Normally the query from a client would get MOVED response, which will trigger the renewal of hiredis-clusters slots distribution. In this case in the async code 5 send errors are allowed before the renewal. Each time the client tries to send the command it first attempts to connect to the redis node.

Ya there are more than 5. I just pasted logs from top of the logfile. If you see my initial post, it goes on for >2hrs. But one thing to note here is that its not actually a response coming from server application is logging, we have configured for command_timeout of 2sec, and since no response is seen for that period, there is a timeout happening which is getting logged by the application with CommandCb tag.
And as i mentioned above, the places where we check for max_retry_count is mostly getting skipped as we never go through one of those flows.

Hi @bjosv do you agree with the above observation?
Is so, is there a way to introduce max_retry_count check in the flow it's taking now?

One other thing I saw while going through the code is that max_retry_count is checked for making cluster commands call is done in following methoods.
__redisClusterGetReplyFromNode()
redis_cluster_command_execute()
redisClusterAsyncRetryCallback()

But in this scenario mentioned in the ticket, I don't think we are going through any of these methods. So it looks like max_retry_count never gets compared against which could explain why we are remaining/stuck in the same state.

Hi @bjosv Correcting my statement above:
redisClusterAsyncRetryCallback() is the callback we pass to hiredis, so it should get called back for each command that is attempted.
So, whenever hiredis api redisAsyncHandleTimeout() gets called during command_timeout, hiredis sets the error to REDIS_ERR_TIMEOUT and callsback into hiredis-cluster with NULL reply. As part of this max_retry_count should have got incremented. So, I tend to agree with your statement.
But the issue here is that, if traffic is less and the key/slot never falls into the issue node, redis-cluster could just continue with the bad state for such a long time (as seen here approx 2hrs, 3 callback errors) with the default value of 5.

One more query in the same context. I was also wondering why a connect() call would succeed in such a scenario?
If the remote ip is not reachable shouldn't the connect() itself fail? The fact that application is able to print connectCallback tagged messages is an indication that the call succeeded with REDIS_OK. Do you have any thoughts on this @bjosv?

But the issue here is that, if traffic is less and the key/slot never falls into the issue node, redis-cluster could just continue with the bad state for such a long time (as seen here approx 2hrs, 3 callback errors) with the default value of 5.

Yes, I believe it was done in this way to handle heavy traffic scenarios, giving an imperfect behavior for low traffic scenarios..

I was also wondering why a connect() call would succeed in such a scenario?

Since the async api is used the connect will only register to the eventsystem, which normally wont fail.
When the eventsystem then polls an error from the socket, due to the failing connect, hiredis will/should call the connect callback with the argument status=REDIS_ERR.
Do you get a REDIS_OK in your case?

Ah, yes, I see now that you got calls to the disconnect callback, which only should be received when there has been a successful connect. Probably from <IP_ADDRESS>.
Do you have authentication enabled on Redis, using password or username/password?

Hi @bjosv

Do you get a REDIS_OK in your case?

Yes, I keep getting REDIS_OK in the connectCallbacks.

When the eventsystem then polls an error from the socket, due to the failing connect, hiredis will/should call the connect callback with the argument status=REDIS_ERR.

Isn't connect() supposed to be a blocking call? I hope you are talking about redisAsyncConnectWithOptions() which then calls redisContextConnectBindTcp() where I could see after connect() call its checking for status of connect().

Ah, yes, I see now that you got calls to the disconnect callback, which only should be received when there has been a successful connect. Probably from <IP_ADDRESS>.

But the IP is no longer present in the cluster. I was thinking there is some issue here due to which its wrongly claiming it's connecting.

Do you have authentication enabled on Redis, using password or username/password?
How about TLS?

There is no password authentication/TLS used for now.

Isn't connect() supposed to be a blocking call? I hope you are talking about redisAsyncConnectWithOptions() which then calls redisContextConnectBindTcp() where I could see after connect() call its checking for status of connect().

I would guess the status check is skipped for async.
redisAsyncConnectWithOptions() sets REDIS_OPT_NONBLOCK which results in that REDIS_BLOCK is not set on the hiredis context.
It could be that the connect fail with errno=EINPROGRESS, which is not uncommon.
hiredis net.c
static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
...
    int blocking = (c->flags & REDIS_BLOCK);
...
        if (connect(s,p->ai_addr,p->ai_addrlen) == -1) {
            if (errno == EHOSTUNREACH) {
                redisNetClose(c);
                continue;
            } else if (errno == EINPROGRESS) {
                if (blocking) {
                    goto wait_for_ready;                <--- skipped
                }

I think this can be indicated and viewed in the hiredis-cluster async tests, I will double-check.

Hmm. Yes, I guess that definitely looks like a possible gap.
However I was checking more in that flow, I see that once we come out of that else if block, we also set the flag to REDIS_CONNECTED.
c->flags |= REDIS_CONNECTED;
And when i searched for places where hiredis could callback onConnect() api with success only 1 search result came up.
which is:
610 static int __redisAsyncHandleConnect(redisAsyncContext *ac) {
611     int completed = 0;
612     redisContext *c = &(ac->c);
613
614     if (redisCheckConnectDone(c, &completed) == REDIS_ERR) {
615         /* Error! */
616         redisCheckSocketError(c);
617         __redisAsyncHandleConnectFailure(ac);
618         return REDIS_ERR;
619     } else if (completed == 1) {
620         /* connected! */
621         if (c->connection_type == REDIS_CONN_TCP &&
622             redisSetTcpNoDelay(c) == REDIS_ERR) {
623             __redisAsyncHandleConnectFailure(ac);
624             return REDIS_ERR;
625         }
626
627         if (ac->onConnect) ac->onConnect(ac, REDIS_OK);   --> only place where callback called with REDIS_OK.
628         c->flags |= REDIS_CONNECTED;
629         return REDIS_OK;
630     } else {
631         return REDIS_OK;
632     }
633 }

redisCheckConnectDone() internally does a connect() on the socket and if return code is 0 then only it should invoke the line 627.  --> Will connect continue to be non-blocking mode here ?
Also, __redisAsyncHandleConnect() is called only if flags dont have REDIS_CONNECTED set.
653     if (!(c->flags & REDIS_CONNECTED)) {
654         /* Abort connect was not successful. */
655         if (__redisAsyncHandleConnect(ac) != REDIS_OK)
656             return;

But if errno == EINPROGRESS block had got hit, the flag would have been already set, and we would never invoke the onConnect() callback.
I'm thinking now is __redisAsyncHandleConnect() at fault here ?

Just as a note, there is this in redisAsyncInitialize()  called just after redisConnectWithOptions() in hiredis redisAsyncConnectWithOptions():
https://github.com/redis/hiredis/blob/6f5bae8c6900e051da6e677756508707565ce56e/async.c#L124-L127

Ya, I suppose I missed noting this. :)

Ok, still trying to understand where would be the REDIS_OK connectedCallback be called from.
I guess when there is a redisAsyncHandleWrite() or a redisAsyncHandleRead() operation, it will invoke __redisAsyncHandleConnect() which inturn calls connect() and connect should returns back 0 ? This is the only place where it could invoke connected callback with REDIS_OK right ?

Exactly what I think, __redisAsyncHandleConnect()  should return REDIS_OK, and completed=1, that is the only way to set the REDIS_CONNECTED flag.
So, somehow it managed that.
Could it be that the K8s pod still exists with the redis instance, but its not included in the redis cluster?

Ya this flapping went on for atleast 2hrs. So very unlikely that could be the case.
Just thinking if there could be any chance connect() can return back a 0 for a non-success case.

Also @bjosv, back to the previous discussion we were having, I was just checking more on the max_retry_count parameter.
One thing I noticed was that the way the increment is happening is:
static void redisClusterAsyncRetryCallback(redisAsyncContext *ac, void *r,  void *privdata) {
node = (cluster_node *)ac->data;
.
node->failure_count++;
if (node->failure_count > cc->max_retry_count) {
.
.
}

But, ac is getting reset back to default with every new redisClusterAsyncCommand() call to a non-connected node at the time of redisAsyncInitialize() right ?
Wouldn't this be a problem now, since we wont be able to track this anymore ?

There should be a connectCallback (connect attempt) for each CommandCb that timed out towards IP x.26.
Do you see more than 5 timed out CommandCb's?

Yes, in the logfile i am looking at i could see in this time interval application Timeout error has occurred for 7 times.

Just one other thing that I thought of:
You ran the cluster nodes command manually to check the redis cluster, ie the log above with >>> Calling cluster nodes. Did you run this command on each node to check each nodes view of the cluster?
This would just making sure they are all in sync since hiredis-cluster can get its slot distribution from any master node, which might not be the one you send the command to.
Probably unlikely a problem though..but it depends on how your K8s Redis operator removes a node (if it send CLUSTER FORGET to all other nodes).

Which operator are you using btw?
KubeDB, AmadeusITGroup/Redis-Operator/.. or own?

Hi @bjosv
cluster nodes command is run as below on redis-cluster kubernetes service FQDN.
redis-cli --cluster call redis-cluster.local:6379 cluster nodes
It is not run for all nodes separately. OK, I see you think operator could also be involved in this. We are using IBM redis-operator for this.

The root problem that a timeout was not triggering a slotmap update is corrected by #144.
The reason for hiredis to be able to connect to a cluster-node that shouldn't be available looks like a Redis operator issue.
Please reopen with more info if that problem can be blamed on hiredis-cluster.


Triggering showAdControls_ and hideAdControls_ on mobile device

On mobile devices is there a way to toggle ad-controls on tap or press events?
I used to run the publicly available ima.hideAdControls_ on initiation, but since making these private I've had to hack it.
It's only now I dig around I can see we're only using the showAdControls and hideAdControls_ on mouse events. Do you think we should be building for at least a touchStart before we show the ad controls?

I think mobile controls were hiding and showing fine when we first released this >2 years ago, but it looks like that has since changed - testing on my android device just shows the controls constantly and they're never hidden. I can look into adding touch event listeners as well, but we'll have to do it in a way that doesn't interfere with the current desktop control functionality.

@shawnbuso thanks for taking a look.
I was having a bit of a mind-blank as to what the ad-control behaviour should actually be, but I've just been playing with bog-standard <video src="..." controls> tag on an Android/Chrome device:

Controls display by default

Probably not applicable for ads, and could detract from the content, particular for ads with a short skipoffset

Controls fade-out after ~5s

Suspect could be shorter for ads without being detrimental to the user experience.

Controls reappear on user-engagement with the video

Does that sound about right to you?
It might be a few weeks, but can take a look if you haven't got to it.

My current plan is to just have it behave the same way as desktop, which is 2+3 in the list above. I'm a bit tied up with other things right now, though, so it will probably be a bit till I get to it as well.

:+1:

FYI I've decided on a slightly different way of handling this - since IMA pauses and resume the ad on tap on mobile, I've set it up to just show the controls on ad pause and hide them on ad resume (on mobile). So when you tap the ad pauses and the controls show, and the controls stay visible until you resume the ad either by hitting the play button or by tapping the ad again.

@shawnbuso that feels intuitive. Thanks a bunch for taking a look!


Used Feed Functional?

After trying to repeat the bug with the user feed I could not get it to fail again.

Just tested locally and on the server the set of actions that caused failure worked this time.

This is a bug related to the sessionId becoming invalid and thus the reviews can't be retrieved from the API. I will fix the error handling on the client side to properly reflect when the session becomes invalid


Add pyproject.toml for Custom Node Registry

We are working with dr.lt.data and comfyanon to build a global registry for custom nodes (similar to PyPI). Eventually, the registry will be used as a backend for the UI-manager. All nodes go through a verification process before being published to users.
The main benefits are that authors can

publish nodes by version and users can safely update nodes knowing ahead of time if their workflows will break or not
automate testing against new commits in the comfy repo and existing workflows through our CI/CD dashboard

Action Required:

[ ] Go to the registry. Login and create a publisher id. Add the publisher id into the pyproject.toml file.
[ ] Write a short description.
[ ] Merge the separate Github Actions PR and run the workflow.

If you want to publish the node manually, install the cli and run comfy node publish
Check out our docs if you want to know more about the registry. Otherwise, feel free to message me on discord at haohao_81202 or join our server if you have any questions!

hey @foxtrot-roger, wanted to send a small nudge and ask if you had any questions!

@foxtrot-roger, wondering if there's any way we can support you on this PR!


[Origin Station] Brig hallway has generic access airlocks instead of security access airlocks

Description
The airlocks on the right of warden's office that lead into the brig section of security are generic access airlocks. These should be security only access airlocks.
Screenshots

apparently this is a feature, #15587

Ah my bad, I didn't see it was added in that feature. It does feel a little weird having that hallway open to the entire crew however. No other stations have security designed like this, and it is a massive headache to security when playing on this map.

This issue is resolved.


Embedding a note within a daily note ignores the notes_subdir, places the note in current directory instead

🐛 Describe the bug
When I create a link from within a daily note inside of a "daily-notes" folder the new note gets added to the current folder ("daily-notes") and not the notes_subdir.
I have configured the plugin exactly as in the readme. Interestingly, the notes folder gets created if it isn't present, but the new note is still placed in the current folder.
Versions
NVIM v0.9.5
Build type: Release
LuaJIT 2.1.1703358377
[obsidian.nvim (v3.2.0)] Commit SHA: 0a6739d2229c8eb30396db550f3818e092088c27
[plenary.nvim] Commit SHA: 4f71c0c4a196ceb656c824a70792f3df3ce6bb6d
[nvim-cmp] Commit SHA: 04e0ca376d6abdbfc8b52180f8ea236cbfddf782
ripgrep 14.1.0

I have a similar problem. the command ObsidianNew creates a note in whatever cwd I'm in.

This looks like a duplicate of https://github.com/epwalsh/obsidian.nvim/issues/395. To get the desired behavior just set completion.new_notes_location = "notes_subdir" in your obsidian.nvim config.
This has come up a few times recently so I think I'll change the default in the README to that.

FYI I moved that option to the top-level in f753d0d. This should make it more obvious.

@epwalsh the thing is I have set that option.
To make my problem clearer when I'm in my daily folder within the vault if I create a link to a file that doesn't exist yet and trigger completion it is still created in the current folder, even with the new_notes_location set to notes_subdir.
The notes_subdir is located one level above the daily dir where .obsidian is.

Sure, @epwalsh, here it is:
return {
	"epwalsh/obsidian.nvim",
	version = "*",
	lazy = true,
	ft = "markdown",
	dependencies = {
		"nvim-lua/plenary.nvim",
	},
	init = function()
		require("which-key").register({ o = { name = "+obsidian" } }, { prefix = "<leader>", mode = { "n", "v" } })
	end,
	keys = {
		{ "<leader>on", "<cmd>ObsidianNew<CR>", desc = "New Note" },
		{ "<leader>od", "<cmd>ObsidianToday<CR>", desc = "Todays Note" },
	},
	opts = {
		dir = "~/Documents/obsidian",
		notes_subdir = "notes",
		completetion = {
			nvim_cpm = true,
			min_chars = 2,
			new_notes_location = "notes_subdir",
			preferred_link_style = "markdown",
		},
		mappings = {
			-- Open obsidian link.
			["fo"] = {
				action = function()
					return require("obsidian").util.gf_passthrough()
				end,
				opts = { noremap = false, expr = true, buffer = true },
			},
			-- Toggle check-boxes.
			["<leader>ch"] = {
				action = function()
					return require("obsidian").util.toggle_checkbox()
				end,
				opts = { buffer = true },
			},
		},
		prepend_note_id = true,
		-- Optional, customize how names/IDs for new notes are created.
		note_id_func = function(title)
			-- Create note IDs in a Zettelkasten format with a timestamp and a suffix.
			-- In this case a note with the title 'My new note' will be given an ID that looks
			-- like '1657296016-my-new-note', and therefore the file name '1657296016-my-new-note.md'
			local suffix = ""
			if title ~= nil then
				-- If title is given, transform it into valid file name.
				suffix = title:gsub(" ", "-"):gsub("[^A-Za-z0-9-]", ""):lower()
			else
				-- If title is nil, just add 4 random uppercase letters to the suffix.
				for _ = 1, 4 do
					suffix = suffix .. string.char(math.random(65, 90))
				end
			end
			-- return tostring(os.date("%Y%m%d%H%M%S")) .. "-" .. suffix
			return tostring(os.time()) .. "-" .. suffix
		end,
		log_level = vim.log.levels.INFO,
		daily_notes = {
			folder = "/daily",
			-- Optional, if you want to change the date format for the ID of daily notes.
			date_format = "%Y-%m-%d",
			-- Optional, if you want to change the date format of the default alias of daily notes.
			alias_format = "%B %-d, %Y",
			-- Optional, if you want to automatically insert a template from your template directory like 'daily.md'
			-- template = "templates/daily.md",
		},
		templates = {
			subdir = "templates",
			date_format = "%Y-%m-%d",
			time_format = "%H:%M",
			-- A map for custom variables, the key should be the variable and the value a function
			substitutions = {},
		},

		-- Optional, by default when you use `:ObsidianFollowLink` on a link to an external
		-- URL it will be ignored but you can customize this behavior here.
		follow_url_func = function(url)
			-- Open the URL in the default web browser.
			vim.fn.jobstart({ "open", url }) -- Mac OS
			-- vim.fn.jobstart({"xdg-open", url})  -- linux
		end,

		-- Optional, set to true if you use the Obsidian Advanced URI plugin.
		-- https://github.com/Vinzent03/obsidian-advanced-uri
		use_advanced_uri = false,

		picker = {
			-- Set your preferred picker. Can be one of 'telescope.nvim', 'fzf-lua', or 'mini.pick'.
			name = "telescope.nvim",
			-- Optional, configure key mappings for the picker. These are the defaults.
			-- Not all pickers support all mappings.
			mappings = {
				-- Create a new note from your query.
				new = "<C-x>",
				-- Insert a link to the selected note.
				insert_link = "<C-l>",
			},
		},
		-- Specify how to handle attachments.
		attachments = {
			img_folder = "assets/imgs",
		},
	},
}

Huh, still not able to repro. Can you make sure you're on the latest commit and send me the output of :ObsidianDebug?

@epwalsh and here is a gif of the workflow, just to make sure we're on the same page:

Thanks @dim-anis. Uhhh I don't see how this could happen and I've set up a vault in the same way, but it works as expected for me 🤔
If you get a chance please try to debug on your own. The relevant function is Client:new_note(). You can call this programmatically like this:
:lua require("obsidian").get_client():new_note("new note", "123-new-note")

Maybe just throw some print statements into Client:new_note() to see what it resolves the path to.

Sure, I'll take a look at it when I have time.
Thanks for this great plugin, @epwalsh!

@epwalsh I did some debugging and found out that it was due to the type in the config, specifically this part:
		completetion = {
			nvim_cpm = true,
			min_chars = 2,
			new_notes_location = "notes_subdir",
			preferred_link_style = "markdown",
		},

So "completetion" instead of completion. new_notes_location was using the default current_dir value as a result.
Oh, well.


[NDTensors] Implement block sparse QR decomposition

Also use combiners to render ITensors down to rank-2 tensors, which greatly simplifies the QR code at the ITensors level.

Need to test locally with Julia 1.6

Codecov Report

Merging #1009 (78c556f) into main (6f6831e) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1009      +/-   ##
==========================================
+ Coverage   65.02%   65.05%   +0.02%
==========================================
  Files          97       97
  Lines       10170    10181      +11
==========================================
+ Hits         6613     6623      +10
- Misses       3557     3558       +1

Impacted Files
Coverage Δ

src/tensor_operations/matrix_decomposition.jl
93.53% <100.00%> (+0.28%)
:arrow_up:

src/indexset.jl
82.63% <0.00%> (-0.60%)
:arrow_down:

src/index.jl
75.15% <0.00%> (+0.63%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Hi Jan, sorry for being so slow about reviewing this. It (finally) looks like we will be using https://github.com/JanReimers/ITensorMPOCompression.jl so it would be good to get this and #1012 merged. I'll leave a few comments but over all I think they both look good, thanks for the contributions!

Looks like some nice code, especially the lower-level function (the one that works for matrix-like block sparse tensors).
Is the part about adding and removing trivial indices to handle the case where the tensor doesn't have row or column indices? Or what is that part for? Thanks.

Yeah, it is based on a bit of a hack I had added to svd to handle the case where the row or column indices are empty, I found that really useful for implementing certain algorithms generically (such as code where an MPS may be missing link indices). It could of course be implemented more efficiently but personally I think it is ok to leave in since the functionality is reasonable and we could always optimize it later.

Thanks for explaining - sounds reasonable to leave it in.


combine_dtls_fragments helper

~# REQUIRES #47 TO LAND FIRST #~
This PR builds upon #47. The aim is to provide a correct helper that combines DTLS fragments into a full message. The test case illustrates how the parser + combine helper are expected to be used together.

Will rebase when/if #47 lands.

I believe this is ready to merge now.

I believe this is ready to merge now.

Hi,
After review I need more input on the choices you made (see above comments). The most important questions are on the Ordered type, others can be solved separately.
Thanks!


PVP particles appear(?) when a player connects/disconnects

Despite having pvp_playerparticle_enable 0, when a player/bot connects/disconnects the particle is shown. But doing the commonly used demo command to fix visual bugs (record fix; stop) removes them except for one at 0 0 0. I am thinking they may be ghost particles? I am on the develop branch.

I want to at least look into this when i have time, dont close it


NULL deref crash in sctp_timer_start when processing fuzzed ASCONF

Hit this issue when fuzzing ASCONF chunks. A PCAP is attached.
The crash occurs when sctp_timer_start() is called here: https://github.com/sctplab/usrsctp/blob/ac364403c13c5cefe2bc5e87b8bf730ba7d491a0/usrsctplib/netinet/sctp_asconf.c#L268
repro.pcap.zip
usrsctp: sctp_process_control: processing a chunk type=193, len=40
usrsctp: SCTP_ASCONF
usrsctp: handle_asconf: serial number = b5ed35a2h (expected next = b5ed35a2h)
usrsctp: handle_asconf: asconf_limit=80, sequence=b5ed35a2h
usrsctp: handle_asconf: Now processing first ASCONF. Try to delete old cache
usrsctp: process_asconf_add_ip: adding
usrsctp: IPv4 address: <IP_ADDRESS>:5000
usrsctp: process_asconf_add_ip: using source addr
usrsctp: AF_CONN address: 0x60c000000280
usrsctp: Adding an address (from:6) to the peer:
usrsctp: AF_CONN address: 0x60c000000280

AddressSanitizer:DEADLYSIGNAL
=================================================================
==224135==ERROR: AddressSanitizer: SEGV on unknown address 0x00000000025d (pc 0x55e65eaec0f8 bp 0x7ffd0dda18d0 sp 0x7ffd0dda1880 T0)
==224135==The signal is caused by a READ memory access.
==224135==Hint: address points to the zero page.
usrsctp: Timer type 17 goes off

usrsctp: Timer now complete (type = 17)

    #0 0x55e65eaec0f7 in sctp_timer_start third_party/usrsctp/usrsctplib/netinet/sctputil.c:2257:12
    #1 0x55e65eb13566 in sctp_process_asconf_add_ip third_party/usrsctp/usrsctplib/netinet/sctp_asconf.c:268:3
    #2 0x55e65eb11502 in sctp_handle_asconf third_party/usrsctp/usrsctplib/netinet/sctp_asconf.c:742:15
    #3 0x55e65eab9e09 in sctp_process_control third_party/usrsctp/usrsctplib/netinet/sctp_input.c:5492:5
    #4 0x55e65eae1865 in sctp_common_input_processing third_party/usrsctp/usrsctplib/netinet/sctp_input.c:5899:10
    #5 0x55e65ea717e7 in usrsctp_conninput third_party/usrsctp/usrsctplib/user_socket.c:3518:2
    <snip>

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV third_party/usrsctp/usrsctplib/netinet/sctputil.c:2257:12 in sctp_timer_start
==224135==ABORTING

Is it possible to share with @weinrank some information such that @weinrank can run a reproducer locally in our lab?

Sure, I will send the info tomorrow.

I added a stand-alone repro program in this branch:
https://github.com/markwo/usrsctp/tree/fuzzer_repro
Since the issue is due to an uninitialized pointer on the stack,  the crash doesn't always trigger (sctp_timer_start checks for NULL inputs). The best way to catch this is with -fsanitize=memory. I was building the project as follows:
cmake -DCMAKE_C_COMPILER=/usr/bin/clang-8 -Dsctp_build_repro=1 -Dsctp_sanitizer_memory=1 -Dsctp_sanitizer_address=0 -Dsctp_debug=1 .
Then run repro/repro_380.  Try to delete old cache
[S][0.005] process_asconf_add_ip: adding [S][0.005] IPv4 address: <IP_ADDRESS>:5001
[S][0.005] process_asconf_add_ip: using source addr [S][0.005] AF_CONN address: 0x1
[S][0.005] Adding an address (from:6) to the peer: [S][0.005] AF_CONN address: 0x1
==212212==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0xa194d3 in sctp_timer_start (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0xa194d3)
    #1 0xb23167 in sctp_process_asconf_add_ip (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0xb23167)
    #2 0xb1c227 in sctp_handle_asconf (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0xb1c227)
    #3 0x554a23 in sctp_process_control (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0x554a23)
    #4 0x53191a in sctp_common_input_processing (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0x53191a)
    #5 0x5014e0 in usrsctp_conninput (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0x5014e0)
    #6 0x4aca05 in main /usr/local/google/home/markwo/repos/usrsctp/repro/repro_380.c:184:3
    #7 0x7f145e53e52a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2352a)
    #8 0x42a439 in _start (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0x42a439)

  Uninitialized value was stored to memory at
    #0 0xa15115 in sctp_timer_start (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0xa15115)
    #1 0xb23167 in sctp_process_asconf_add_ip (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0xb23167)
    #2 0xb1c227 in sctp_handle_asconf (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0xb1c227)
    #3 0x554a23 in sctp_process_control (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0x554a23)
    #4 0x53191a in sctp_common_input_processing (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0x53191a)
    #5 0x5014e0 in usrsctp_conninput (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0x5014e0)
    #6 0x4aca05 in main /usr/local/google/home/markwo/repos/usrsctp/repro/repro_380.c:184:3
    #7 0x7f145e53e52a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2352a)

  Uninitialized value was created by an allocation of 'net' in the stack frame of function 'sctp_process_asconf_add_ip'
    #0 0xb1f080 in sctp_process_asconf_add_ip (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0xb1f080)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/usr/local/google/home/markwo/repos/usrsctp/repro/repro_380+0xa194d3) in sctp_timer_start

@markwo Please retest and report.

Tested the fix - looks good.


Cannot replay transactions with v=0 using py-evm

py-evm Version: Using web3["tester"]@latest bundle with 5.24.0
OS: osx
Python Version (python --version): 3.8

What is wrong?
I believe there is a validation check in current main that is incompatible with London transactions (https://github.com/ethereum/py-evm/blob/master/eth/vm/forks/frontier/transactions.py#L92). This validation check currently checks if v >= 27 but, for example, Hardhat transactions now all have v=0 in London. Specifically, these are type 1/2 transactions in EIP-1559.
How can it be fixed
It may already be fixed in the upcoming py-evm version that will ship with london support but wanted to flag in case it isn't :).

@Pet3ris Welcome and thanks for the inquiry! Yeah, the signing also changes for typed transactions and this was implemented differently from the legacy transactions in the Berlin changes for py-evm and the London changes picked up from there.
That validation check you linked to is for a Frontier fork transaction. The "legacy" transactions for each fork inherit from each other all the way up to the most recent LondonLegacyTransaction which inherits from BerlinLegacyTransaction... which inherits from MuirGlacierTransaction and so on... down to the FrontierTransaction that you linked above.
However, when we get to Berlin and London (the introduction of typed transactions), there is also AccessListTransaction and DynamicFeeTransaction and unsigned versions of those as well. The signing field is also no longer referred to as v but y_parity.
The validate method that you linked to is a nice-to-have method for validating the FrontierTransaction class only and thus would not apply to the typed transactions mentioned above. The y_parity field can also be 1 thought, it is not always 0.
I'm going to close this based on the above but feel free to re-open if I missed something.

Thank you for the detailed explanation. I'm not fully aware of what the implications are for how to implement type 2 transactions in web3.py so perhaps it's simpler if I offer more context of what I was trying to do:
Effectively I was:

downloading a recent transaction from mainnet by using eth.getTransaction
turning it into a raw transaction with the following code:

v = tx["v"]
r = int.from_bytes(tx["r"], "big")
s = int.from_bytes(tx["s"], "big")
unsigned_transaction = serializable_unsigned_transaction_from_dict(transaction)
raw_tx = encode_transaction(unsigned_transaction, vrs=(v, r, s))

then using eth_sendRawTransaction to send this raw_tx to a local node to replay it

Does w3.eth.sendRawTransaction(raw_tx) need some adjustment to interpret the transaction as a type2 perhaps or would the transaction itself need some changes than what is returned from eth.getTransaction?

You would have to clean up the transaction in this case to only have the necessary fields for the transaction type you want (so getting rid of blockHash, blockNumber, gasPrice, hash, input, etc... from the eth_getTransaction call) so that it has only the valid params of an unsigned transaction. Then it can be passed into serializable_unsigned_transaction_from_dict.
You can also re-create the transaction and use w3.eth.account.sign_transaction() as outlined here.
In the latter case you can strip all the parameters that don't exist in an unsigned transaction and run it similarly to the example above without having to re-write everything.
Are you attempting to use eth-tester with py-evm? Because if not, none of this actually touches py-evm. So I'm curious about what the py-evm inquiry is here.

I see, really appreciate the clarification. I definitely still had gas, gasPrice and other arguments in there and it worked well previously but stopped working with type 2 transactions. I'll try stripping out those arguments to see if I can make py-evm route it to the type 2 validator and not check the v argument and hopefully it will work.
This is what I was doing to make it work pre-london:
  transaction = {
      "to": tx["to"],
      "gas": tx["gas"],
      "gasPrice": tx["gasPrice"],
      "value": tx["value"],
      "nonce": tx["nonce"],
  }
  if "data" in tx:
      transaction["data"] = tx["data"]
  if "input" in tx:
      transaction["data"] = tx["input"]

Because these are mainnet transactions, unfortunately I don't have access to the keys so not able to sign them directly.
Yeah I'm sending the transaction to py-evm under the hood.


[ZSeries] Include Error Codes and PAUSE state into Graphs

Tonight I had an error code "sensor overheat error: 4" for the first time on one of my Z units during a brew day (first time ever). It would be helpful to either highlight the graph with red (like the UI in BrewHouse) or to add an error flag above the graph showing the "current data" both temps, time remaining, any error code/state code, and including location.
Reason for error code was that the brew room for whatever reason was at an all time high for brewing (though it was cooler outside). The below graph obviously shows there was an error when HEX, wort and drain started dropping so sharply during the boil phase.

Sounds like you need A/C 🤣

Haha or if I knock down a wall I bet the air will circulate a bit more... Too many appliances all in a closed space. Washer and dryer near by were running too 🤪

Personally like the winter brewing months more as it gives a constant 60-70 in the basement so can cold crash 8 kegs in the storm door without taking up kegerator or keezer space and can fill the entire basement with fermenters (if only I had more). 🤣🍻

Plotbands is the name of the high charts feature that would be used here.
We would also add in the error code into the hover plot description.
Example of bands on Y axis: https://www.highcharts.com/demo/spline-plot-bands
Examples of bands on X axis (what we would need) https://www.highcharts.com/demo/combo-timeline

Plotbands is the name of the high charts feature that would be used here.
We would also add in the error code into the hover plot description.
Example of bands on Y axis: https://www.highcharts.com/demo/spline-plot-bands
Examples of bands on X axis (what we would need) https://www.highcharts.com/demo/combo-timeline


Tooltip misplaced when closest positioned ancestor is body element

I'm using the tooltip component in an AngularJS application. The initial configuration was the default (tooltip-append-to-body=false).
Then, I noticed that my tooltip is misplaced when I scrolled down the window in Chrome. After some debugging, I realized that the closest ancestor which has a explicit position (offsetParent) was the "body" element. This element seems to be a bit special, because (at least in Chrome), is the only one for which both "scrollTop" and $window.pageYOffset properties get updated. For any other element, we will have pageYOffset updated when we scroll the window, and scrollTop, when we scroll the element (if made scrollable with overflow: scroll).
As a result, the geometry arithmetic in position() function will not work with the "body" element:
https://github.com/angular-ui/bootstrap/blob/master/src/position/position.js#L47
Should I set "tooltip-append-to-body" directive to true when the body is the offsetParent? If so, I had another issue: If the body is positioned "relative" with top/left different to 0, the tooltip will be misplaced by that same amount. The reason lies here:
var positionTooltip = function (){
// Get the position of the directive element.
position = appendToBody ? $position.offset( element ) : $position.position( element );
}
The offset of the element is calculated from the top-left coordinates of the document, not necessarily the body (which can be offset relatively, as explained above).
If this is the intended way the code should work, I think the documentation (http://angular-ui.github.io/bootstrap/) should be updated to make the API a bit clearer, because the only way for me to understand this was debugging the code.
Thanks,
Aritz

Finding this post helped me out.  I would have spent hours figuring out why my tooltips weren't lining up right in Chrome.  Thanks Aritz!

plunkr please

@Foxandxss what do you think? Is this worth updating the docs for?

Well, to be honest, I have no idea about positioning, not my topic. Now, if the current implementation is OK but there is a behavior that is not documented, absolutely yes.

Closing in favor of #4762.


Non breaking words

Hello,
When I run:
let text = "textwrap: a small library for wrapping text.";

println!(
    "{}",
    textwrap::Wrapper::new(1).break_words(false).fill(text)
);

I get the non-expected results:
textwrap:

a
small
 librar
y
 fo
r
 wrappin
g
 text
.

Yet I expect:
textwrap:
a
small
library
for
wrapping
text.

Same with 2, 3, 4 ...
Thank you!

Hi @truchi! Oh wow, that looks odd... I'll take a look and try to fix it as soon as possible. Thanks for the report.

You are welcome, sir. Thank you for the fix!


What are the "transitions"

What are the "transitions." It seems like they have something to do with the dynamic programming, but I can't quite make sense of them?
https://github.com/dirko/hacrf_sparse_cpp/blob/master/crf_ed.cpp#L334-335

Hi! Hope you are doing well.
A transition is a pair (see https://github.com/dirko/hacrf_sparse_cpp/blob/master/crf_ed.cpp#L267) that represents either a 'match/substitution', 'insertion', or 'deletion'. For example the transition (-1, -1) represents a match or substitution. When the delta (-1, -1) is added to a lattice coordinate, we move one up and one left in the lattice.
So you're right about the dynamic programming - we fill the two-dimensional tables (which represents a lattice) by visiting each cell in the table and filling it with values obtained by using values in neighbouring cells. We decide which neighbouring cells to visit by enumerating all the 'transitions' and adding these coordinate pairs to the current cell's coordinates.


Substring match statement behaves unexpectedy

Prerequisites

[x] Put an X between the brackets on this line if you have done all of the following:

Check that your issue is not already filed.
Reduce the issue to a minimal, self-contained, reproducible test case. Avoid dependencies to mathlib4 or std4.

Description
Most users probably have the expectation that a Substring behaves like a String in most circumstances. The behavior of match expressions on Substrings is therefore quite unintuitive.
Context
Discussion on Zulip
Steps to Reproduce
import Std.Data.String.Basic

def isRed (s : Substring) : Bool := match s with
  | "red" => true
  | _ => false

def redSubstring : Substring := Substring.mk ".red." ⟨1⟩ ⟨4⟩

#eval isRed "red" -- true
#eval redSubstring -- "red".toSubstring

Note that #eval isRed (redSubstring.toString) returns true.
Expected behavior:
#eval isRed redSubstring -- true

Actual behavior:
#eval isRed redSubstring -- false

Versions
4.5.0-rc1 on Ubuntu 20.04.6 LTS
Additional Information

Impact
Add :+1: to issues you consider important. If others are impacted by this issue, please ask them to add :+1: to it.

I feel the result is right. The Substring is not String and will keep the information of the origin String.
You can show this via.
example  : isRed redSubstring = false := by
  simp only [isRed, redSubstring]

then the goal will be.
(match { str := ".red.", startPos := { byteIdx := 1 }, stopPos := { byteIdx := 4 } } with
  | { str := "red", startPos := { byteIdx := 0 }, stopPos := { byteIdx := 3 } } => true
  | x => false)

You can fix the problem by:
def isRed (s : Substring) : Bool := match s.toString with
  | "red" => true
  | _ => false

I feel the Repr of the Substring is not very clear and leads your misunderstanding.

Closing this issue since this is not a bug.


Allow to inject data from externally crafted secrets

Let's say a broker is installed inside the same namespace as my integration. The broker is secured and provides a secret with authentication data: username and password.
Can I inject those two fields in my integration?
Afaik, currently we can inject only secrets containing a "xxx.properties" file.
Is this the case?
@jamesnetherton , @lburgazzoli , @astefanutti

We now support only properties but it won't be to complex to load any kind of property however some additional properties may still be required, like to set those properties to a component:
camel.component.jms.connection-factory.host = {{my-other-property}}

Mind opening an issue on camel-k-runtime ?

Yeah, we can still load automatically .properties files in the secrets, plus make the content of each entry of a secret available in a secrets namespace, so users can reference them by name, e.g.
camel.component.xxx.password = {{secrets.broker.authToken}}

Thinking big (for the far future), we can also provide an addon module where we add auto-configuration for connecting to known systems.. this require some effort to be maintained, but a user would be able to do:
kamel run integration.groovy --connect activemq:mybroker

... and have all camel.component.activemq.xxx properties set.

Ref: https://github.com/apache/camel-k-runtime/issues/239

Yeah, we can still load automatically .properties files in the secrets, plus make the content of each entry of a secret available in a secrets namespace, so users can reference them by name, e.g.
camel.component.xxx.password = {{secrets.broker.authToken}}

We can use properties functions like
{{secret:$secret-name:property-name}}

so your example will become:
camel.component.xxx.password = {{secret:broker:authToken}}

I did it in the past for Karaf and blueprints, need to resurrect that code :)


[cli/runAndroid] removing xterm from linux spawn

Simply removing the xterm spawn from android packager, since you can archive exactly the same results using the standard shell :smile:

I believe @nmccready had a reason to use xterm here?

Since the shell is configurable using sh by default sounds reasonable to me:
https://github.com/facebook/react-native/commit/970782d1473b3e2e962aa26adccc1d24bc01c505

The reason for it was to choose what terminal you want to run it with. I
use iterm. The other benefit of this is because the normal osx terminal is
intrusive. Where iterm just spawn a tab in the background for the
background processes for react-native.
Sent from my mobile device
VR,
Nicholas McCready
twitter:nmccready
<EMAIL_ADDRESS>github:nmccready
On Apr 20, 2016 09:21, "Martin Konicek"<EMAIL_ADDRESS>wrote:
I believe @nmccready https://github.com/nmccready had a reason to use
xterm here?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/facebook/react-native/pull/7077#issuecomment-212420901

@nmccready This PR is for Linux only, to use sh rather than xterm on Linux.

Oh sorry I was just replying to the motivation to the changes. As far as xterm I was told to use this in the PR for the linux code block.

OK, thanks for the quick reply @nmccready. Let's wait for @rscnt to add a test plan.

@mkonicek sorry for the late reply, here is a video showing the result of running react-native run-android, without having installed xterm, first not using the small change committed on this PR and after with the changes applied.
https://asciinema.org/a/43097

This is an awesome test plan @rscnt! https://asciinema.org/a/43097
I see "Starting JS server". Does a new window with the packager running open?

But yeah let's merge this because using xterm simply throws an error for you.
@facebook-github-bot shipit

I'm on fedora, and xterm was causing me all sorts of issues.  I also needed to install xorg-x11-fonts-misc as xterm was complaining about fonts being missing.

Interestingly enough, with my fix in https://github.com/facebook/react-native/pull/7316, xterm did indeed open a new window!


Base image uses Debian Buster, but Mono release for Stretch is installed

This line: https://github.com/jonemo/pythonnet-docker/blob/c3dd8429cd0d86d0cef0f1d0d7bf90562473931a/_dockerfiles/python3.7.7-mono5.20-pythonnet2.5.0#L7
I just noticed while investigating a warning during installation of dotnet-sdk-3.1.

I was wondering if I could get away with this. I didn't see any issues during my cursory tests, but it looks like you found the problem...
It looks like there are no Mono 5.x packages available for Buster. Therefore, I should probably downgrade the Mono 5.20 images to Stretch and then create a separate set of images with Buster an 6.x. I have not yet tested PythonNET with Mono 6.x, so that might be another stumbling block.
@michaelosthege Does that plan sound right to you?

Sounds right.
As I'm using .NET Core within the image, I'm not sure if I would even encounter potential problems with Mono, but I'll let you know if I do.


⚠ Bofrost.es has degraded performance

In 38e0a8c, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1840 ms

Resolved: Bofrost.es performance has improved in 5182e37 after 6 minutes.


Defect:  'Tutorial and More' section has incorrect text size, no background image

'Tutorial and More' section has incorrect text size, no background image
Steps:

Navigate to https://main--vg-partsasist--hlxsites.hlx.page/ from index.docx by clicking preview
Validate the 'Tutorial and More' section

Expected:

Actual:
'Tutorial and More' section is not as expected as Text size is incorrect and no background image is set

Hi @andreibalanescu : The font size is incorrect.

Hi @girishdigrajkar
It should be fixed. Could you retest it?

Background image issue, font  is fixed however, text case is not as expected ( All letters should be in capital case and 'MORE' should be on next line). Except for this minor authoring change defect could be closed.

@andreibalanescu : Could you please check if the changes are published and confirm ... thanks

I improved the style to match the legacy layout. Please check https://github.com/hlxsites/vg-partsasist/pull/87

Fixed now


Add support for iOS web browsers

This is not working on ios browsers, I guess because hls are natively supported on ios browsers and even hls.js (https://github.com/video-dev/hls.js) has to be handled separately for ios browers
As I am using react-hls, do you have suggestion on what to include in codes to make it work on ios browsers?
                            <ReactHlsPlayer
                                url={this.state.video.hls_url}
                                autoplay={false}
                                controls={true}
                                width="100%"
                                height="auto"
                            />

It looks like hls.js doesn't support iOS Safari, because iOS doesn't support media source extensions yet. Currently we are using hls.js as the only way of playing the video, so iOS isn't supported by this package.

Also doesn't worked in Chrome on IOS, so not a Safari issue.

Also doesn't worked in Chrome on IOS, so not a Safari issue. I guess what is really needed on IOS is a fallback to native player.

As far as I'm aware, all web browsers on iOS use the same engine that Safari uses. Apple doesn't allow custom web browsers. So even Google Chrome uses Safari's engine under the hood.
Unless something has changed recently

I moved to react-player and now hls videos are playing on ios browsers
https://www.npmjs.com/package/react-player

Thanks for sharing that @durgendra

I was able achieve this by using the .isSupported() method from hls.js.
Example React component as follows:
// Other imports
import ReactHlsPlayer from 'react-hls-player';
import HlsJs from 'hls.js';

const Video = () => {
    let videoSource = VIDEO_SOURCE_MP4;

    // Safari browsers (iOS, ipadOS and macOS) have built in HLS support
    if (document.createElement('video').canPlayType('application/vnd.apple.mpegurl')) {
        videoSource = VIDEO_SOURCE_HLS;
    }

    return (
        <div className="video">
            {HlsJs.isSupported() ? (
                <ReactHlsPlayer
                    poster={VIDEO_POSTER}
                    url={VIDEO_SOURCE_HLS}
                    autoplay
                    muted
                    controls
                    loop
                    width="100%"
                />
            ) : (
                <video
                    poster={VIDEO_POSTER}
                    src={videoSource}
                    autoPlay
                    muted
                    controls
                    loop
                    width="100%"
                />
            )}
        </div>
    );
});

Minor gripe that the casing of autoPlay between the video element and react-hls-player library and incompatible so you can't simply spread the common props.

@toughdeveloper I've fixed the props issue in the latest version. The root level of the video player now supports all regular video element props with the proper casing. Another big change is the move from using url to now using src. This should make it easier to use all the common props that you'd expect from a video player. I haven't added the fallback to a regular video component yet when HLS is supported, but that's definitely on the agenda for something I'd like to do.

Latest version works with Safari, using @toughdeveloper 's recommendation above. Closing this issue. Thank you for providing the solution to this.


Have to rebuild node-sass each time I restart server

Hi every time I stop my local server I have to rebuild node-sass npm rebuild node-sass
version<EMAIL_ADDRESS>Due to this error - extremely frustrating
Module build failed: Error: ENOENT: no such file or directory, scandir '/Users/rgb/code/afsk/node_modules/node-sass/vendor'

Any advice? I have used node-sass for a long time across many projects and frameworks never had this issue until the last two weeks.

Something is deleting node_modules/node-sass/vendor folder. You'll need to track down what is doing that and stop it. There isn't anything we can do.


Install shared farmework compatible with particular version of CLI

@dotnet-bot test Windows_NT / Debug Build please


Invalid SITE_URL value:  InvenTree server cannot start.

Deployment Method

[ ] Installer
[ ] Docker Development
[x] Docker Production
[ ] Bare metal Development
[ ] Bare metal Production
[ ] Digital Ocean image
[ ] Other (please provide a link Steps to Reproduce

Describe the problem*
i have my own docker <-- Proxy  (Traefik) disabling it is not an option,  and i don't really think its the cause,  but as such,  i h ave disabled the use of caddy for inventree because my traefik is configured to auto deploy  containers to a DNS wildcard  containername.docker.example.com    this setup should work just as Caddy does as it just proxys the dns to the container on its service port.  so i don't think its a problem..
But i can't get the inventree-server to stop flapping (restarting)   it just contantly loops with this error
Invalid SITE_URL value: '"http://inventree.docker.example.com"'. InvenTree server cannot start.
Yet, the variable SITE_URL  is configured properly and works.   the server just never starts.
Steps to Reproduce
Described above.
Relevant log output
No response

please provide full log output for the server. And I assume the url you have provided is not your actual URL?

Yeah clearly,  i just don't want my url posted publicly for obvious reasons.
There isn't really any log  it just repeats what i said over and over with standard info lines above and below
Python version 3.11.9 - /usr/local/bin/python
Invalid SITE_URL value: '"http://inventree.docker.*******.com"'. InvenTree server cannot start.
Loading config file : /home/inventree/data/config.yaml

@decryptedchaos I tried to reproduce this fault, by using the literal URL http://inventree.docker.example.com - the server did not fail in this case. Is there perhaps something particular about the specific URL you are using?
Here is the validation being performed (in https://github.com/inventree/InvenTree/blob/cdac7465b22e0eaaf1f2d2a469f2ef32502d03db/src/backend/InvenTree/InvenTree/settings.py#L987)

No nothing special,  the only thing that's even remotely different on my setup is that i don't use your supplied Caddy proxy like i say in my OP, because i already have a configured traefik proxy serving multiple containers that i can't simply change for one image.
Aside from that the only other thing is, i use DNS wildcards   so at my DNS server *.docker.mydomain.com returns the docker server ip this shouldn't matter,   lastly,  the docker host is behind a NAT,  but the proper ports for the proxy  (80,443) are forwarded to the docker host and traefik answers and function 100% fine for other containers

I've tried deleting everything and starting from scratch twice and its the same result every time, is there something else i need to do to purge any dangling remnants before trying to start over?

No, I do not think that there's any "remnants" here causing this problem. There is some reason that the URL validator is failing on your particular URL.
We just use the django URLValidator class here:
https://github.com/django/django/blob/adae619426b6f50046b3daaa744db52989c9d6db/django/core/validators.py#L127
You could run a simple test on your end by passing your specific URL through the validator class and see what particular error it is throwing?

Interestingly it throws no error assuming the variable is being read between the " " from your variable and not including the "   when i pass my URL exactly as typed between the quotes it returns valid/correct.  obviously if i don't include http:// or if i include the quotes  it will fail.  but i can't imagine why it would be passing the deliminator's or excluding parts of the string,  and while it seems unlikely that's happening,  i'm at a loss because the URLvalidator function has no problem with my URL when running from a simple python try exception script i threw together, assuming it's being passed correctly from the .env file to inventree when, it should work  there must be a bug here, that isn't obvious.  because either its parsing the .env wrong or its striping out or including something in the eval  that it shouldn't  seems to be the only possible causes if the raw fucntion works for my URL

Ahh. ok so the extra quotes killed it, that is interesting. Might be worth automatically stripping quotes from the provided data

i don't use your supplied Caddy proxy like i say in my OP, because i already have a configured traefik proxy serving multiple containers

How do you serve the static files then?

How do you serve the static files then?

Haven't really gotten that far yet,  currently gnuicorn is serving them directly  to my proxy,  probably not efficient, but i haven't found anything that doesn't work yet.  and i don't know if there is a way to get traefik to access file directly.  so that could be a flaw in the setup.
But i still think its the least horrible,  as compared to having  Client --> Traefik --> Caddy --> gnuicorn   Which i think would have been far more troublesome.
Since the server is  Single IP/NAT  it can only have one listener for HTTP/HTTPS that has been traefik  because its ability to integrate and automatically deploy for all containers easily.
That's a pretty common way to use docker too,  so i always kinda question it when projects want to use its own proxy server,  because the likely only way that would work right is if you could assign a dedicated IP to that set of containers.


Mirror for Sonatype OSS Index Analyzer?

Since there is a mirror of  NVD (CPE/CVE JSON) data from NIST, see [https://github.com/stevespringett/nist-data-mirror]. Is there a mirror for  Sonatype OSS Index Analyzer?
In my company, dependency-check is used in a offline server which cannot connect to https://ossindex.sonatype.org/

Unlikely; it is a violation of the TOS
You could contact Sonatype for an on-premise solution

You can disable the OSS Index Analyzer - small increase in false negatives and false positives.

Since there is a mirror of NVD (CPE/CVE JSON) data from NIST, see [https://github.com/stevespringett/nist-data-mirror]. Is there a mirror for Sonatype OSS Index Analyzer?
In my company, dependency-check is used in a offline server which cannot connect to https://ossindex.sonatype.org/

Can you help me use it in a offline server please


Custom Drag and drop

Hi,
This lib is awsome and thanks for solving my most of problems .
It would me great if you provide document for custom drag and drop.
if anyone have any idea about this please share. i tried to create but unble to achive.
thanks

I'd like to second this.
First of all thanks for this awesome library - it almost packs everything I was looking for out of the box.
My scenario is the following:
We're currently using react-dnd application-wide for typical drag and drop scenarios and react-beautiful-dnd for some specific list-based scenarios.
What I really like when using react-dnd is how easy the hooks make it to just spread the appropriate attributes onto a component to give it the role of e.g. a drop handler for a certain type.
The components I'd like to reuse inside react-flow also have rather feature-rich click handler implementations and contain popover components that provide an overlay with additional functionality (e.g. context menu).
I've only done a straight forward PoC based on the react-flow examples thus far and I noticed that everything contained in the component tree of a node is treated as a drag handle for the node.
It would be really great if there was a way to have control over which components inside the node should be treated as the drag surface more granually (ideally something like the aforemenitoned react-dnd).
Is there already a built-in way to achieve this or can I help defining the API surface for this?

hey @rajeshwari-differenz123
what do you want to customize? We have a simple drag and drop example on the website https://reactflow.dev/examples/drag-and-drop/
@5ebastianMeier I think this is another question.. however you can use the nodrag class for elements inside your node. It's documented here https://reactflow.dev/docs/api/node-types/ under "Prevent dragging & selecting
".

@moklick thanks a bunch! The use case I had for my PoC worked like a charm with it. I only had to add a cursor: default and add the class nodrag like you mentioned.
That enabled me to have a "menu border" around my reused component (which resides in a container with the added classes). Now the menu border can host command buttons and the area not covered by buttons can theoretically be used to move the node. 🤪

Great that it works :)

@moklick  i am trying to use custom shapes in drag and drop but in flow of that i am not getting how can i pass custom nodetype to side bar and graph may i missing something ?

I don't get your question. The sidebar is not part of the library so you can display anything you want. What is not working with your example? For me it looks good.

In that example i have added  connectionLineType="smoothstep"  And connectionLineStyle={{strokeWidth: 4}} is not working . it there something i had done wrong i don't understand.
Also in input,output and default node , the selected node'd outline is little bit darker so it is easy to identify which is selected but this thing is not working with custom node may i know what can be done for same?

In that example i have added  connectionLineType="smoothstep"  And connectionLineStyle={{strokeWidth: 4}} is not working . it there something i had done wrong i don't understand.
Also in input,output and default node , the selected node'd outline is little bit darker so it is easy to identify which is selected but this thing is not working with custom node may i know what can be done for same?

connectionLineType and connectionLineStyle seem to work. The presentation of the dragged element has nothing to do with react flow.

connectionLineType and connectionLineStyle seem to work. The presentation of the dragged element has nothing to do with react flow.


Continous WebJob restarts with "Always On" set in portal

In our AppService we have 2 continuous web jobs running. The AppService has the "Always On" set to "On". Even with this setup, WebJobs restart under 2 different cases
Case 1 is when the website is shutting down, but no-one has shut it down or restarted it:
[11/08/2017 12:38:19 > 63fceb: SYS INFO] WebJob is stopping due to website shutting down [11/08/2017 12:38:19 > 63fceb: SYS INFO] Status changed to Stopping [11/08/2017 12:38:24 > 63fceb: SYS INFO] WebJob process was aborted [11/08/2017 12:38:24 > 63fceb: SYS INFO] Status changed to Stopped [11/08/2017 12:38:24 > 63fceb: SYS INFO] WebJob singleton lock is released [11/08/2017 12:38:54 > 63fceb: SYS INFO] Status changed to Starting [11/08/2017 12:38:54 > 63fceb: SYS INFO] WebJob singleton setting is True [11/08/2017 12:38:55 > 63fceb: SYS INFO] WebJob singleton lock is acquired [11/08/2017 12:39:10 > 63fceb: SYS INFO] Run script 'run.cmd' with script host - 'WindowsScriptHost' [11/08/2017 12:39:10 > 63fceb: SYS INFO] Status changed to Running
Case 2 is when the WebJobs files have changed, but no-one has touched any of the files in the folder or even in the temp folder where it's running:
[10/28/2017 21:51:13 > de8f96: SYS INFO] Detected WebJob file/s were updated, refreshing WebJob [10/28/2017 21:51:13 > de8f96: SYS INFO] Status changed to Stopping [10/28/2017 21:51:18 > de8f96: ERR ] Thread was being aborted. [10/28/2017 21:51:18 > de8f96: SYS INFO] WebJob process was aborted [10/28/2017 21:51:18 > de8f96: SYS INFO] Status changed to Stopped [10/28/2017 21:51:18 > de8f96: SYS INFO] WebJob singleton lock is released [10/28/2017 21:51:18 > de8f96: SYS INFO] Status changed to Starting [10/28/2017 21:51:18 > de8f96: SYS INFO] WebJob singleton lock is acquired [10/28/2017 21:51:18 > de8f96: SYS INFO] Run script 'run.cmd' with script host - 'WindowsScriptHost' [10/28/2017 21:51:18 > de8f96: SYS INFO] Status changed to Running
We contacted Support and got nowhere. Support says that a change in the portal settings would make this happen (Seems like they didn't consider the 2 different cases). Support said in their logs someone had changed the settings in the portal, but we don't see it in our activity log and even when inspecting the files we don't see what the change was. Support says they have no way to getting us more details, but that log entry is lacking when we search for it. We have no entry of anyone changing the settings.
These are .NET Core WebJobs by the way. I don't know if it's a bug with .NET Core. There have been some times where we have changed the files, but normally we shut it down before that happens. Sometimes we have restarted the AppService, but not every time the log shows these entries. We don't have an app.config, so I'm not sure if that's even relevant. Most posts are not about .NET Core, so it's hard to find anybody with a similar situation.
One thing we thought it was at first was the auto-publish when doing a git-merge, but the times don't match at all, we're talking about sometimes hours and even days differences in publishing when the WebJob(s) restart.
Please, what do you need from us? We can send an email with credentials to get on our account if needed or do you just need our config files? It's such a special case that I'm not sure what information is relevant. This has been on-going for over a month now and it's just a big hurdle. We started talking to the Google Cloud Platform team to see what options they have for us, but if we can find a solution here we would prefer not to migrate.

Anybody at all? We really love Azure, but this bug (or whatever it is) is not a real hit with us. CS is useless. We would love to avoid switching over to Google, but we have started our talks with Google and are looking to setup a migration plan. Would definitely love to avoid all of that, but we need some help guidance on this bug.

can you share your Web App name, directly or indirectly, so we can take a look

Make sure you provide all times as UTC. Also, please include the support case number for full context.

The case number is<PHONE_NUMBER>59158.
I created a dummy site: dummy-app-ra
The 2 affected sites are:

ends with l
starts with w

I can give you the dates of when we detected originally, but this has been happening every other day, if not every day.
10/25/2017 23:38:04
10/25/2017 02:00:02
We appreciate the help. If anything I missed, let me know and so I can provide.

@pabloarista some of the logs only go back one month, so that is a little too old. Can you provide a more recent time for the site that starts with 'w' (the one in the support case), and please make sure the time is UTC.

@pabloarista looking at the support case, it looks like it was closed just a couple days ago, as the support engineer was not getting a response from you. Did you see the latest mails that they had sent? I'd like to avoid redoing all the investigation here is they already have an ongoing investigation there.

Sorry, I didn't respond earlier. Yes the last thing CS said was to add to settings.job:
"is_in_place": true
This isn't a solution, all it's doing is not moving the assemblies to a temp folder. But yes CS did close out the ticket after this. Had some urgent matters that was unable to focus on that, life of working in a start up.  I do have some newer dates for you:
12/05/2017 17:14:30 Detected WebJob file/s were updated, refreshing WebJob 12/01/2017 19:12:19 WebJob is stopping due to website shutting down
Any ideas?

I also have the same behavior. while we didn't have any update to the app, webjobs just restart magically by itself.
[12/13/2018 23:08:55 > 7ae254: SYS INFO] WebJob is still running
[12/13/2018 23:09:00 > 66803a: SYS INFO] WebJob is still running
[12/14/2018 05:45:14 > 7ae254: SYS INFO] Detected WebJob file/s were updated, refreshing WebJob
[12/14/2018 05:45:14 > 7ae254: SYS INFO] Status changed to Stopping
[12/14/2018 05:45:15 > 66803a: SYS INFO] Detected WebJob file/s were updated, refreshing WebJob
[12/14/2018 05:45:15 > 66803a: SYS INFO] Status changed to Stopping
[12/14/2018 05:45:15 > 7ae254: SYS INFO] Status changed to Success
[12/14/2018 05:45:15 > 7ae254: SYS INFO] Status changed to Stopped
[12/14/2018 05:45:15 > 7ae254: SYS INFO] Status changed to Starting

This can happen due Azure AppService upgrade where we have to switch out a file server for maintenance.   As a result, continuous webjob may be restarted.

That would be insane if this is an ongoing thing every day because that’s what’s happening to us still now we’ll over a year. We modified all of our webjobs and stored procedures to just continue when this happens, because if not imagine we can lose over an hours worth of processing. We’re actually in talks with Google on migrating. There are a lot of good things with Azure, but also a lot of nuisances like this. I mean this is one of many explanations that I’ve heard from Microsoft. Doesn’t give us confidence in the service.

how were you able to get these logs?

@koverda you can go to https://appName.scm.azurewebsites.net/azurejobs/#/jobs
where appName is the name of your app service.
For us it stopped happening so often. We read buried on the interwebs somewhere that it could be related to the system resources. so we moved one of the offending webjobs to an isolated appService. still happens on occasion, but a lot less. This may not be a solution for everyone. Crazy thing is now we actually do need for it to restart automatically. the way it was doing previously on 1 webjob and that one completely stopped lol.
we've had such a bad experience with the webjobs (and Azure overall). we had to move 1 process to a VM and now there may be 1 other process (1 particular data calculation is what I mean by process) to that same VM because it has some real weird behavior.
We're not very happy with Azure. We started the migration to the Google Cloud Platform. We had a lot of meetings with them and explained this to them. They are on board and guarantee that these types of problems won't happen and have dedicated resources to help us to ensure if and when it does happen, to get it fixed fast. To date we still have no real response from MS Azure CS on the detailed error messages. the logs that are on the kudo site are very generic messages without the details.

Ah yep i'm familiar with that, unfortunately my job processes thousands of messages per minute, so going back in the logs is troublesome.

@koverda you can also go https://appName.scm.azurewebsites.net/ and go in either in the console or powershell and I believe it's the root/data/logs/jobs/ somewhere there, and you can download the logs. if you do get a lot of logs, Azure doesn't always clear these you have to clear them yourself. We've ran into this issue before. you can also download the publishing profile from the appService summary view. there you can ftps (or ftp if you're feeling adventurous lol) to the server and go to the same location and just download all of the logs in one shot. it would be easier to do a search. you can open up the folder on say VS Code and then search. could be much faster.

Anybody got a solution for this? We have a .NET v4.7 webjob which runs for months (20+) without any problem but the last 3 weeks when it is processing a large file which takes about 45minutes it shuts dows the website(!) and thus restarts the webjob. This happened before once in a while, so I've implemented a lot of code to handle this and to make sure the process is starting again when the webjob comes alive again, I'm using a Queue triggered webjob. Last night the large file was processed again. No errors where logged, only this:
[11/20/2019 17:31:26 > b5cf2c: SYS INFO] WebJob is stopping due to website shutting down
[11/20/2019 17:31:26 > b5cf2c: SYS INFO] Status changed to Stopping
[11/20/2019 17:31:31 > b5cf2c: SYS INFO] WebJob process was aborted
[11/20/2019 17:31:31 > b5cf2c: SYS INFO] Status changed to Stopped
[11/20/2019 17:31:50 > b5cf2c: SYS INFO] Status changed to Starting
[11/20/2019 17:31:50 > b5cf2c: SYS INFO] WebJob singleton setting is False
and several times later that night again:
[11/20/2019 17:31:26 > b5cf2c: SYS INFO] WebJob is stopping due to website shutting down
[11/20/2019 18:42:14 > b5cf2c: SYS INFO] WebJob is stopping due to website shutting down
[11/20/2019 21:17:21 > b5cf2c: SYS INFO] WebJob is stopping due to website shutting down
[11/20/2019 21:36:51 > b5cf2c: SYS INFO] WebJob is stopping due to website shutting down
[11/20/2019 21:58:11 > b5cf2c: SYS INFO] WebJob is stopping due to website shutting down
[11/20/2019 22:17:34 > b5cf2c: SYS INFO] WebJob is stopping due to website shutting down
[11/21/2019 03:13:41 > b5cf2c: SYS INFO] WebJob is stopping due to website shutting down
[11/21/2019 03:36:21 > b5cf2c: SYS INFO] WebJob is stopping due to website shutting down
Processing the large file started at:
[11/20/2019 16:46:30 > b5cf2c: INFO] Start processing 5436
and again at:
[11/20/2019 17:41:42 > b5cf2c: INFO] Start processing 5436
[11/20/2019 18:03:35 > b5cf2c: INFO] Start processing 5436
[11/20/2019 18:52:14 > b5cf2c: INFO] Start processing 5436
[11/20/2019 19:17:10 > b5cf2c: INFO] Start processing 5436
[11/20/2019 19:48:55 > b5cf2c: INFO] Start processing 5436
[11/20/2019 21:29:10 > b5cf2c: INFO] Start processing 5436
[11/20/2019 21:44:34 > b5cf2c: INFO] Start processing 5436
[11/20/2019 22:08:20 > b5cf2c: INFO] Start processing 5436
[11/20/2019 22:27:40 > b5cf2c: INFO] Start processing 5436
[11/20/2019 22:46:04 > b5cf2c: INFO] Start processing 5436
[11/20/2019 23:38:56 > b5cf2c: INFO] Start processing 5436
[11/21/2019 00:45:45 > b5cf2c: INFO] Start processing 5436
[11/21/2019 01:06:35 > b5cf2c: INFO] Start processing 5436
[11/21/2019 01:37:52 > b5cf2c: INFO] Start processing 5436
[11/21/2019 01:55:18 > b5cf2c: INFO] Start processing 5436
[11/21/2019 02:27:54 > b5cf2c: INFO] Start processing 5436
[11/21/2019 03:06:05 > b5cf2c: INFO] Start processing 5436
[11/21/2019 03:46:49 > b5cf2c: INFO] Start processing 5436
[11/21/2019 04:03:57 > b5cf2c: INFO] Start processing 5436
[11/21/2019 04:49:05 > b5cf2c: INFO] Start processing 5436
[11/21/2019 05:12:18 > b5cf2c: INFO] Start processing 5436
[11/21/2019 05:28:41 > b5cf2c: INFO] Start processing 5436
[11/21/2019 07:01:30 > b5cf2c: INFO] Start processing 5436
During the evening other (smaller) files do get processed due to
config.Queues.BatchSize = 8
At night the large file is the only file that is processed.
I'm now running the webjob locally in VS 2017 and no errors are reported and it finished as expected in 37 min.
The webjob is parsing a previously uploaded Excel-file and saves the data in a SQL Server table.
Smaller files (= less data) are no problem. Previously we had no problems with these large files.
Did Microsoft change some settings again in Azure which we need to adjust? I did notice the GUI of Azure Portal was also changes around 3 weeks ago.
We had to make changes previously to other parts of our application because of the sudden changes made by Microsoft.

@pmeems you said this has gone on for months without problems, but how big is the file? was always the same size? is there a lot of memory/CPU being used? I would assume that there were also no code changes in how the file is handled. I found out that when there are too many resources, it just restarts. doesn't tell you why, but you have access to that data too. you can confirm what it's using. if you have several webjobs and app services in the same app service plan, that happens. happened to us.

The file is between 115-125 MB. The most memory is used when saving the data to the database. When running locally in debug mode I can see the CPU goes up to 100%, but I have 8 cores and my other programs stay responsive.
I also thought about too many resources. That might be possible during the evening, but during the night it is the only process that is running. No other web jobs, no other files.
We already run an S2 website.
Can I somehow see what the reason for the restart is? If it is too many resources we could temporarily increase our website to an S3 just before parsing this large file. This specific file is only uploaded once a week.

@pmeems  Please provide a webapp name here or indirectly (my apology if it was in your explanation somewhere already) as well as a webjob name.    It is also useful to look up the information from https://docs.microsoft.com/en-us/azure/app-service/overview-diagnostics.

Thanks @suwatch for your assistance.
Our subscription is 20583...634867
The app service it the one ending with 2 and then the -prod slot.
The Webjob is ending with -WeeklyImports
The date is the night of 20 nov - 21 nov.
In the diagnostics I do see this:

MyWebApp[Kudu] is consuming an average of 61.29% of Physical Memory on instances consuming high Memory
MyWebApp[Kudu] has High CPU: 72.19%

Not sure what to do next.

Today it is happening again. The webjob restarts every 25-30 minutes. Around the time the file is processed and is saved using EF6.
So ProActive Auto-Healing wasn't the problem.
How can I tell Azure not to restart my webjob when it is a bit busy.
Or can we tell Azure not to give all available memory to our webjob, but just 60%?
We don't care it takes a bit longer if it can't allocate all the available memory.
I'm now running the webjob locally inside VS. And after 25 min. my fans are starting to blow, so it must be working hard. But my progress manager says it is just using around 55% of my CPU and a stable 53% (8.8GB) of my memory.

You are using S2 plan with 3.5 GB memory.   At 2019-11-27 10:37 UTC, SubAdvies.WeeklyImportsJob.exe job spiked to commit from 500MB to 5GB of memory for a sustaining period of time!   This caused the process to be unstable and shutdown.    If this is by design for your webjob to consume such, consider switch to S3 (7GB) plan instead.

I know the default answer of Microsoft for almost every issue is to upgrade to a more expensive plan, but we are already paying $$$$ for a small webapp (we also use PowerBI and ElasticPool).
I don't understand why Azure is shutting down my webjob when it finally has some work to do. Most of the time it is running on less than 10%, with some occasional spikes to 60% for about 5-10 minutes and once a week it needs all resources for about 30 minutes and this is not allowed.
Can I not tell Azure to leave my webjob to do its work and don't restart? Perhaps using a per session variable? Or can I limit my webjob to not consume more than 80%?

As @suwatch suggested it seems to be a resource issue. I've now created a tool that is splitting my large Excel sheet in multiple smaller versions and those smaller files are processed fine.
So, for now, it solves my problem.
But I still think the webjob should not restart on its own or at least log it somewhere why it is restarting.

Hi
If the problem persists and is related to running it on Azure App Service, please open a support incident in Azure:
https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request
This way we can better track and assist you on this case
Thanks,
Joaquin Vano
Azure App Service

the company switched to GCP a few years ago. I do consultant work for them so whenever I have to go on the backend I do see that this isn't an issue with GCP. I would hope by now Azure has fixed all of these issues, but I have no way to confirm any of it as the only work I do for Azure now is just an iOS app I created that deals with Azure Storage Tables.


Webpack Plugin Resolution

When attempting to webpack my application after adding the background-geolocation/background-fetch plugins it fails when trying to resolve the plugins. Is webpack supported by the backgorund-geolocation plugin? If so, what steps can I take to attempt to remedy the module resolution?
Your Environment

Plugin version:
Platform: iOS or Android
OS version: ^1.6.0
Nativescript version (tns info): 2.5.5
Plugin config

{
    license: '***',
    desiredAccuracy: BackgroundGeolocation.DESIRED_ACCURACY_LOW, // Default: HIGH
    distanceFilter: 10,
    stationaryRadius: 25, // 25 RAISE TO 100 when done testing
    useSignificantChangesOnly: true, // false
    activityRecognitionInterval: 120 * 1000, // 10000
    stopTimeout: 1,
    disableMotionActivityUpdates: true,
    maxRecordsToPersist: 1,
    stopOnTerminate: false,
    startOnBoot: true,
    heartbeatInterval: 300,
    debug: false,
    logLevel: BackgroundGeolocation.LOG_LEVEL_INFO,
    logMaxDays: 3
}

Expected Behavior
When running webpack the plugin(s) should resolve
Actual Behavior
Webpack fails to resolve the plugins
Webpack Error
ERROR in ./core/services/background-location.service.ts
Module not found: Error: Can't resolve 'nativescript-background-geolocation-lt' in '/Users/shawnpavel/Development/mobileClassic/app/core/services'
 @ ./core/services/background-location.service.ts 3:47-96
 @ ./app.module.ngfactory.ts
 @ ./main.aot.ts

ERROR in ./shared/utils/background-geolocation.util.ts
Module not found: Error: Can't resolve 'nativescript-background-geolocation-lt' in '/Users/shawnpavel/Development/mobileClassic/app/shared/utils'
 @ ./shared/utils/background-geolocation.util.ts 3:47-96
 @ ./shared/utils/index.ts
 @ ./shared/providers/firebase-remote-config.ts
 @ ./app.module.ngfactory.ts
 @ ./main.aot.ts

ERROR in ./shared/utils/app-event-setup.util.ts
Module not found: Error: Can't resolve 'nativescript-background-fetch' in '/Users/shawnpavel/Development/mobileClassic/app/shared/utils'
 @ ./shared/utils/app-event-setup.util.ts 3:38-78
 @ ./shared/utils/index.ts
 @ ./shared/providers/firebase-remote-config.ts
 @ ./app.module.ngfactory.ts
 @ ./main.aot.ts

The plugin is not involved with Webpack.  This is NativeScript responsibility.

The issue appears to be that module resolution depends on the package export either being index.js OR remove the .js extension in the package.json file.  I'm not sure what the issue in resolution is, but there is definitely a difference in the way the two cases are resolved.

It's how webpack works actually. So it's fine to remove the extension.

@bradmartin Out of curiosity, does that difference in resolution happen outside the NativeScript ecosystem?

It works fine outside of NS.

https://github.com/bradmartin/mystical-notification/blob/master/package.json#L5 Here is an example of a browser thing I wrote which uses webpack to build, notice the package json main key 👍

I suppose the plugin doesn't have to drop the .js extension but it doesn't matter to the plugin to have the extension, it just lets the plugin support webpack and all NS apps 👍 so whatever. I don't use it so doesn't affect me really.

So the resolve.extensions field in webpack.config dictates file type I suppose?

I can confirm that removing the .js from the main value resolved the issue.

That I do not know about. I'm no webpack expert, I just know you don't have to specify the extension in the package.json for webpack to resolve a module. Outside of that, I don't know much about it.

Gotcha, appreciate the discussion @bradmartin.  It's nice to run across people who are actually helpful.=)

@ShawnPavel do you want me to change in my package.json?  I'm due to push a big update soon.

@christocracy If you were to change it to not require the .js it would allow anyone who uses the plugin to webpack. So I would say. I would appreciate it very much. This same issue is present in the background-fetch plugin as well.

So `"main": "background-geolocation" ?

Yes, sir.

nativescript-background-geolocation-lt @ 1.7.0 and nativescript-background-fetch @ 1.0.1 are now published.  .js removed from package.json in both plugins.


Update cloud-services-serverexplorer-create-cloud-container.md

Hi @CawaMS, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no contribution license agreement is required at this point. Real humans will now evaluate your PR.

TTYL, AZPRBOT;


feat: table GitHub pr comments

table that pulls each comment for each pr in a repository

Codecov Report

Merging #206 (e79e5d2) into main (38045ae) will increase coverage by 0.38%.
The diff coverage is 77.67%.

@@            Coverage Diff             @@
##             main     #206      +/-   ##
==========================================
+ Coverage   69.60%   69.98%   +0.38%
==========================================
  Files          44       45       +1
  Lines        2227     2339     +112
==========================================
+ Hits         1550     1637      +87
- Misses        435      449      +14
- Partials      242      253      +11

Impacted Files
Coverage Δ

extensions/internal/github/github_pr_comments.go
77.27% <77.27%> (ø)

extensions/internal/github/github.go
72.50% <100.00%> (+1.44%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 38045ae...e79e5d2. Read the comment docs.


Documentation

There is a serious lack of documentation in this project. Adding documentation will be a boon for future use.

Hello, I am looking to contribute to open source projects here on GitHub, as part of my university module. I see you have an issue with regards to documentation of this project and would like to help. Thanks, karna

What experience do you have in the problem domain?

Not much, depends on the issue... would you mind telling me in more detail of what it needs to be done and then i could see

Being able to read and write modern C++ is a must. The required work involves reading existing code, and providing Doxygen that details the observable behaviour, preconditions that must be met, postconditions that need to be upheld, and exceptions that can be thrown.
You probably don't need graphics programming experience, but that will probably be useful too.
doge::wrap is an example of what the documentation should look like: https://github.com/cjdb/doge/blob/master/include/doge/gl/texture.hpp

I see. I haven't learned c++ and in all honesty, i don't think I could fix the issue, had also asked a mate to, however, same reason falls through.