SSR performance issue with HTML in markdown

I've found a strange bug in v6.6.8 that is seriously impacting performance during SSR. When the markdown contents have a specific combination of colons and HTML, it sometimes takes around 30-40 seconds to render (on a 2.6GHz i5 Mac Mini), and will occasionally completely freeze. It also pushes the CPU usage to 100% for the entire duration of the program.
We had some auto-triggered builds that were generating large static markdown sites, and the builds started to freeze and timeout since 6.6.8 was released. I've put together a simple example that is able to replicate this on a small scale:
index.js
import fs from 'fs'
import React from 'react'
import Markdown from 'markdown-to-jsx'
import { renderToString } from 'react-dom/server'

// renders immediately
const good = `# This is a normal markdown file. <a href="/">Foo Bar</a>`
console.log(renderToString(<Markdown>{good}</Markdown>))

// takes a long time with high CPU
const bad = `Lorum *ipsum*: <a href="" style="float: right"><small>foo</small></a><span style="float: right"><small> </small></span><a href="" style="float: right"><small>bar</small></a>`
console.log(renderToString(<Markdown>{bad}</Markdown>))

package.json
{
  "scripts": {
    "start": "babel-node index.js"
  },
  "dependencies": {
    "babel-cli": "6.26.0",
    "babel-core": "6.26.3",
    "babel-preset-env": "1.7.0",
    "babel-preset-react": "6.24.1",
    "markdown-to-jsx": "6.6.8",
    "react": "16.4.1",
    "react-dom": "16.4.1"
  },
  "babel": {
    "presets": [
      "env",
      "react"
    ]
  }
}

I haven't been able to nail down exactly which part of the bad markdown string is causing the issue, as removing different pieces will make the render time fluctuate--sometimes making it longer, sometimes shorter. It's weird. Downgrading to v6.6.7 fixes the issue.

Might be a funky regex recursion situation. I’ll take a look!
On Fri, Jul 6, 2018 at 4:34 PM Jason Maurer<EMAIL_ADDRESS>wrote:

I've found a strange bug in v6.6.8 that is seriously impacting
performance during SSR. When the markdown contents have a specific
combination of colons and HTML, it sometimes takes around 30-40 seconds to
render (on a 2.6GHz i5 Mac Mini), and occasionally will completely freeze.
It also pushes the CPU usage to 100% for the entire duration of the
program. I've put together a simple example that is able to replicate this:
index.js
import fs from 'fs'import React from 'react'import Markdown from 'markdown-to-jsx'import { renderToString } from 'react-dom/server'
// renders immediatelyconst good = # This is a normal markdown file. Foo Barconsole.log(renderToString({good}))
// takes a long time with high CPUconst bad = Lorum *ipsum*: foo barconsole.log(renderToString({bad}))
package.json
{
"scripts": {
"start": "babel-node index.js"
},
"dependencies": {
"babel-cli": "6.26.0",
"babel-core": "6.26.3",
"babel-preset-env": "1.7.0",
"babel-preset-react": "6.24.1",
"markdown-to-jsx": "6.6.8",
"react": "16.4.1",
"react-dom": "16.4.1"
},
"babel": {
"presets": [
"env",
"react"
]
}
}
I haven't been able to nail down exactly which part of the bad markdown
string is causing the issue, as removing different pieces will make the
render time fluctuate (sometimes making it longer, sometimes shorter.)
Downgrading to v6.6.7 fixes the issue.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/probablyup/markdown-to-jsx/issues/190, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AAiy1tcZn-916Nf58NeboJcOcumym7Q8ks5uD9fTgaJpZM4VGDcE
.

Fixed in 6.6.9


Be aggressive about removing all intermediate containers

Autumn has been having space issues (and has been running out of inodes). It's not clear what is causing this, since docker system prune doesn't seem to help. We already run docker run with --rm argument. This commit changes the build also to remove intermediate containers in case of build failures.

Interesting :/ I poked around and ran more docker pruning (which didn't help much), but there's something I don't understand with the docker overlay2 directories:
$ df --inodes
Filesystem                  Inodes    IUsed     IFree IUse% Mounted on
/dev/mapper/vg_boot-data  13033472 11697303   1336169   90% /data
                                   ^^^^^^^^

$ ls /data/docker/overlay2 | wc -l
376

It looks like the overlay2 contain various copies of the opam-repository, which accounts for 6m of inodes (of the 11m used) :
$ cat <the-following> | awk '1{s+=$1} END{print s}'
6262728

$ find . -name 'opam-repository' | grep -v 'packages/opam-repository' | xargs -n 1 du --inodes --summarize
158132	./8d9c7622c5da83b9fc97eb5b35d15c1a091c67f6ac510cb420d2af8723298e9a/diff/home/opam/opam-repository
163196	./e1d20c07bfee4daede15d45acb60099946cd5d081454119349e4636d9676b2ad/diff/home/opam/opam-repository
158979	./9ae3f8364f3224c219b23991db042949d22dbb99c635fffce4fd344a4971db81/diff/home/opam/opam-repository
162945	./58f68ebcbcda9dc5c70fe24939517b5cefcc47f61469c3223269de29a11d00db/diff/home/opam/opam-repository
160626	./65c845f4119446ba159aecb67f1e8d5e58384198bb8bc9a2c07abfd7b0d861c9/diff/home/opam/opam-repository
160237	./ebd0d0da0a46d151489345565a9a637b4fc68414491e2d694f2b46108c1f9f9b/diff/home/opam/opam-repository
159933	./526c93367c739f7dbfd63b445193442317ceb836ddbc209b9b83277c29f8b2aa/diff/home/opam/opam-repository
162455	./8f3de902206ac0203291b6125079e5547b88eeac29ebb306676ec0b6f07a4001/diff/home/opam/opam-repository
161983	./63f97ea97e5efc15af3472fa82df599d44da3573e264f2cafb144e0b278eb657/diff/home/opam/opam-repository
163196	./4b7ff32e31e4670ba225d74d9c0c1fe19853b5d78a6ce15aa6267e040537673c/diff/home/opam/opam-repository
162945	./fbd1f3b5ad1337a73a0a1fe773246ffe8663d481ffa9be4f6e9dcf067d402f66/diff/home/opam/opam-repository
160092	./13b185ec4cb9d780a7c25051bbeb5da9392c553a68853e29579beaaf1cc4e7a7/diff/home/opam/opam-repository
160940	./1a5c890c385ee1e6be5585b76decf4298969912c920b188cb02834d4048b323c/diff/home/opam/opam-repository
158725	./3321809141f44bb75abb43f079c09cc2923bc07b20c1a621704515efe2138381/diff/home/opam/opam-repository
160235	./62d8d3fa1f05c55689005b0bb5e5719c2f618c0d62dec3b862693b54f03a8558/diff/home/opam/opam-repository
7	./tjoeyzh1257fnrjmb0b8sn990/diff/src/opam-repository
157643	./371480b9ed57b53bdc8495f258cbb70fde20882676f9bb7933616063b01f0457/diff/home/opam/opam-repository
161763	./5f050a5df47f336243e15ad60a1d1dc91cd2a1ab2a3ca97b77d5a7cabcd7a9ec/diff/home/opam/opam-repository
157116	./b85ea590205f118781e1bbb448fa279f94dd03653f4f8f694ca09c304ecf8002/diff/home/opam/opam-repository
158521	./457e26059d439457efc10332061c8f1d04abdbf3edac884580383e1502c070dc/diff/home/opam/opam-repository
161983	./2ef48424c7374673593f919ab443319d089814b9cf0c99bdad69de562e4976b1/diff/home/opam/opam-repository
158254	./97d6e62ca51a0b215ec23cd72e9984b832bfc1eff20b48622d365ee5b072c007/diff/home/opam/opam-repository
3702	./lwxghksbh8d1skc4tmx5pp08i/diff/src/opam-repository
160098	./2cc40f56e5f0d85085722fbebe3680144906f36bfa7009cec3ead1e17dc453de/diff/home/opam/opam-repository
158621	./e9d9cfda2f7bc577d68de592189ddd016a5d2d42df891261a494f2b6798d7b77/diff/home/opam/opam-repository
156695	./644cf526c2351a0f456b96bca35fb5851225d309fc4a4886e2696c942b1f09be/diff/home/opam/opam-repository
156221	./6f968e34071e3ac4ce1c0c6a85f3b73bf749ef201cd59dd580c36945af485584/diff/home/opam/opam-repository
2	./uiioy4lwo0msm688dkibvjs2r/diff/home/opam/.opam/5.0/.opam-switch/overlay/opam-repository
11	./uiioy4lwo0msm688dkibvjs2r/diff/home/opam/.opam/5.0/.opam-switch/sources/opam-repository
162455	./3c481b00d911fd12c6b182db597fca6b5669cc80cdb47a95da2de9314ba36c50/diff/home/opam/opam-repository
162202	./ead3220a0dff5d2c7b49a33f057270227685c0efa6c9b0634b0f4f71337a7b91/diff/home/opam/opam-repository
158340	./97a4973997c62f0d7bee6ed9e2a98cd099eb7d380324551e6ab751333e64f2d1/diff/home/opam/opam-repository
156337	./54abd607b2242d36cd933221e447808b94bdbe4e4220a62e85aec821fe84ccc3/diff/home/opam/opam-repository
157272	./3a471f284ba5275ffc2ccf2bee333e79785dec2adad72354a99aec636f0189d8/diff/home/opam/opam-repository
157388	./18df9b7873e1bd86a22140b6ce25911a4b4d87f4633e08d5e7511e90cc78377c/diff/home/opam/opam-repository
158586	./3b4a554a350318c47ca6b570570548871ff50a4cc912a8fc64bb93441bd5afe7/diff/home/opam/opam-repository
158240	./6d8611846428aee1e7762ddf9823b8e7b11e3b1707fc383bc7b398a924d00428/diff/home/opam/opam-repository
155238	./a21b015b4ea4ad8b48ffa54ffafc8e1dc5cc1f5374ac4268f5e52d46a084f764/diff/home/opam/opam-repository
159297	./401dfb26517d289c1787d33d47436b4e52540148ff7bdb755a8a27f65606ebe1/diff/home/opam/opam-repository
160909	./b96cf2942ca8a5de6e54bfc3b86fdb8bef6cca09c35a74785dcd8066ef14d260/diff/home/opam/opam-repository
158032	./97cbe2e7cac83cfec1f11fd6308721e3d68b280aa03183fcf4c087d5ae9f60cf/diff/home/opam/opam-repository
161094	./a3a0aa4a7b9b559d830a10fc628b1893bc5ac143ddbefdf873139921d69aa49a/diff/home/opam/opam-repository
156082	./bbbd77d846a0c82a38467bf893e9c85af641e12acd31ecd64338a3c5dacd1105/diff/home/opam/opam-repository
36000	./w6ewat90g5etzyw8zpa7w0t1y/diff/src/opam-repository

Why not, but some of those directories are a bit old:
drwx------ 3 root root 4.0K Mar 26 08:52 a21b015b4ea4ad8b48ffa54ffafc8e1dc5cc1f5374ac4268f5e52d46a084f764
drwx------ 3 root root 4.0K Mar 27 17:32 bbbd77d846a0c82a38467bf893e9c85af641e12acd31ecd64338a3c5dacd1105
drwx------ 3 root root 4.0K Apr  3 23:25 54abd607b2242d36cd933221e447808b94bdbe4e4220a62e85aec821fe84ccc3
drwx------ 3 root root 4.0K Apr  6 10:24 6f968e34071e3ac4ce1c0c6a85f3b73bf749ef201cd59dd580c36945af485584
...

And I can't find why docker retains them after all of our system pruning:
$ docker inspect $(docker ps --all -q) | grep a21b0
$ docker image inspect $(docker image ls --all -q) | grep a21b0

Any idea? I don't want to remove them manually as I don't understand why they are there :P Perhaps we could stop all of our dockers, run the system prune, and see if anything remains?

Any idea? I don't want to remove them manually as I don't understand why they are there :P Perhaps we could stop all of our dockers, run the system prune, and see if anything remains?

It may be worth trying, but the service will be down for a while, before all the current-bench containers build.
@mtelvers and I tried to remove all of the overlay2 directories that seemed unused, but that caused build failures. But, later I moved out a handful of them using dates, and it did create some space for redeploying the production instance.  There's a thread on #ci-dev that I'll DM you on slack, which might add more information.

I have created my own implementation of current bench to try to understand what is happening.  Building the system creates around 18GB of Docker images.  Then for each run of the test project there seem to be three directories left over in the docker/overlay2 folder.

Having experimented some more on my installation, I propose the following course of action:

Stop Docker, and edit /etc/docker/docker.json and set a new Docker root on the spare spinning disk in the machine
Deploy a empty copy of current-bench in the new Docker installation
Copy the Docker volumes over from the original installation (thereby taking a copy of previous jobs logs and importantly the database)

If this proves successful, we can move this installation on to the SSD.  If it fails, edit docker.json and we are back where we are now.

See also https://github.com/ocaml/infrastructure/issues/70

Having experimented some more on my installation, I propose the following course of action:

I've followed the steps proposed by you, and the docker service is now using the spinning disk. If everything works fine, overnight, we could plan on moving things to the SSD.

The docker service now uses the SSD.


Fix the Simulation Manager to fail properly in case of GAMA failure.

Changed Bee to Bull.
Can better handle failures and errors now.
Need to make it better at logging, and report failures on redis connection, DB connections etc.

Check tripgen calculations
Work on instance termination, on a second trigger when trips are generated.
Create CW dashboard for the incoming logs.
Test with various charging scenarios.

Use EC2 role with worker instances instead of API keys.


translated the first folder

First folder done.

Please follow the issue https://github.com/ardanlabs/gotour/issues/71 for the PR name


This library can be used to fake fingerprints in the browser starting with selenium?

No. The use case of the library is calculating a hardware fingerprint when running inside a an application running on an actual PC/VM.


Fabric workers should exit if the client exits

Overview
Fabric workers should exit if the client exits.
I enhanced the dev haproxy.cfg so that we can see haproxy log output too (I also capture the x-couch-request-id so we can match it to couch.log output)
Note: I make another copy of all the jobs in order to index them by the new client monitor reference. The job record is small, two pids and three references, so I'm not hugely concerned. However we could eliminate it and do a table scan in the fallback case in DOWN if the Ref is not a worker itself.
Testing recommendations

start dev/run with log level debug
make a new empty database
tail couch.log of all nodes in other terminals
in a terminal run curl -i 'localhost:5984/dbname/_changes?feed=continuous&since=now&timeout=100000000000000000000'
CTRL-C in the curl terminal
notice the debug log for two fabric workers that exit (q=2 by default)

I intend to add automated tests for this but wanted to get the diff in front of the team first.
Related Issues or Pull Requests
N/A
Checklist

[x] Code is written and works correctly
[ ] Changes are covered by tests
[ ] Any new configurable parameters are documented in rel/overlay/etc/default.ini
[ ] Documentation changes were made in the src/docs folder
[ ] Documentation changes were backported (separated PR) to affected branches

good point on fabric_doc_update and possibly others.
I confirmed directly and via haproxy that a client that disconnects (curl and CTRL-C in the first case and simply doing something that hits haproxy's server timeout like _changes?feed=continuous&timeout=10000000) that the mochiweb process is killed, and this triggers the new code above to take down workers.

I confirmed directly and via haproxy that a client that disconnects (curl and CTRL-C in the first case and simply doing something that hits haproxy's server timeout like _changes?feed=continuous&timeout=10000000) that the mochiweb process is killed, and this triggers the new code above to take down workers.

Cleanup happens without this patch already form what I've observed. Here is how I tested on main:
diff --git a/src/chttpd/src/chttpd_db.erl b/src/chttpd/src/chttpd_db.erl
index e2de301b2..6c2beba83 100644
--- a/src/chttpd/src/chttpd_db.erl
+++ b/src/chttpd/src/chttpd_db.erl
@@ -127,6 +127,8 @@ handle_changes_req1(#httpd{} = Req, Db) ->
         db_open_options = [{user_ctx, couch_db:get_user_ctx(Db)}]
     },
     Max = chttpd:chunked_response_buffer_size(),
+    couch_log:error(" +++TRACING _changes++++ ~p:~p@~B REQPID:~p", [?MODULE, ?FUNCTION_NAME, ?LINE, self()]),
+    dbg:tracer(), dbg:p(self(), p),
     case ChangesArgs#changes_args.feed of
         "normal" ->
             T0 = os:timestamp(),
diff --git a/src/fabric/src/fabric_db_update_listener.erl b/src/fabric/src/fabric_db_update_listener.erl
index 78ccf5a4d..fb508294a 100644
--- a/src/fabric/src/fabric_db_update_listener.erl
+++ b/src/fabric/src/fabric_db_update_listener.erl
@@ -37,6 +37,8 @@
 }).

 go(Parent, ParentRef, DbName, Timeout) ->
+    couch_log:error(" +++TRACING UPDATE NOTIFIER+++ ~p:~p@~B ~p Parent:~p", [?MODULE, ?FUNCTION_NAME, ?LINE, DbName, Parent]),
+    dbg:tracer(), dbg:p(self(), p),
     Shards = mem3:shards(DbName),
     Notifiers = start_update_notifiers(Shards),
     MonRefs = lists:usort([rexi_utils:server_pid(N) || #worker{node = N} <- Notifiers]),
@@ -82,6 +84,8 @@ start_update_notifiers(Shards) ->

 % rexi endpoint
 start_update_notifier(DbNames) ->
+    couch_log:error(" +++TRACING UPDATE NOTIFIER WORKER+++ ~p:~p@~B~p", [?MODULE, ?FUNCTION_NAME, ?LINE, DbNames]),
+    dbg:tracer(), dbg:p(self(), p),
     {Caller, Ref} = get(rexi_from),
     Notify = config:get("couchdb", "maintenance_mode", "false") /= "true",
     State = #cb_state{client_pid = Caller, client_ref = Ref, notify = Notify},
diff --git a/src/fabric/src/fabric_rpc.erl b/src/fabric/src/fabric_rpc.erl
index fa6ea5116..64fdbf4b5 100644
--- a/src/fabric/src/fabric_rpc.erl
+++ b/src/fabric/src/fabric_rpc.erl
@@ -69,6 +69,8 @@ changes(DbName, Args, StartSeq) ->
 changes(DbName, #changes_args{} = Args, StartSeq, DbOptions) ->
     changes(DbName, [Args], StartSeq, DbOptions);
 changes(DbName, Options, StartVector, DbOptions) ->
+    couch_log:error(" ++TRACING CHANGES WORKER+++++ ~p:~p@~B~p", [?MODULE, ?FUNCTION_NAME, ?LINE, DbName]),
+    dbg:tracer(), dbg:p(self(), p),
     set_io_priority(DbName, DbOptions),
     Args0 = lists:keyfind(changes_args, 1, Options),
     #changes_args{dir = Dir, filter_fun = Filter} = Args0,

That's setting up process even traces on the request process, worker process, db updater process and db update worker process. The goal is that these should be cleaned as soon as we attempt any write to the closed socket.
(db is Q=1 empty db)
% curl -i $DB/db/_changes'?feed=continuous&since=now'
HTTP/1.1 200 OK
...

^C

[error] 2023-08-18T22:54:36.654375Z<EMAIL_ADDRESS><0.949.0> cc6744542c  +++TRACING _changes++++ chttpd_db:handle_changes_req1@130 REQPID:<0.949.0>
(<0.949.0>) spawn <0.974.0> as erlang:apply(#Fun<rexi_monitor.1.77517421>,[])
(<0.949.0>) link <0.974.0>
(<0.949.0>) getting_unlinked <0.974.0>
[error] 2023-08-18T22:54:36.654949Z<EMAIL_ADDRESS><0.976.0> --------  +++TRACING UPDATE NOTIFIER+++ fabric_db_update_listener:go@40 <<"db">> Parent:<0.949.0>
(<0.949.0>) spawn <0.976.0> as fabric_db_update_listener:go(<0.949.0>,#Ref<0.830586435.801374217.85503>,<<"db">>,60000)
(<0.949.0>) link <0.976.0>
(<0.949.0>) spawn <0.978.0> as erlang:apply(#Fun<rexi_monitor.1.77517421>,[])
(<0.949.0>) link <0.978.0>
(<0.949.0>) spawn <0.979.0> as erlang:apply(#Fun<fabric_streams.3.106508571>,[])
[error] 2023-08-18T22:54:36.655018Z<EMAIL_ADDRESS><0.977.0> cc6744542c  ++TRACING CHANGES WORKER+++++<EMAIL_ADDRESS>(<0.976.0>) spawn <0.980.0> as erlang:apply(#Fun<rexi_monitor.1.77517421>,[])
[error] 2023-08-18T22:54:36.655123Z<EMAIL_ADDRESS><0.982.0> --------  +++TRACING UPDATE NOTIFIER WORKER+++<EMAIL_ADDRESS>(<0.976.0>) link <0.980.0>
(<0.976.0>) spawn <0.981.0> as erlang:apply(#Fun<fabric_db_update_listener.2.111341222>,[])
(<0.977.0>) exit normal
(<0.949.0>) getting_unlinked <0.978.0>

(<0.976.0>) exit normal
(<0.982.0>) exit killed
(<0.949.0>) exit shutdown

0.949.0 request process is killed with a shutdown as expected. Probably after trying to write a closed socket.
0.976.0 db update notifier exits normal
0.982.0 db update notifier worker is killed by the cleaner process

Since db is empty we're not actually starting any workers but those should also by cleaned by the cleanup helper process in fabric_streams.

@nickva 😱 interesting!
What we've observed (a huge rise in interactive ioq requests after a series of _find requests that haproxy classified as sH--, and the new mango_stats showing a mango process that exits some hours after those after having examined the same number of documents as the database has in total) can only currently be explained by these processes simply continuing despite the client having gone away (the client being haproxy here, it closes the connection after its own timeout server interval).
I can confirm what you're seeing, and I have also done so for an unindexed _find request by adding timer:sleeps in various places of the remote side of fabric_view_all_docs.
So I guess we're looking for a gap in the fabric_streams error handling rather than a complete absence of orphaned worker cleanup code.

took me too long to realise, but I think the difference is the attempt to send data (between _changes and _find). that's the only time we notice the socket closed, and thus the processes exit (and all the cleanup you pointed out then happens).
The mochiweb http process does stay alive after the sH-- event. If I kill it (from remsh) then my cleanup kicks in and kills the orphaned workers. without my change the fabric_streams cleanup kills them.

took me too long to realise, but I think the difference is the attempt to send data (between _changes and _find). that's the only time we notice the socket closed, and thus the processes exit (and all the cleanup you pointed out then happens).

Yup that's the key. We have to read, write or find some other way to periodically check the socket state and then terminate the connection process. The stream cleanup mechanism as is will only work as long as the connection process dies. We looked at this before and there was no simple way to detect it out-of-band in Erlang from what I could see.
We could try to emit empty line heartbeats in _find this might be the simplest solution.
If there was a socket recv with options function in Erlang we might have been able to periodically try to recv with a MSG_PEEK option, to pretend we're reading the data and see what's there, I don't know if that's possible currently.  Maybe there is some non-portable raw inet:getopts/2 option we can query but I think just trying periodic pings and flushing them should do it.

In principle at least the experimental socket module does allow peek-ing with recv since OTP 24.0: https://www.erlang.org/doc/man/socket#type-msg_flag

msg_flag() =
    cmsg_cloexec | confirm | ctrunc | dontroute | eor | errqueue |
    more | oob | peek | trunc

But we can't use socket for non-socket connections and it might be a while if/until mochiweb starts using that module. Especially since in some cases socket has been performing as well as the old inet driver.

hm, not seeing a way to change count/idle/interval, so I bet the macos defaults are quite high.

On localhost keepalive won't behave any differently for disconnects, as network handling is just a direct callback into the other peer's socket code at the kernel level. In other words the socket will find out it's closed anyway. For remote connection it's a different story and keepalive will be useful there.
So keepalive is useful as a step 0 - to allow the socket at the kernel level to become closed. However that is not enough for us, what is missing is at the HTTP protocol stage in mochiweb we do not have a reason do any socket operations (and we can't do any extra ones outside of mochweb in a sneaky way so far it seems, which is what I was getting at above). We're forced to wait for the mango application to return a response. Without any attempted writes or reads to socket at the mochiweb level won't have a way to find out the socket is closed. Which could come after days of waiting in pathological cases like we saw already.
The only solutions I see so far is to either modify mango to periodically handle a timeout and push a \n between row (after forcing to emit headers on the first timeout) or find some possibly non-portable raw socket getops query to detect that the socket is closed out-of-band as it were. We could enhance the existing fabric_stream monitor to periodically check it or invent some new thing (maybe some new thing in mochiweb even).

I'm not entirely giving up on tcp keepalive (it ought to kill the process/port for the socket, which I think will kill the mochiweb process, but subject to testing)
However, I'm beginning to think what we need is application level. There's clearly a modest, finite number for how long we're prepared to wait for a response to start, which Cloudant, for example, already expresses in haproxy timeout server. So I think couchdb needs a similar setting, that if a response hasn't begun in that timeframe, we exit (and fabric_streams cleanup then triggers). The only endpoint that delays the start of the response by a potentially very long time is _find, because it can discard work. A filtered changes response has at least started the response even if nothing passes the filter.

it ought to kill the process/port for the socket, which I think will kill the mochiweb process, but subject to testing

It doesn't hurt to try I don't see how that could plausibly work. There is no active socket monitoring or polling in mochiweb. If we were in active mode or were in C and used a select/poll/epoll we'd probably find out about it.

The only ways I see as viable are:

Add a mango application level timeout like changes has to emit a periodic newline between json rows

Add an active custom socket inactivity process with a timeout. Either as part of mochiweb, fabric stream or a chttpd. After we receive the request we spawn a process which starts timer. When we send or receive from the socket we reset the timer, if we fail to send or receive anything on a socket the timer expires and blows up killing the connection process. But that involves reseting an extra timer and an extra process so it has performance implications. Maybe we can do something clever with erlang counters there...

changed approach, though I don't think this is the final answer either.
this diff adds a deadline by which a delayed response must start or be terminated. This helps when using _find if it takes a long time to find the first match (or if it's plowing through a large database via _all_docs and won't find any matches)
It's not broad enough but the previous effort with monitors was pointless (nick is correct that fabric_streams cleans up if the parent process exits)

I managed to find a way to seemingly detect the connection state change after the local client (curl) disconnects by querying the equivalent of getsockopt(fd(), IPPROTO_TCP, TCP_CONNECTION_INFO, &info, &info_size)
(node1@<IP_ADDRESS>)22> rp(inet:getopts(#Port<0.24>, [{raw, 6, 262, 1000}])).
{ok,[{raw,6,262,
          <<4,6,6,0,7,0,0,0,0,0,0,0,0,0,0,0,204,63,0,0,0,192,255,
            63,18,0,2,0,0,57,6,0,0,0,0,0,88,58,6,0,1,0,0,0,1,0,0,
            0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,177,1,0,0,0,0,0,0,
            0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,148,0,0,0,0,0,0,0,0,
            0,0,0,0,0,0,0,0,0,0,0,0,0,0,0>>}]}
ok

Ctrl+C...
(node1@<IP_ADDRESS>)24> inet:info(#Port<0.24>).
#{counters =>
      #{recv_avg => 24,recv_cnt => 6,recv_dvi => 10,recv_max => 51,
        recv_oct => 148,send_avg => 216,send_cnt => 2,
        send_max => 427,send_oct => 433,send_pend => 0},
  input => 0,
  links => [<0.12859.0>],
  memory => 40,monitors => [],output => 433,
  owner => <0.12859.0>,
  states => [connected,open]}

(node1@<IP_ADDRESS>)25> rp(inet:getopts(#Port<0.24>, [{raw, 6, 262, 1000}])).
{ok,[{raw,6,262,
          <<5,6,6,0,7,0,0,0,0,0,0,0,0,0,0,0,204,63,0,0,0,192,255,
            63,18,0,2,0,0,57,6,0,0,0,0,0,88,58,6,0,59,58,0,0,1,0,
            0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,177,1,0,0,0,0,0,
            0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,148,0,0,0,0,0,0,0,
            0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0>>}]}
ok

Looks like Linux has a similar facility so there is some hope, though it does look very ugly

updated again. I've expanded fabric_streams existing 'cleanup' process to be a watchdog (in the classic sense).
before starting a fabric_stream you must call fabric_stream:enable_watchdog(). If you do, the cleanup process starts a timer. If the watchdog is not kicked at least once (with fabric_stream:kick_watchdog().) before the timer fires, the watchdog kills the coordinator process (for us, the mochiweb/httpd process). this in turn causes the watchdog to get a DOWN message as it was monitoring the coordinator, and it then cleans up the workers.
by default this is not active, so all calls to fabric:all_docs etc are unaffected.
in mango_httpd I enable the watchdog and only when adding a row to the response do I kick the watchdog. If we go a long time between rows, the watchdog fires.
similar enable/kick can be done on other endpoints.

I managed to find a way to seemingly detect the connection state change after the local client (curl) disconnects by querying the equivalent of getsockopt(fd(), IPPROTO_TCP, TCP_CONNECTION_INFO, &info, &info_size)
(node1@<IP_ADDRESS>)22> rp(inet:getopts(#Port<0.24>, [{raw, 6, 262, 1000}])).
{ok,[{raw,6,262,
          <<4,6,6,0,7,0,0,0,0,0,0,0,0,0,0,0,204,63,0,0,0,192,255,
            63,18,0,2,0,0,57,6,0,0,0,0,0,88,58,6,0,1,0,0,0,1,0,0,
            0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,177,1,0,0,0,0,0,0,
            0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,148,0,0,0,0,0,0,0,0,
            0,0,0,0,0,0,0,0,0,0,0,0,0,0,0>>}]}
ok

This is nice. If we can cleanly encapsulate it in mochiweb then it might be a viable option. I am planing to look into mochiweb code to see if it is doable.

For Linux it might be IPPROTO_TCP / TCP_INFO and I haven't checked exactly which bits conclusively show us the closed state yet.

I managed to find a way to seemingly detect the connection state change after the local client (curl) disconnects by querying the equivalent of getsockopt(fd(), IPPROTO_TCP, TCP_CONNECTION_INFO, &info, &info_size)
(node1@<IP_ADDRESS>)22> rp(inet:getopts(#Port<0.24>, [{raw, 6, 262, 1000}])).
{ok,[{raw,6,262,
          <<4,6,6,0,7,0,0,0,0,0,0,0,0,0,0,0,204,63,0,0,0,192,255,
            63,18,0,2,0,0,57,6,0,0,0,0,0,88,58,6,0,1,0,0,0,1,0,0,
            0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,177,1,0,0,0,0,0,0,
            0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,148,0,0,0,0,0,0,0,0,
            0,0,0,0,0,0,0,0,0,0,0,0,0,0,0>>}]}
ok

This is nice. If we can cleanly encapsulate it in mochiweb then it might be a viable option. I am planing to look into mochiweb code to see if it is doable.

I came to the same conclusion as Nick. It is quite a hack especially given the fact that it only would work on Linux. I also looked into using undocumented function unrecv. Which can be called in a quick succession after a recv.
(node1@<IP_ADDRESS>)49> f(Port), Port = hd(lists:reverse(erlang:ports())).
#Port<0.54>
(node1@<IP_ADDRESS>)51> f(Res), Res = inet_tcp:recv(Port, 1, 10).
{error,timeout}
(node1@<IP_ADDRESS>)52> inet_tcp:unrecv(Port, <<>>).
ok
(node1@<IP_ADDRESS>)53> f(Res), Res = inet_tcp:recv(Port, 1, 10).
{error,closed}
(node1@<IP_ADDRESS>)54> inet_tcp:unrecv(Port, <<>>).
ok

still_connected(Socket) ->
    // read one byte with 10ms timeout
    case recv(Socket, 1, 10) of
        {error, timeout} ->
            true;
        {ok, Data} ->
            gen_tcp:unrecv(Socket, Data),
            true;
        {error, closed} ->
            false
    end.

We could use timer:apply_after/4 to call still_connected/1. However the problem is that timer:apply_after would be called concurrently. Therefore there is a chance that the request handler process would call recv in between recv/unrecv. This means we need locking. We could implement locking using ets table, BUT.
The conclusion is it is not easy.

Interesting find @iilyak! One worry about  the gen_tcp:unrecv/2 function is that it may mess up the HTTP protocol stream in  case of any pipelining. We'd be reading data out of band and then putting it back into the erlang's TCP from a different process than mochiweb's request client.  At least with TCP_INFO at worse we'd get an unknown state or unsupported OS and we'd just do what we did before without the danger of messing up the HTTP TCP stream.

A bit more of a cleaned up connection script version which should work on macos, linux and freebsd:
#!/usr/bin/env escript

is_closed(Sock) ->
    OsType = os:type(),
    case tcp_info_opt(OsType) of
        {raw, _, _, _} = TcpInfoOpt ->
            case inet:getopts(Sock, [TcpInfoOpt]) of
                {ok, [{raw, _, _, <<State:8/native, _/binary>>}]} ->
                    tcp_fsm_state(State, OsType);
                {ok, []} ->
                    false;
                {error, einval} ->
                    % Already cleaned up
                    true;
                {error, _} ->
                    false
            end;
        undefined ->
            false
    end.

% All OS-es have the tcpi_state (uint8) as first member of tcp_info struct

tcp_info_opt({unix, linux}) ->
    %% include/netinet/in.h
    %%   IPPROTO_TCP = 6
    %%
    %% include/netinet/tcp.h
    %%   #define TCP_INFO 11
    %%
    {raw, 6, 11, 1};
tcp_info_opt({unix, darwin}) ->
    %% include/netinet/in.h
    %%   #define IPPROTO_TCP   6
    %%
    %% netinet/tcp.h
    %%   #define TCP_CONNECTION_INFO  0x106
    %%
    {raw, 6, 16#106, 1};
tcp_info_opt({unix, freebsd}) ->
    %% sys/netinet/in.h
    %%  #define  IPPROTO_TCP  6 /* tcp */
    %%
    %% sys/netinet/tcp.h
    %%  #define  TCP_INFO    32 /* retrieve tcp_info structure */
    %%
    {raw, 6, 32, 1};
tcp_info_opt({_, _}) ->
    undefined.

tcp_fsm_state(State, {unix, linux}) ->
    %% netinet/tcp.h
    %%   enum
    %%   {
    %%     TCP_ESTABLISHED = 1,
    %%     TCP_SYN_SENT,
    %%     TCP_SYN_RECV,
    %%     TCP_FIN_WAIT1,
    %%     TCP_FIN_WAIT2,
    %%     TCP_TIME_WAIT,
    %%     TCP_CLOSE,
    %%     TCP_CLOSE_WAIT,
    %%     TCP_LAST_ACK,
    %%     TCP_LISTEN,
    %%     TCP_CLOSING
    %%   }
    %%
    lists:member(State, [4, 5, 6, 7, 8, 9, 10]);
tcp_fsm_state(State, {unix, darwin}) ->
    %% netinet/tcp_fsm.h
    %%   #define TCPS_CLOSED             0       /* closed */
    %%   #define TCPS_LISTEN             1       /* listening for connection */
    %%   #define TCPS_SYN_SENT           2       /* active, have sent syn */
    %%   #define TCPS_SYN_RECEIVED       3       /* have send and received syn */
    %%   #define TCPS_ESTABLISHED        4       /* established */
    %%   #define TCPS_CLOSE_WAIT         5       /* rcvd fin, waiting for close */
    %%   #define TCPS_FIN_WAIT_1         6       /* have closed, sent fin */
    %%   #define TCPS_CLOSING            7       /* closed xchd FIN; await FIN ACK */
    %%   #define TCPS_LAST_ACK           8       /* had fin and close; await FIN ACK */
    %%   #define TCPS_FIN_WAIT_2         9       /* have closed, fin is acked */
    %%   #define TCPS_TIME_WAIT          10      /* in 2*msl quiet wait after close */
    %%
    lists:member(State, [0, 5, 6, 7, 8, 9, 10]);
tcp_fsm_state(State, {unix, freebsd}) ->
    %% netinet/tcp_fsm.h
    %%   #define TCPS_CLOSED 0 /* closed */
    %%   #define TCPS_LISTEN 1 /* listening for connection */
    %%   #define TCPS_SYN_SENT 2  /* active, have sent syn */
    %%   #define TCPS_SYN_RECEIVED 3 /* have sent and received syn */
    %%   #define TCPS_ESTABLISHED 4 /* established */
    %%   #define TCPS_CLOSE_WAIT 5 /* rcvd fin, waiting for close */
    %%   #define TCPS_FIN_WAIT_1 6 /* have closed, sent fin */
    %%   #define TCPS_CLOSING 7/* closed xchd FIN; await FIN ACK */
    %%   #define TCPS_LAST_ACK 8/* had fin and close; await FIN ACK */
    %%   #define TCPS_FIN_WAIT_2 9/* have closed, fin is acked */
    %%   #define TCPS_TIME_WAIT 10 /* in 2*msl quiet wait after close */
    %%
    lists:member(State, [0, 5, 6, 7, 8, 9, 10]).

monitor_loop(Sock) ->
    timer:sleep(1000),
    io:format("[mon] is_closed: ~p~n", [is_closed(Sock)]),
    monitor_loop(Sock).

client(Port) ->
    timer:sleep(1000),
    io:format("[client] connecting on ~p~n", [Port]),
    {ok, ClientSock} = gen_tcp:connect("<IP_ADDRESS>", Port, [{active, false}, binary]),
    io:format("[client] connected ~p sleeping for 5 seconds and exiting~n", [ClientSock]),
    timer:sleep(5000),
    io:format("[client] exiting ~n", []),
    ok.

main(_) ->
    {ok, ListenSocket} = gen_tcp:listen(0, [binary, {active, false}]),
    {ok, Port} = inet:port(ListenSocket),
    io:format("[srv] spawning client to connect to port ~p~n", [Port]),
    spawn_link(fun() -> client(Port) end),
    {ok, ClientSocket} = gen_tcp:accept(ListenSocket),
    io:format("[srv] accepted a connection ~p, sleeping for 10 seconds then exiting ~n", [ClientSocket]),
    spawn_link(fun() -> monitor_loop(ClientSocket) end),
    timer:sleep(10000).

./tcp_server_erlang.escript
[srv] spawning client to connect to port 59121
[client] connecting on 59121
[srv] accepted a connection #Port<0.8>, sleeping for 10 seconds then exiting
[client] connected #Port<0.7> sleeping for 5 seconds and exiting
[mon] is_closed: false
[mon] is_closed: false
[mon] is_closed: false
[mon] is_closed: false
[client] exiting
[mon] is_closed: true
[mon] is_closed: true
[mon] is_closed: true
[mon] is_closed: true
[mon] is_closed: true

updated again. I've expanded fabric_streams existing 'cleanup' process to be a watchdog (in the classic sense).
before starting a fabric_stream you must call fabric_stream:enable_watchdog(). If you do, the cleanup process starts a timer. If the watchdog is not kicked at least once (with fabric_stream:kick_watchdog().) before the timer fires, the watchdog kills the coordinator process (for us, the mochiweb/httpd process) and its workers.
by default this is not active, so all calls to fabric:all_docs etc are unaffected.
in mango_httpd I enable the watchdog and only when adding a row to the response do I kick the watchdog. If we go a long time between rows, the watchdog fires.
similar enable/kick can be done on other endpoints.

That's a decent solution. It would detect inactivity on the application side (workers are slow or no index defined or if anything else gets stuck and we don't emit rows). The timeout might be a bit low as there might be existing (even if perhaps pathological) queries which could emit rows every 61 seconds, or if we raise to 5 minutes, there might still be a some cases which we might still break. It's sort of a balancing act as if we make it too large, it might reduce the effect of this watchdog (requests might pile up).
I still think it might be better to be able to detect if the client socket has closed so I kept pressing with the tcp_info options check to see how many OS-es we it can cover and to make it a bit less ugly. So far I couldn't figure how to get windows to do it but got all the others I think. Here we can be more bold with a polling interval even at 5 or 10 seconds if needed as we can confidently clean if the client disconnects.
Another direction to take it is to emit newline (heartbeats) between json rows. I couldn't quite see where to do that without disturbing the views and all_docs callback mechanism too much. Maybe have a last timestamp on the cursor in https://github.com/apache/couchdb/blob/e60e27554708f46a9e6528fa6049d025c1aba859/src/mango/src/mango_cursor_view.erl#L487-L491 and we can issue a handle_doc(...) callback with a timeout and then in the mango_http we'd emit   a \n?  That would maintain the connection alive and quit it as soon as a send failure happens? I am not too familiar with mango so that may be completely off, too.

I would also prefer to detect the client disconnect directly but I am less keen to see the raw packet inspection stuff in couchdb in general, unless we can be confident that it causes no harm on any platform we haven't handled. That it wouldn't work on Windows isn't a deal breaker for me, we'd need to mention that in the release notes, and perhaps someone more familiar with Windows could add support.
As for heartbeats in the _find response, this obviously changes the response in a way that no client is currently expected. The idea that a blank line (or whitespace in general) will be silently handled is reasonable but requires a lot of testing to be confident.

Maybe this is also off, but let me add that mango_cursor_view implements its own keep-alive mechanism: https://github.com/apache/couchdb/blob/e60e27554708f46a9e6528fa6049d025c1aba859/src/mango/src/mango_cursor_view.erl#L455-L467
Does not it cause problems in this case?  Or perhaps this could be utilized to hook up a countermeasure action.

I think that's the opposite direction, the coordinator making sure the workers are still alive and exiting if not.
The problem we're trying to address is when the coordinator and the workers are all working but the client, who would receive the http response, is gone, and thus they should all exit.

To see how it would look tried a PR with the tcp_info getopts: https://github.com/apache/couchdb/pull/4736
I plugged it into the fabric_stream cleaners (and the db updater one) that part looks small enough. The ugliest thing is the socket options thing but it's confined to the util functions mostly.

closing in favour of approach in https://github.com/apache/couchdb/pull/4736


Choice of displaying the links below the embedded graph

User Story:
As an editor I would like to have the choice to embed my Visualization in a web page (iFrame or external application) without all the links below the chart and without links to the Metadata.
Information to be removed for the "Minimal View" (see also screen shot)

Dataset
Latest update
Download data
switch to table view
Learn more about the dataset
run SparQl query
Metadata Button + all Links to Metadata

The link "created with Visualize" must be kept, but the label has to be replaced with "more Information" or something like that.
I do not have a concrete idee for the labels in the box “Embed Code”. Maybe you could have : “Embed full view” vs “Embed minimal view”.
The current Embed codes are of course still available for editors who wish to implement the full view.

Nice! I especially like that, when you switch between the Standard & Minimal View, that you immediately see what it changes in the chart, due to the new placement of the Share and Embed Button. Nice improvement!


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v5.0 to before v5.16.5",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=2813b9c0296259fb11e75c839bab2d958ba4f96c",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=dd22253c13d2795c566f52c30f1dba48d51c482d"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "2813b9c0296259fb11e75c839bab2d958ba4f96c",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "dd22253c13d2795c566f52c30f1dba48d51c482d",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "mm, kasan: use compare-exchange operation to set KASAN page tag\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.16.5 by commit dd22253c13d2795c566f52c30f1dba48d51c482d, it was introduced in version v5.0 by commit 2813b9c0296259fb11e75c839bab2d958ba4f96c. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2022-1000617


Task 2.0.0 Feedback

We did a Task 2.0 deep dive today and this issue collects the combined feedback received.
Task Properties

echo should be true by default so that users get a better idea of the task executed
we should support flattening the JSON properties as we do with other settings (e.g "terminal.echo": true instead of "terminal": { "echo": true }
we should better separate task declaration from task customization. Currently both happens in the tasks array property and the differentiating element is the customize property.

Task Picker
The task picker should support a decription / detail message to better explain what the task does. We also discussed whether the task picker should support MRU which would automatically move the most used tasks up to the top without customization. If we provide a task panel (see below) such a MRU list could be useful there as well.
Task Customization
The task customization is currently instance based where the task instance is identified by a task id, which can be quite 'cryptic'. An alternative way would be to identify a task based on a type and a key that is defined by the type and therefore can vary from task runner to task runner. So instead of having something like:
	{
		"customize": "vscode.npm.run build",
		"taskName": "npm: run build"
	}

a customization would look like this
	{
		"customize": {
			"type": "npm",
			"file": "${workspaceRoot}/package.json",
			"script": "build"
		},
		"problemMatcher": ["$tsc"]
	}

where the structure of the identifying attribute is defined by the extension providing the runner support. In the example above the fact that a npm script is identified by the location of the package.json file and the script name comes from the npm extension. This might look different for task contributed by gulp of TypeScript
Problem Matchers
For task that execute a target of a task runner (e.g. gulp, npm, grunt) VS Code can't decide which problem matchers to attach. Since picking the right problem matcher currently requires editing the tasks.json file we should better support the user with this. We discussed the following ideas:

always attach all known problem matchers if a task has no problem matchers. This might result in false positives and might require us to support removing problem matchers via a configuration mechanism.
we could help the user attaching a problem matcher by generically scanning the output for problems and suggest problem matchers that exactly match these. If selected we update the tasks.json file for the user.

Task panel
It was suggested to run tasks in its own panel instead of reusing the terminal panel. They should still be executed using the terminal to get ANSI control character support and other nice terminal features (e.g. Ctrl+C). The panel could offer additional features like:

a task summary (execution time, exit code, ...)
the output of a task could be folded (like travis does for example)
allow for in place customization of a task (for example attach a problem matcher) without the user being forced to edit a tasks.json file.

We also discussed the need of a task viewlet but decided that this will not add any big additional value over the task picker and a separate panel.
Top Level Menu
We were also discussing whether the tasks deserve a top level menu. In general this was consider a good idea. However such a menu should be semantically not technical. So instead of naming it Tools or Tasks it would be better to offer top level menus like Build, Rebuild All, Clean, Run Tests. The downside of such an approach is that such a menu looks out of place if none of these entries trigger any meaningful action in the workspace (e.g. if a simple JS project is open at lease Build, Rebuild All and Clean don't make any sense)
Better Selfhosting
Would be great if we can improve the self hosting coverage of tasks. This is currently hard to do due to the fact how we build. But @dbaeumer will tweak our tasks.json to support more tasks currently run in the terminal (e.g run tests, linting, ...)

@Microsoft/vscode please read. Feedback and corrections welcome.

We also discussed the need of a task viewlet but decided that this will not add any big additional value over the task picker and a separate panel.

@Lixire and @aefernandes are currently looking into this I believe.

Most of this is In response to:

It was suggested to run tasks in its own panel instead of reusing the terminal panel.

First off, it definitely looks like these sorts of improvements will make tasks more useful and usable. I just wonder if this is the right direction we should be taking:

We already have output, debug console and terminal which are all very similar.
We don't deal with overflowed panel actions too well right now, this will make the panel less usable on small monitors and/or high scale/zoom settings. This is what the panel looks like right now no my 13" laptop screen. A new panel may start to squash the panel actions off screen with this window position:

Are tasks generally useful enough to warrant their own panel? Personally I've found little benefit over shell scripts, particularly if you want the project to be portable as tasks are only usable in VS Code.
Perhaps time invested in tasks would be better spent bringing some of these sorts of features to the terminal, namely https://github.com/Microsoft/vscode/issues/20676 + https://github.com/sourcelair/xterm.js/issues/576 which could lead to exit code marking, folding of terminal commands and clever problem matching (leveraging settings system for definitions).

Technical details
On the implementation side, this is definitely technically possible. It will put a permanent halt on moving tasks to an extension, I think that effort has fizzled out anyway though.
TerminalInstance is fairly generic right now and it wouldn't require that much tweaking to get working inside another panel. It doesn't have a dependency on TerminalService and communicates with it via events. The main problems you will run into are how terminal keybindings and the terminal context keys interact with the new tasks system:

terminalTextSelected is fairly easy, this could be changed to be if text is selected in tasks/terminal and that panel is active.

terminalFocus is a little trickier as making this apply to both task panel focus and terminal panel focus would cause issues with my keybindings, for example:
{ "key": "ctrl+`", "command": "workbench.action.terminal.focus",
                   "when": "!terminalFocus" }

I don't think too much of TerminalService can be reused if we make a tasks panel.
Some alternative ideas:

We implement terminal tabs and each task gets its own tab https://github.com/Microsoft/vscode/issues/10546
We embed TerminalInstances (ie. xterm.js) within the output panel, disabling stdin by default, then move tasks back to output?
Refocus effort on tasks on improving the terminal

See #28351

Added support for MRU list:

@Tyriar I don't think the idea of having a Task panel competes with your mentioned improvements for Terminal. If there is a UX path where Tasks and Terminal can fit in the same panel, that would also be great.
Some reasons why IMHO a Task panel would help:

It would be nice to see somewhere, in the form of a list, the current running tasks and some sort of history with past ran tasks. From the list, it would be nice to be able to expand output for a task in case there is a problem (sort of like travis).
Such a list would be especially helpful for managing background tasks
A panel might help with task discoverability. At this time, there is no obvious UI path to go launch a task or configure a task from a terminal. There is no mention of tasks or the possibility of gathering errors from a task in the UI.

And what would also be very nice:

It would be nice that typing in the terminal auto-completes with information from the task providers. e.g. typing "gulp c" should complete with the gulp tasks that start with letter c.
It would be nice that running a command in a terminal can be configured into a task via a one-click. e.g. it would be nice if, when someone runs gcc in the terminal (without a task configured) that we can detect that this would be a prime candidate for a task and we would guide the user into creating a task. I think many folks don't know that a command they run in the terminal can be "lifted" to be even more integrated into VS Code.

I would still want us to explore the possibility of a panel for tasks, I think UX scalability as reason to not do it is not very fair, we need to solve the panel overflow no matter what in the same way we solved it for the activity bar (being able to show/hide panels, reorder them, see a "..." overflow menu when space is too little).
I do agree though there is some overlap in Debug Console, Integrated Terminal, Output and Tasks, however I am not sure if it would make sense to put them all into one. There is very specific scenarios for each of the panels:

output: after tasks moved to terminal I would argue output is only ever being used as a way to see diagnostics (I wonder if we should make output less prominent)
debug console: has rich functionality when debugging, but otherwise is quite useless (maybe we should only show it when you are in a debug session?)
terminal: this is where I execute shell commands. nothing in here is related to VS Code, it is just convenient to not have to leave VS Code to use it
tasks: this is VS Code specific: I execute potentially multiple long running tasks, each with output and progress and possible error conditions (markers)

Here is a mockup for the Tasks panel:

We can provide an explicit configure action per task which would guide users to configure problem matchers.

@bpasero thanks for the nice mockup.

New ideas wrt task customisation and referencing (contributed) tasks by an id or something else? I think from an API POV that is a thing to tackle.

Not being an active task user myself, I wonder if the suggested Tasks panel could also show processes run from the Terminal. That could then open a path to offer problem matchers for tools run in the Terminal.

The mockup omits the problems panel. I wonder whether that could be rolled into the tasks panel to save space?

debug console: has rich functionality when debugging, but otherwise is quite useless (maybe we should only show it when you are in a debug session?)

Hiding it when a debug session hasn't been run would be useful. After a debug session completes, I still want to examine the contents of the console though.

As I'm not an active task user but I'm also thinking about why I don't use Task too often

If no one sets up all the tasks in a project, I don't think I'll ever know this feature from daily use. To new users, you don't have an idea when the Task will get in the way. A task viewlet or panel can help with discoverability in this case.
I didn't know where to launch/kill the task until one day I happen to type task in the command palette. Task 2.0 uses Terminal which helps with the problem a little as I can kill the task the same way as a shell cmd.
Task 2 now uses Terminal for emitting output, the first time experience was not intuitive as it took over the Terminal even though I was using it. Besides it shares the same persistence problem as Terminal, after reloading the window I no longer have access to background tasks.

Speaking of debug and output panel

I'm not a fan of auto show/hide unless we make it correct.
a. If we have a clean state of the workspace, launch debugger, the debug panel would show if there is no prelaunch task and no output.
b. If we have a prelaunch task, the terminal shows up first as it provides the task info the users but after the task is executed successfully, the debug panel doesn't show up.
c. I don't have a good reproduce of step a and b. I don't have a good expectation which will show up last.
d. Last time I get the same confusion is the competence of Output panel and Debug panel

Lastly, do you think it make sense to emit output directly terminal instance the same way as Task 2? The only thing you can do in the output panel is view/copypaste/search and all of them are doable in Terminal, the discoverability won't be worse than current Task 2.

@jrieken the initial description contains what we discussed in terms of how to reference a task (see Task Customization). I haven't started implementing anything. First need to see how feasible that is. Any comments to the proposal in terms of API feedback.

As a heavy user of the task/debug infrastructure within Visual Studio Code, I felt that it might be worth it to document my observations with the existing system.
My development environment is heavily focused on testing which has caused a large number of test executables to be created along side the actual application being developed. Additionally, all code compilation as well as test executable execution happens remotely (Virtual Studio Code running on OS X, where as compilation / test executable happen on a Linux machine).
I have found tasks to be very flexible and with the most recent additions to the Insiders Build, building and executing tests with a few keystrokes has really streamlined my development environment. I have been able to augment the tasks I execute today with a simple SSH script used to bridge the gap between Virtual Studio Code and the remote machine where all of the compilation happens.
That being said, one thing I have noticed is that I find myself duplicating task and launch configurations to handle the number of executables I deal with on a day-to-day basis. Having both my tasks.json and launch.json filled with upwards of 30+ configurations only differing by the executable is a little cumbersome. Additionally, the test executables themselves take arguments relating to specific filters of which subtests to actually execute (which is very helpful in executables which contain hundreds of tests). This means having to open up and edit my tasks.json any time I want to execute a smaller subset of a test executable.
While it is not clear to me what belongs in an extension versus what belongs in the core functionality of the editor, some things I would like to see are:

The ability to use 1 task configuration as a template, and feed in an array variations on that same template to help curb the duplication of task configurations.
The ability to specify a command (i.e. ${command:getDynamicArgument}) to be used to prompt a user for additional arguments on a task (which would help make it more fluid in scenarios where the arguments to an executable are constantly changing). Additionally a history of recent inputs would really improve the user experience of it.
Something similar to pipeTransport within the cpptools C++ extension which would allow the remote execution of a task.

I understand that some of these items are (and I have solved them via script development) achievable via external scripts, but I see these as problems that numerous developers across all walks of life would run in to and most likely solve in very similar ways.

We again discussed the tasks.json schema today and came to the following conclusion.

there shouldn't be a separate customization section. So everything goes below the tasks section
there should be a clear visual distinction between defined tasks and task customizations.

Below is a proposal how this will look in tasks.json
{
    "version": "2.0.0",
    "tasks": [
        // This is a defined tasks not contributed by an extension
        {
            "type": "raw",
            "name": "yeoman",
            "command": "yo",
            "runtime": "shell",
            "group": "build",
            "presentation": {
                "reveal": "always",
                "focus": false,
                "panel": "dedicated"
            }
        },
        // This is a customization of the npm build script
        {
            "type": "npm",
            "script": "build",
            "configure": {
                "problemMatcher": ["$tsc"]
            }
        },
        // This is a npm script task which the extension needs to
        // fully resolve. It will execute the osx and the osx-min
        // script
        {
            "type": "gulp",
            "tasks": ["osx", "osx-min"]
        }
    ]
}

This has a weird title. But here is my Issue with v.2.0.0:
If you run a background task, and close the VSCode, the background task is left running. This is quiet annoying.
Is this feedback issue meant for this kind of feedback? Thanks.

This is unrelated to my bug above, but I would like to see tasks having own tabs here:

If I could see e.g. "build" or "testwatch" as a tab there, it would be more accessible and easier to notice it's running those tasks. The dropdown is too subtle for me, but this is matter of taste.

@Ciantic terminal processes leaking tracked in https://github.com/Microsoft/vscode/issues/26807. See https://github.com/Microsoft/vscode/issues/28235#issuecomment-308374308 for proposal on a tasks tab.

Thoughts on what we (+ @Lixire) seem to expect from a task panel and current mockups. Feedback on brain dump and mockups welcome.

The panel should show a summary of each completed task, including execution time, exit code, and the foldable output if applicable. By default, we would show tasks in MRU order and prioritize tasks that are currently running. This would also include available/auto-detected tasks.
Allow user to start, stop, or (if already run) restart a task with one click.
I would expect the tasks panel to open from the newly-added Tasks menu and from the command palette. However, other panel items open from the View menu - we could include a Tasks option there as well.
We should provide guidance if tasks have not been configured and panel is open. Perhaps a link to the documentation, or a single button that configures .
Provide an explicit configure action for each task, and an action to configure problem matchers when viewing output.
A stretch goal could be to allow users to configure or make quick edits without ever touching JSON (about 40% of users interviewed indicated JSON configuration was a barrier to usage).
To support multi root workspaces, an option is to show a filepath next to each task name similar to the way debug works.

Example UI options:
A)
B)
C)

Thoughts on what we (+ @Lixire) seem to expect from a task panel and current mockups. Feedback on brain dump and mockups welcome.

The panel should show a summary of each completed task, including execution time, exit code, and the foldable output if applicable. By default, we would show tasks in MRU order and prioritize tasks that are currently running. This would also include available/auto-detected tasks.
Allow user to start, stop, or (if already run) restart a task with one click.
I would expect the tasks panel to open from the newly-added Tasks menu and from the command palette. However, other panel items open from the View menu - we could include a Tasks option there as well.
We should provide guidance if tasks have not been configured and panel is open. Perhaps a link to the documentation, or a single button that configures .
Provide an explicit configure action for each task, and an action to configure problem matchers when viewing output.
A stretch goal could be to allow users to configure or make quick edits without ever touching JSON (about 40% of users interviewed indicated JSON configuration was a barrier to usage).
To support multi root workspaces, an option is to show a filepath next to each task name similar to the way debug works.

Example UI options:
A) Task list with output opening in sidebar

B) Task list in MRU order

C) Task list with groupings

I'd like to point out that in my opinion it's a mistake to advertise this as default option for VSCode stable (1.14.0), if even the termination of the tasks does not work. https://github.com/Microsoft/vscode/issues/26807

Just to clarify things: the termination without leakage doesn't work for npm tasks. It does work for all others kind of tasks.

@dbaeumer I have a question/suggestion about this new Task Window...
When in Visual Studio, using the Task Runner Explorer, it is possible to click on one or more of these configurable buttons:

These then append additional arguments onto the task that is being executed.  Is it possible to do something similar here?  In the screenshot above, I am selecting to add the Verbose flag onto the task so that additional information is added to the output log.

It would be great if vscode would support pycharm style , task configure and task runner on the top right with run button,
Suppose user need to run multiple configuration file for same file , he can very well setup and need to go to  command pannel for task running ,ther is no one click solution

Love this idea! I also like the screenshot https://github.com/Microsoft/vscode/issues/28235#issuecomment-316376605. This reminds me also how JetBrains manage their panels, with clickable buttons etc.
You can see an idea of what I refer to here https://github.com/Microsoft/vscode-react-native/issues/481
So far VS Code shows text in the panels. Would it be possible to extend it with buttons, lists etc? That would make the panels more interactive.

@gep13 - that's more or less what I'd like to see.  Having Tasks show up in the Side Bar is where they belong in my opinion.  It elevates the Tasks concept to the proper level.  What should be displayed in the Output/Debug/Terminal area is task output from various task executions.

So I have been thinking about how I would like it to be. As I already mentioned I like the solution from JetBrains and from Visual Studio.
Here is a suggestion. What do you think? Of course there are things I haven't figured out yet, but it's a first draft suggestion at least.

@nonameolsson I very much like the Tasks panel in the Side Bar.  I'm not sure you really need the "RUNNING TASKS" view given that the Tasks panel in the Side Bar can indicate (via icon or text) which tasks are currently running.  And selecting a task from there could switch the tasks output view to the output of the selected task.  Everything else looks really good to me!

@rkeithhill That is a good suggestion! I like it, that way the UI will be a lot cleaner. Maybe a checkbox in the task panel to show only running tasks?

Update based on suggestion in https://github.com/Microsoft/vscode/issues/28235#issuecomment-319429369

I agree, moving Tasks to Activity Bar is the perfect place for it to be.
With that change, I don't think you need a separate Task Panel. The Tasks can remain in Terminal Panel and work on Tabs for Terminal #10546 and Split Terminal #7504 to gain the same benefits while extending it to popular use cases.

In the screenshot that I showed (https://github.com/Microsoft/vscode/issues/28235#issuecomment-316376605), the icons down the left hand side are not tasks, they are additional, optional, parameters that can be passed to the selected task.  I would still like to see this ability included as well.

Love @nonameolsson suggested UX
non ux suggession:
Adding peak memory usage for each task can be also added gives much controll over each task making the vscode to run smooth

@Jeyanthinath Thanks!
@dragonwolf83 Great suggestions regarding tabs for terminal and split terminal also! That would be awesome! I would love to be able to see multiple running tasks at the same time in a split terminal.
@gep13 Are those icons general, or for each task? In other words: do you choose verbose for a task before starting it, or does it affect all tasks?

@nonameolsson said...
Are those icons general, or for each task? In other words: do you choose verbose for a task before starting it, or does it affect all tasks?

Not in a position to test just now, but I believe that once selected those buttons affect each task that is ran.  i.e. you enable/disable them as required as you are executing tasks.

Any update on this? I really think that improvements to the Task section could make VS Code better.

No, our current focus it to support multi root for Tasks.

Any Updates on this ? I hope now multi-roots are into stable build .

What is the status of this?

@nonameolsson the technical features for task 2.0.0 got implemented. However no work has started yet to build a special task UI. However we added proposed API for extensions to query and execute tasks, so that task providers can add special UI (for example npm can contribute actions to the package.json file to directly execute scripts from the explorer).

@bpasero @nonameolsson borth your ideas seems really cool & mandatory to me.
I personnally need this feature, and Imo this is critical.
What i lack is mostly an overview panel of running / available tasks.
This list ui should stay visible when looking at a particular task log.
It should be like tabs/folds you can click to see each one details.
Terminal dropdown is not so good at it.

I like the @nonameolsson GUI, however I would add it that running tasks should be shown as tabs in the panel so that toggling between the outputs is faster.

My assumption is that there is very few running tasks and tabs are easy way to distinguish between the running task outputs.

I had a simple proposal for task discovery- show configured tasks under "Task" menu. #47172

I really would like to see some improvements to the task session. However, I'm really used to use the terminal for all that now. And being able to split into two terminals at the same time in VS Code has been a big help for me.
With the latest update there is now a NPM explorer, https://code.visualstudio.com/updates/v1_23#_npm-script-running.
I think it could be improved, but it is a really good start!

I will rename the issue since the major item left in the issue discussed is the task runner viewlet / UI.
Note that with the new API this might be implementable by an extension (see the npm script runner viewlet)

🙏 I'd love to see a sidebar like this in VS Code

please give us  viewlet


Data partition algorithm failling with parallel indexing tasks

Rally version: esrally 0.7.2
Invoked command: esrally --track=logs --offline --target-hosts=<IP_ADDRESS>:9200 --pipeline=benchmark-only --team-repository=private --challenge=index-logs
Configuration file (located in ~/.rally/rally.ini)):
[meta]
config.version = 11

[system]
env.name = lu_bench

[node]
root.dir = /home/fdeschamps/.rally/benchmarks

[runtime]
java.home = /usr/lib/jvm/jdk-8-oracle-x64

[benchmarks]
local.dataset.cache = ${node:root.dir}/data

[reporting]
datastore.type = elasticsearch
datastore.host = localhost
datastore.port = 9200
datastore.secure = False
datastore.user =
datastore.password =

[tracks]
default.url = https://github.com/elastic/rally-tracks

[teams]
default.url = https://github.com/elastic/rally-teams

[defaults]
preserve_benchmark_candidate = False

[distributions]
release.1.url = https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-{{VERSION}}.tar.gz
release.2.url = https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/{{VERSION}}/elasticsearch-{{VERSION}}.tar.gz
release.url = https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{{VERSION}}.tar.gz
release.cache = true

JVM version:
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

OS version: Ubuntu 16.04.2 LTS
Description of the problem including expected versus actual behavior:
When using several index operation in parallel, esrally fail to partition correctly the data between the clients, resulting in docs not being indexed for the 2nd and subsequent indices.
Steps to reproduce:

Use this track :

{
  "challenges": [
    {
      "name": "index-logs",
      "schedule": [
        {
          "parallel": {
            "tasks": [
              {
                "operation": "index_logs1"
              },
              {
                "operation": "index_logs2"
              }
            ]
          }
        }
      ],
      "description": "parallel indexing of logs",
      "default": true,
      "index-settings": {
        "index.number_of_replicas": 1
      }
    }
  ],
  "description": "This test indexes all logs from 2 sources in parallel",
  "operations": [
    {
      "name": "index_logs1",
      "bulk-size": 5000,
      "operation-type": "index",
      "index": "logs1"
    },
    {
      "name": "index_logs2",
      "bulk-size": 5000,
      "operation-type": "index",
      "index": "logs2"
    }
  ],
  "indices": [
    {
      "name": "logs1",
      "types": [
        {
          "name": "logs",
          "document-count": 303058,
          "documents": "logs1.json.bz2",
          "uncompressed-bytes": 242118291,
          "compressed-bytes": 5147740,
          "mapping": "mappings.json"
        }
      ]
    },
    {
      "name": "logs2",
      "types": [
        {
          "name": "logs",
          "document-count": 15483,
          "documents": "logs2.json.bz2",
          "uncompressed-bytes": 8046294,
          "compressed-bytes": 124503,
          "mapping": "mappings.json"
        }
      ]
    }
  ],
  "short-description": "Parallel logs indexing"
}

Launch the bench :
esrally --track=logs --offline --target-hosts=<IP_ADDRESS>:9200 --pipeline=benchmark-only --team-repository=private --challenge=index-logs

In output, we can see that rally is using a bad offset for index logs2:

2017-10-19 09:49:35,715 PID:6043 rally.track INFO Client [0] will index [303058] docs starting from line offset [0] for [logs1/logs]
2017-10-19 09:49:35,715 PID:6044 rally.track INFO Client [1] will index [15483] docs starting from line offset [15483] for [logs2/logs]

It should be starting at offset 0 for both indices.
From what i've seen in the code, the partition algorithm is using the client_id and total_clients to partition data. This doesn't work with parallel index tasks because it does not take into account that the data source is different for each task.

That's a valid point @fdeschamps. Without having looked into the code closely right now I am also pretty sure that this is not really handled so far.


Make custom expandable cell

How to make a custom expandable cell? I tried:
    let passwordRow = CustomRowFormer<PasswordCell>().onSelected {
        $0.cell.passwordTextFieldsView.isHidden = false
        $0.update()
    }.dynamicRowHeight { tableView, indexPath -> CGFloat in
        return 50
    }

but it's not working.
Maybe someone already did this.

Hello @Recouse, were you able to figure this out or did you still need help?

@ZacharyKhan thanks, I think I already figured it out)


lab04 finished

Finished lab04 - simple and advanced drawings

@rammalyala Your feedback is updated. Note - set your own title in PictureViewer; it is still "YOUR NAME HERE's Drawing".


"failed to extract output name from latex log" with long filenames

When a filename is long enough for TeX to wrap its message:
Output written on "latex.out/L04 Intel 64 Architektur und C-Toolchain.pdf" (1 p
age, 74341 bytes).
Transcript written on "latex.out/L04 Intel 64 Architektur und C-Toolchain.log".

This causes latexrun to abort with:
failed to extract output name from latex log
There were errors; output not updated

This is on Archlinux with latexrun on 38ff6ec and pdfTeX 3.14159265-2.6-1.40.21 (TeX Live 2020/Arch Linux).

Ah, looks like there's #19 already - sorry, I didn't look at closed issues.


Dragonfly fixes 2016-12

This fixes libc again for DragonFly.

r? @alexcrichton
(rust_highfive has picked a reviewer for you, use r? to override)

Looks good to me but I think CI is failing?

now CI is fine!

@bors: r+

:pushpin: Commit 51cc3b7 has been approved by alexcrichton

:hourglass: Testing commit 51cc3b7 with merge 411cafc...

:sunny: Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 411cafc47c3ca1e7f89c3891ec76308918e33f00 to master...


Use ReadOnlySpan where known from const pointer in header file

This addresses a suggestion from #14. I have gone through header files to find pointer arguments marked const X* and then marked them as [In] in our P/Invoke definitions. [In] doesn’t have any effect on pointers (can be used for array marshalling for example) but its semantics fit this use case. (Alternatives: invent our own attribute, use special parameter names)
The code generation was changed to check for the attribute when generating span wrapper. [In] arguments will now use ReadOnlySpan instead of Span. The attribute is stripped from the generated wrappers.
(Some small non-functional improvements on the code generator were performed in the first commit as well)

Many thanks

@AnthonyLloyd Do you have any plans for publishing a new release on NuGet?

I can publish one in the next couple of days. I will also get round to publishing new versions of the native packages.

I can publish one in the next couple of days. I will also get round to publishing new versions of the native packages.

That’d be much appreciated, thank you 👍
By they way I noticed that Intel changed the naming of the native libraries (adding a version number in the file name). I think your package renames them (might be wrong there). For people who install the native libraries themselves it’d be good to know so they can create a symlink.


add job history to judge application status/2625

What is the purpose of the pull request
add job history status url to judge application status, when application number threshold is reached(default 10000,maybe it was set to 1000).
you can see issue #2625

Codecov Report

Merging #2848 into dev will decrease coverage by 0.09%.
The diff coverage is 45.45%.

@@             Coverage Diff              @@
##                dev    #2848      +/-   ##
============================================
- Coverage     37.22%   37.13%   -0.10%
+ Complexity     2557     2553       -4
============================================
  Files           434      434
  Lines         19992    20004      +12
  Branches       2425     2426       +1
============================================
- Hits           7443     7429      -14
- Misses        11880    11911      +31
+ Partials        669      664       -5

Impacted Files
Coverage Δ
Complexity Δ

.../org/apache/dolphinscheduler/common/Constants.java
83.33% <ø> (ø)
1.00 <0.00> (ø)

...che/dolphinscheduler/common/utils/HadoopUtils.java
34.67% <45.45%> (+2.05%)
21.00 <1.00> (+1.00)

.../server/worker/processor/TaskExecuteProcessor.java
10.16% <0.00%> (-27.12%)
1.00% <0.00%> (-2.00%)

...he/dolphinscheduler/api/service/LoggerService.java
82.60% <0.00%> (-8.70%)
9.00% <0.00%> (-1.00%)

...nscheduler/server/entity/TaskExecutionContext.java
81.18% <0.00%> (-5.95%)
57.00% <0.00%> (-3.00%)

...inscheduler/common/task/sqoop/SqoopParameters.java
80.00% <0.00%> (ø)
15.00% <0.00%> (ø%)

...he/dolphinscheduler/common/thread/ThreadUtils.java
77.77% <0.00%> (+3.17%)
15.00% <0.00%> (+1.00%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c15345...2d7aafc. Read the comment docs.

this is not my change file :)

could you submit this PR to dev-1.3.0,  we will fix bug on the dev-1.3.0 first
when published 1.3.0, will have more energy for reviewing the PR on the dev

could you submit a same PR to dev-1.3.0 which is used for publishing 1.3.0 version

ok, but i can't pull/push with github at home because of network. I will finish it next Monday.


InvalidDataException on Debian 6.0 but not on Windows 10

Heya,
I use a package, CoreRCON, which allows me to connect to Source Game Servers to remotely send console commands. On start, my app creates the connections to the servers, and stores them for later. The problem is, one of the rcon server connection throws a System.IO.InvalidDataException on my Debian server, but testing on my Windows PC, no error is thrown. Trying to catch the exception in a try-catch statement doesn't work.
Exception: https://hastebin.com/eyibotehup.css
Server: Linux botsrv 4.5.7-std-3 #1 SMP Tue Jul 12 09:56:30 UTC 2016 x86_64 GNU/Linux
Result on Server:

Result on PC:

I think this is a bug in CoreRCON. It assumes that its TCPPacketReceived method will be called for every "packet". But TCP is a stream-based protocol, it does not really have packets (or rather, it uses packets only as an implementation detail), you can't assume that the buffer you receive contains exactly the information you want, not more or less, as is explained in the documentation:

For byte stream-style sockets, incoming data is placed into the buffer until the buffer is filled, the connection is closed, or the internally buffered data is exhausted.

If it seems to work reliably on Windows, then that's probably just luck.
And CoreRCON is also the reason why the exception is uncatchable: it's thrown from TCPPacketReceived, which is the event handler for SocketAsyncEventArgs.Completed. As far as I know, if the handler throws an exception (and doesn't handle it itself), you can't catch it.

Sounds like @svick has helped conclude there is nothing here to fix in .NET Core. If it gets root caused as an issue in the platform, let us know. I'm going to close this for now.


Client wont recieve broadcasts

I have one instance running on my win 10 pc and another instance (client) running on my linux VM. But the VM won't receive anything... Any ideas?

Changeing  to XXX.XXX.XXX.255 fixed it! So for example:
<IP_ADDRESS>


[e2e test]. Need to automate creating "non-default" namespace

Today, to run e2e test,  user requires to create a "non-default" namespace in the cluster.  This needs to be automated.

The e2e tests have been updated to only create objects in the non-default namespace (referencing a common constant). I think it might be better to create a new namespace for each test run and delete it afterward.


Removal of convert_to_ndarray

This PR removes all usage of the utils function convert_to_ndarray, which is entirely redundant with the Numpy function atleast_1d.

This PR resolves #583.


Fix undefined ref to hipUbindTexture for texture types

github issue: #1506
When call function hipUnbindTexture, hcc complain about
/tmp/tex3d-f8e3d0.o: In function hipError_t hipUnbindTexture<float, 3, (hipTextureReadMode)0>(texture<float, 3, (hipTextureReadMode)0>&)': /usr/local/include/hip/hcc_detail/hip_runtime_api.h

Why was this issue not seen in HIP CI ? https://github.com/ROCm-Developer-Tools/HIP/blob/master/tests/src/texture/hipBindTexRef1DFetch.cpp#L95 uses hipUnbindTexture.

Why was this issue not seen in HIP CI ? https://github.com/ROCm-Developer-Tools/HIP/blob/master/tests/src/texture/hipBindTexRef1DFetch.cpp#L95 uses hipUnbindTexture.

There are 2 texture unbind API's:
In hipBindTexRef1DFetch.cpp it uses hipUnbindTexture(const textureReference* tex) API.


0.8 migration - Icons

Add an icon resource file as soon as this functionality becomes available for custom nodes...

In progress, but:
https://github.com/DynamoDS/Dynamo/issues/5071

done


Fujitsu FR60/80 Support

Summary
I developed a FR 60 Plugin and I would like to upstream this work. I am unsure if the maintainers of this project are interested in accepting this functionality to mainline. If there is interest, would it be for generalizable functionality of the architecture or the core specific features as well.
I am also wondering what the best avenue is to ask questions regarding this work if I have any (github issues perhaps?).
Details
I am working on analyzing the DVR processor in the Sony PSX which uses the Fujitsu MB91302A processor.
To aid with this I've built a Ghidra plugin which enables decompilation of the firmware and automatically setting up memory space and identifying & labeling registers and Fujitsu uTron kernel system calls.
This plugin is available on my github here: https://github.com/desrdev/ghidra-fr60
This was developed with a single goal in mind, and after completing the work I spotted the following projects targeting similar goals and architecture:
https://github.com/simeonpilgrim/ghidra_fujitsu_fr
https://github.com/ammaraskar/ghidra-fujitsu-risc
With the exception of ghidra-fujitsu-risc, the projects are released with compatible licenses to Ghidra. Both projects are targeting a single processor on a specific kind of device however and are not truly general plugins for the Fujitsu 32bit processors.
From a quick glance, all three of our implementations are fairly close to each other which suggests the CPU instruction architecture can be generalized across all 32bit Fujitsu processors.
Fujitsu no longer manufactures or advertises these components, however Infineon/Cypress do market these cores and they are still in active use today.
Given that the core is actively used and there are now 3 separate targets repeating work, I feel there is a case to be made to upstream some part of these projects to Ghidra mainline to allow reuse of the generalizable aspects of all three of these projects.
I am open to doing this work and maintaining this new feature, but I am unfamiliar with the process, style guide, and best practices when designing Ghidra cores or contributing to the Ghidra project in general.
Even in my own plugin there are features and functions I wish to add, but I am not sure of the idiomatic way to approach them.
To reiterate, I am willing to do the work required to get this mainline ready, and the plugin I developed is fully functional and I use it almost daily, but I will likely need some assistance to do so and I understand if the maintainers do not have the bandwidth/interest to take this on.
Thank you for your time.

Should you choose to submit a PR, a processor specification used in a wide number of devices would be acceptable.  We can't make a judgement call on whether the work is worthwhile for you to undertake.  The ghidra community will try to help/answer questions as bandwidth allows should you need guidance.  Processors in general are usually fairly contained and don't affect the rest of Ghidra, so they are easier to review and accept.
Also acceptable would be scripts/extensions/loaders/analyzers that are generic, such as formats (ie: elf), or a larger general development framework that have a wide usage across general software (ie: gcc).  In general, we can't accept any scripts/code that specifically targets a vendors platform or software, especially game systems such as the PSX.
We would be open to including the processor in the baseline distribution.  It should be generalized, as you mention, to target the base instruction set that is common across the variants for the processor family.   Registers for a very specific device variant may not be acceptable depending on their generality across other devices/variants.
The processor would need to go through a code review before inclusion in the baseline.  It is highly recommended to implement the automated PcodeTest for the processor if there is an available C compiler.  It helps both you the developer and reviewers to know that the pcode semantics are sound.  I would endeavor to create a sample binary that is from open source code such as our pcode unit test code so the bytes are easily shareable/reproducible.
If you plan to submit a PR, please submit one for only the processor related artifacts.
If you think there are general plugin/loaders/scripts that we could accept as well, submit that as a separate PR.  I'd keep each PR very targeted to a specific topic for example one just for an ELF loader modification to handle the FR60.  The more focused/self contained a PR, the easier it is for others to consider and comment on.
We can't promise any level of responsiveness, as there are definitely other processors or issues that are of higher priority.  However getting the breadth of processors available for someone who might need it in the future is a worthy goal. You can look at other processors for general style, or to see how similar processor features are implemented (ie: zero overhead loops).  Any questions should be general to Ghidra or reverse engineering with ghidra related, and not targeted at any particular device or binary unless legally available to the public such as a binary compiled from open source.
Processors can take a while to develop and review by the Ghidra community, as you can see from the backlog.  Each one has some unique feature that can stress sleigh processor modeling.  However, we appreciate the effort!

Hey @emteere.
Thank you for taking the time out of your day for this excellent write up.
I think I understand what would be compatible with the mainline project goals and will look towards developing a PR. I don't have any expectations or needs for this from a timeline/responsiveness perspective, so I completely understand that.
This is also the first I heard of PcodeTest and the pcode unit tests, these will be very valuable even if it's just to polish my plugin.
Regarding the test suite, would it be a show stopper if the suite is created from open source code, but the compiler/assembler is proprietary? GCC does have a FR30 target, but most embedded systems will use the SoftTune assembler/C/C++ compiler. I haven't taken a look to see if the ABI is similar between the two compilers yet.

It would be better if anyone could re-create the tests as needed, or if the pcode test code changes.
However if this isn't possible, then creating the test binaries is good enough, and way better than not creating them.
They could be done with both, each compiler will choose different instructions to implement the same code, so you would get more coverage.  The idea behind the pcode tests is to test the instructions you will normally find in compiled code.  It won't get everything, but gets most of what you see and gives you a good idea if the general addressing modes are correct.
That said they can be finicky on small microprocessors (8/16-bit) unless the compiler is good.
This processor, if it is 32-bit should do OK.
We've thought about how to make them smaller, and even have a simpler way to run them / compile.
For example, just being able to drop a .o into the directory might be enough, if the code/assembly routine to run and test could be found in debug information, kind of like introspection in JUNIT.
It would allow tailored tests that weren't even from the base pcode unit test code, or augment the pcode unit tests with hand coded assembly tests.
If the ABI's are different, it shouldn't matter for the pcode tests.  You can have multiple .cspec files for each compiler if they vary.

Do you have any additional questions, or can I close this issue?

Hey @emteere. No further questions. Thank you for the assistance. I managed to build the test cases using gcc, the architecture is simple enough that the ABI is identical between that and the proprietary Fujitsu compiler.
The test suite identified a lot of subtle errors and I'm now at 98% test coverage. Thank you very much for pointing me in this direction as it greatly improved the disassembly.
Looking forward to closing out that last 2% and preparing a PR sometime in the future. I will close this ticket 😄


Service account documentation improvement

Fixes #9344.
Depends on #9821.
Update Secrets documentation to explain how secrets can be
created/used manually, or automatically with service accounts.
Greatly expanded service account documentation.
Added a service account admin guide.
Lots of cross-references.

@liggitt @pmorie fyi

@erictune Some comments

LGTM

thx.


OpenOCD not working for windows

Hi .. I am using Zephyr RTOS to build a firmware for Raspberry Pi Pico and using RaspberryPi Debug Probe for debug purpose. Although the build is happening succesfully, I am facing issues with the flashing.
Things I tried:

I followed the instructions mentioned here. Installed the OpenOCD using MSYS2 but I am getting this error.

C:\Users\protocentral\protocentral_healthypi5_zephyr>west flash                                                               -- west flash: rebuilding                                                                                [1/10] Performing build step for 'second_stage_bootloader'                                                              ninja: no work to do.                                                                                [3/3] Completed 'second_stage_bootloader'                                                                                -- west flash: using runner openocd                                                                                FATAL ERROR: required program C:/Program Files/OpenOCD/bin/openocd.exe not found; install it or add its location to PATH

I installed OpenOCD using the instructions given here. But I am getting this error

C:\Users\protocentral\protocentral_healthypi5_zephyr>west flash -- west flash: rebuilding [1/10] Performing build step for 'second_stage_bootloader' ninja: no work to do. [3/3] Completed 'second_stage_bootloader' -- west flash: using runner openocd -- runners.openocd: Flashing file: C:/Users/protocentral/protocentral_healthypi5_zephyr/build/zephyr/zephyr.hex Open On-Chip Debugger 0.12.0 (2023-10-02) [https://github.com/sysprogs/openocd] Licensed under GNU GPL v2 libusb1 09e75e98b4d9ea7909e8837b7a3f00dda4589dc3 For bug reports, read         http://openocd.org/doc/doxygen/bugs.html set_adapter_speed_if_not_set Can't find interface/cmsis-dap.cfg FATAL ERROR: command exited with status 1: 'C:/Program Files/OpenOCD/bin/openocd.exe' -s 'C:/Users/protocentral/protocentral_healthypi5_zephyr/boards/arm/healthypi5_rp2040\support' -s C:/Users/protocentral/zephyr-sdk-0.16.3/sysroots/AMD64-pokysdk-linux/usr/share/openocd/scripts -f 'C:/Users/protocentral/protocentral_healthypi5_zephyr/boards/arm/healthypi5_rp2040\support\openocd.cfg' -c 'source [find interface/cmsis-dap.cfg]' -c 'transport select swd' -c 'source [find target/rp2040.cfg]' -c 'set_adapter_speed_if_not_set 20000' '-c init' '-c targets' -c 'reset init' -c 'flash write_image erase C:/Users/protocentral/protocentral_healthypi5_zephyr/build/zephyr/zephyr.hex' -c 'reset run' -c shutdown
what is it that I am doing wrong?
`

i followed the instructions given here and I was able to proceed with the flashing of the firmware.

i followed the instructions given here and I was able to proceed with the flashing of the firmware.

@MishalJasmine So can this issue be closed?

Yes the issue can be closed.


Links with an ampersand (&) can not added (via docker-compose)

I noticed that links with an ampersand in it can not be added to archivebox:
❯ docker-compose run archivebox add --extract singlefile "https://grml.org/&"

❯

Escaping & makes it work:
❯ docker-compose run archivebox add --extract singlefile "https://grml.org/\&"
[i] [2023-07-24 10:36:01] ArchiveBox v0.6.2: archivebox add --extract singlefile https://grml.org/&grml
    > /data

[+] [2023-07-24 10:36:02] Adding 1 links to index (crawl depth=0)...
    > Saved verbatim input to sources/1690194962-import.txt
    > Parsed 1 URLs from input (Generic TXT)
    > Found 1 new URLs not already in index

[*] [2023-07-24 10:36:02] Writing 1 links to main index...
    √ ./index.sqlite3

[▶] [2023-07-24 10:36:02] Starting archiving of 1 snapshots in index...

[+] [2023-07-24 10:36:02] "grml.org/&"
    https://grml.org/&
    > ./archive/1690194962.156108
      > singlefile
        3 files (240.7 KB) in 0:00:05s

[√] [2023-07-24 10:36:08] Update of 1 pages complete (5.99 sec)
    - 0 links skipped
    - 1 links updated
    - 0 links had errors

    Hint: To manage your archive in a Web UI, run:
        archivebox server <IP_ADDRESS>:8000

~/software/docker/archivebox took 10s
❯

If've made some tests:
❯ cat test.sh
#!/bin/bash
bash -c "echo $*"

❯ ./test.sh "http://grml.org/?&" "https://grml.org?&grml"
http://grml.org/?
bash: line 1: https://grml.org?: No such file or directory
bash: line 1: grml: command not found

❯ cat test.sh
#!/bin/bash
bash -c "echo '$*'"

❯ ./test.sh "http://grml.org/?&" "https://grml.org?&grml"
http://grml.org/?& https://grml.org?&grml

Maybe the following line in docker_entrypoint.sh should be changed:
https://github.com/ArchiveBox/ArchiveBox/blob/40ddd3360207aefd0e2c168833c72a6868c9c80b/bin/docker_entrypoint.sh#L44
to:
exec gosu "$ARCHIVEBOX_USER" bash -c "archivebox '$*'"

Thanks for this awesome software.

Oh, I just noticed that the given URL is listed as 'Pending':

Selecting the item and clicking "Title ⬇" updates the title. And yes, https://grml.org/&grml does not exist and returns a "404 not found". Its just an example.

This should be working as of v0.7.2 (I changed the docker_entrypoint.sh to requote shell arguments properly).


Bug fix/rlp 1097/batching get logs

What was wrong?
RSK doesn't allow eth_getLogs calls with a big block size number. The fromBlock and toBlock params have to be near because the RSKj node doesn't suppport big queries.
How was it fixed?
To fix this we added support to make small batch queries instead of making a huge one. Also we added some cache to store the logs to avoid having to query them all the time.

Closing due to #4


Training for Indian Multi-Speaker/Multi-lingual VC

Hi, @Wendison Thank you so much for your excellent work. very nice paper.
When I saw this reply on the below issues, it helped me to motivate to go further.
https://github.com/Wendison/VQMIVC/issues/14#issuecomment-937900528
https://github.com/Wendison/VQMIVC/issues/17#issuecomment-971136691
I am trying Common Voice Indian English Multi-Speakers and VCTK Training. I need a few suggestions from you
Steps:

I add Common Voice Indian English Multi-Speakers (40 speakers - each having 30 minutes Datasets) along with VCTK 109 Speakers. and start training  use_CSMI=True use_CPMI=True use_PSMI=True

After the model is trained with good accuracy, will go for fine-tuning with other Indian regional languages of Common Voice (Tamil, Hindi, Urdu, etc)

is this approach good?
@Wendison kindly request, please give your suggestions. Thanks

Hello @MuruganR96 ,
Have you been successful training for Multi lingual VC?