Loosing messages when producing

Hi @emmett9001 ,
We have a problematic thing going on with producing messages. We have an application running with a number of consumers and two producers. Consuming occurs continuously, but producing is just every 30 minutes (due to batch processing of input file). When starting the application and running the first batch, everything works fine. Which means every message that is produced, is available on the topic and can be consumed by the consuming application. But after 30 minutes when the 2nd batch is processed, we are getting "Error encountered when producing to broker x. Retrying." errors in the debuglog of pykafka and we are loosing some messages. With other words we send for example 100 messages, but always a few are gone (1 or 2). So they never arrive on the topic and are thus not consumed by the consuming application. This happens on both producers, each producing to a separate topic. We are running the producer in asynchronous mode (sync=False on get_producer).
When running in synchronous mode (sync=True), same error appears in the debuglog, but none of the messages are gone. So 100% of the messages produced appear on the topic.
We are running pykafka 2.8.0, but also tried 2.7.0. We have a Kafka cluster setup of 3 nodes, with replication-factor 3 for all the topics. Kafka version is 1.1.0.
Underneath is a part of the debuglog of pykafka where you can see the errors I mentioned.
Looking at the pykafka code, I think the "Error encountered when producing to broker x. Retrying." errors is because of a SocketDisconnectedError. I guess that in the 30 minutes between producing, the connection is lost somehow due to timeouts. It successfully reconnects in both sync and async mode, but unfortunately messages are lost in async mode.
Any thoughts on this?
Thank you very much.

2019-01-25 12:11:00,622 Sending 1 messages to broker 2
2019-01-25 12:11:00,622 Sending 1 messages to broker 0
2019-01-25 12:11:00,632 Error encountered when producing to broker ip:9092. Retrying.
2019-01-25 12:11:00,633 Updating cluster, attempt 1/3
2019-01-25 12:11:00,634 Error encountered when producing to broker ip:9292. Retrying.
2019-01-25 12:11:00,634 Updating cluster, attempt 1/3
2019-01-25 12:11:00,635 Connecting to ip:9092
2019-01-25 12:11:00,635 Connecting to ip:9092
2019-01-25 12:11:00,636 Successfully connected to ip:9092
2019-01-25 12:11:00,636 Successfully connected to ip:9092
2019-01-25 12:11:00,640 RequestHandler.stop: about to flush requests queue
2019-01-25 12:11:00,641 Discovered 3 brokers
2019-01-25 12:11:00,641 Reconnecting to broker id 0: ip:9092
2019-01-25 12:11:00,643 Connecting to ip:9092
2019-01-25 12:11:00,644 Successfully connected to ip:9092
2019-01-25 12:11:00,644 RequestHandler.stop: about to flush requests queue
2019-01-25 12:11:00,645 RequestHandler.stop: about to flush requests queue
2019-01-25 12:11:00,647 Discovered 3 brokers
2019-01-25 12:11:00,647 Broker ip:9092 metadata unchanged. Continuing.
2019-01-25 12:11:00,647 Broker ip:9192 metadata unchanged. Continuing.
2019-01-25 12:11:00,648 Reconnecting to broker id 2: ip:9292
2019-01-25 12:11:00,649 Connecting to ip:9292
2019-01-25 12:11:00,649 Broker ip:9192 metadata unchanged. Continuing.
2019-01-25 12:11:00,649 Broker ip:9292 metadata unchanged. Continuing.
2019-01-25 12:11:00,650 Discovered 16 topics
2019-01-25 12:11:00,650 Adding 1 partitions
2019-01-25 12:11:00,650 Starting new produce worker for broker 0
2019-01-25 12:11:00,650 Successfully connected to ip:9292
2019-01-25 12:11:00,650 RequestHandler.stop: about to flush requests queue
2019-01-25 12:11:00,652 Discovered 16 topics
2019-01-25 12:11:00,652 Adding 1 partitions
2019-01-25 12:11:00,652 Starting new produce worker for broker 2
2019-01-25 12:11:00,656 Successfully sent 0/1 messages to broker 0
2019-01-25 12:11:00,656 Successfully sent 0/1 messages to broker 2
2019-01-25 12:11:00,663 Sending 1 messages to broker 0
2019-01-25 12:11:00,663 Sending 1 messages to broker 2
2019-01-25 12:11:00,668 Fetched 1 messages for partition 0
2019-01-25 12:11:00,668 Partition 0 queue holds 1 messages
2019-01-25 12:11:00,669 Successfully sent 1/1 messages to broker 2
2019-01-25 12:11:00,672 Successfully sent 1/1 messages to broker 0
2019-01-25 12:11:00,675 Fetched 1 messages for partition 0
2019-01-25 12:11:00,675 Partition 0 queue holds 1 messages
2019-01-25 12:11:00,680 Sending 1 messages to broker 2
2019-01-25 12:11:00,681 Sending 1 messages to broker 0
2019-01-25 12:11:00,687 Successfully sent 1/1 messages to broker 2
2019-01-25 12:11:00,687 Fetched 1 messages for partition 0
2019-01-25 12:11:00,687 Partition 0 queue holds 1 messages
2019-01-25 12:11:00,688 Successfully sent 1/1 messages to broker 0
2019-01-25 12:11:00,692 Sending 1 messages to broker 0
2019-01-25 12:11:00,692 Sending 1 messages to broker 2
2019-01-25 12:11:00,693 Sending 1 messages to broker 2
2019-01-25 12:11:00,694 Fetched 1 messages for partition 0
2019-01-25 12:11:00,695 Partition 0 queue holds 1 messages
2019-01-25 12:11:00,697 Successfully sent 1/1 messages to broker 2
2019-01-25 12:11:00,697 Successfully sent 1/1 messages to broker 0
2019-01-25 12:11:00,698 Sending 1 messages to broker 2
2019-01-25 12:11:00,698 Sending 1 messages to broker 0
2019-01-25 12:11:00,701 Successfully sent 1/1 messages to broker 2
2019-01-25 12:11:00,702 Successfully sent 1/1 messages to broker 0
2019-01-25 12:11:00,704 Fetched 2 messages for partition 0
2019-01-25 12:11:00,704 Partition 0 queue holds 2 messages
2019-01-25 12:11:00,705 Sending 1 messages to broker 2
2019-01-25 12:11:00,706 Sending 1 messages to broker 0
2019-01-25 12:11:00,710 Successfully sent 1/1 messages to broker 2
2019-01-25 12:11:00,710 Sending 1 messages to broker 2
2019-01-25 12:11:00,711 Successfully sent 1/1 messages to broker 0
2019-01-25 12:11:00,711 Sending 1 messages to broker 0
2019-01-25 12:11:00,713 Successfully sent 1/1 messages to broker 2
2019-01-25 12:11:00,714 Fetched 3 messages for partition 0
2019-01-25 12:11:00,714 Partition 0 queue holds 3 messages
2019-01-25 12:11:00,714 Successfully sent 1/1 messages to broker 0
2019-01-25 12:11:00,717 Sending 1 messages to broker 0
2019-01-25 12:11:00,718 Sending 1 messages to broker 2
2019-01-25 12:11:00,720 Error encountered when producing to broker ip:9292. Retrying.
2019-01-25 12:11:00,720 Updating cluster, attempt 1/3
2019-01-25 12:11:00,722 Successfully sent 1/1 messages to broker 2
2019-01-25 12:11:00,722 Connecting to ip:9092
2019-01-25 12:11:00,722 Successfully sent 1/1 messages to broker 0
2019-01-25 12:11:00,722 Successfully connected to ip:9092
2019-01-25 12:11:00,726 RequestHandler.stop: about to flush requests queue
2019-01-25 12:11:00,728 Fetched 3 messages for partition 0
2019-01-25 12:11:00,728 Partition 0 queue holds 3 messages
2019-01-25 12:11:00,728 Discovered 3 brokers
2019-01-25 12:11:00,728 Broker ip:9092 metadata unchanged. Continuing.
2019-01-25 12:11:00,728 Broker ip:9192 metadata unchanged. Continuing.
2019-01-25 12:11:00,728 Reconnecting to broker id 2: ip:9292
2019-01-25 12:11:00,729 Connecting to ip:9292
2019-01-25 12:11:00,730 Successfully connected to ip:9292
2019-01-25 12:11:00,730 RequestHandler.stop: about to flush requests queue
2019-01-25 12:11:00,731 Discovered 16 topics
2019-01-25 12:11:00,731 Adding 1 partitions
2019-01-25 12:11:00,731 Starting new produce worker for broker 2
2019-01-25 12:11:00,731 Fetched 2 messages for partition 0
2019-01-25 12:11:00,731 Partition 0 queue holds 2 messages
2019-01-25 12:11:00,731 Successfully sent 0/1 messages to broker 2
2019-01-25 12:11:00,740 Sending 1 messages to broker 2
2019-01-25 12:11:00,740 Sending 1 messages to broker 0
2019-01-25 12:11:00,741 Sending 1 messages to broker 2

Hi @emmett9001
Thanks you for thinking along! I tried the required_acks=2 but unfortunately I'm getting a direct error on producer.produce:
Traceback (most recent call last):
File "/home/xxx/Test/Kafka/pure_producer.py", line 103, in
p.start(json_data)
File "/home/xxx/Test/Kafka/pure_producer.py", line 40, in start
self._producer.produce(simplejson.dumps(json_data))
File "/usr/local/lib/python2.7/dist-packages/pykafka/producer.py", line 255, in exit
self.stop()
File "/usr/local/lib/python2.7/dist-packages/pykafka/producer.py", line 331, in stop
stop_owned_brokers()
File "/usr/local/lib/python2.7/dist-packages/pykafka/producer.py", line 324, in stop_owned_brokers
self._wait_all()
File "/usr/local/lib/python2.7/dist-packages/pykafka/producer.py", line 567, in _wait_all
self._raise_worker_exceptions()
File "/usr/local/lib/python2.7/dist-packages/pykafka/producer.py", line 240, in _raise_worker_exceptions
reraise(*self._worker_exception)
File "/usr/local/lib/python2.7/dist-packages/pykafka/producer.py", line 621, in queue_reader
self.producer._send_request(batch, self)
File "/usr/local/lib/python2.7/dist-packages/pykafka/producer.py", line 521, in _send_request
exc = ERROR_CODESpresponse.err
KeyError: 21
But thinking about my main issue (loosing messages)... It's seems strange that sync-producer gets all the messages produced successfully, while the async-producer looses some at the beginning of the batch. It certainly has to do with the producer loosing a connection in 20-30 minutes inactivity of the producer. But looking at the pykafka code, a mechanism looks in place to_retry sending lessages after a connection loss. And this doesn't seems to happen. So I could fix this by stopping and starting the producer... but it doesn't solve the real issue. Loosing a connection could also happen by accident and then the async-producer shouldn't loose messages neither.
I'm very willing to help you giving the right information, but I'm not a pykafka expert. I tried to understand the code behind producing, but it's a bit complex. So may be you can give me some hints on giving you the right information to pinpoint the problem.
Thanks!

hi @emmett9001, do you maybe have a little bit of time to look into my last comment? Can I help in getting the right information for you to analyse this issue?

Looks like I gave you some wrong advice. Looking more closely at the Kafka protocol documentation, 2 is not a valid value for required_acks on produce requests. It can only be 1, -1, or 0. It looks like -1 is what you might want in this case.
As for the issue you're noticing with the asynchronous producer, I agree that it looks a lot like a real bug in the python implementation. I personally don't have the bandwidth to dig into it, but I'll keep this ticket open for reference in case a community member wants to look into it.

Hi @rubinatorz, are you using any compression? e.g. lz4? Can you provide the code that you created the producer?

hi @carsonip Nope, not using any compression. I'm setting up a KafkaClient object:
client = KafkaClient(hosts='my cluster', broker_version='1.1.0', ssl_config=None)
Next step is getting a Producer object from the topic list:
p = client.topics[topic_name].get_producer(min_queued_messages=1, linger_ms=5, delivery_reports=True, compression=0, use_rdkafka=False, sync=True)
And then calling:
producer.produce(msg, timestamp=datetime.now())
Any ideas on the cause or having the same problem?

The last time I got an error like this, since the error was unknown to the client side (pykafka), it was easier for me to look at the logs of kafka and find out the exact error happened in kafka that caused the problematic response.

@rubinatorz Sorry I misunderstood the issue earlier today. It would still be useful if you check if there are any errors on the kafka's side.
I would try to reproduce the issue when I have time. Also, is there a firewall / NAT or anything that triggered the disconnection? It would help to reproduce the issue.


Add Microsoft.Extensions.Logging.AzureWebAppDiagnostics to NuGetPackageVerifier.json

@Tratcher

:shipit:

Per @pranavkm, this is not required, as standard checks apply to all packages. Only if you need to have some exceptions or something, you can ignore this file when you have a new package.
I recommend closing this PR without merging.

When did this change? And why do we still get CI warnings about missing packages?

Not sure since when this is just a warning and not an error. @pranavkm is thinking about removing the warning.


Added ViewRecentRoomHistory + tests

1.Added ViewRecentRoomHistory + tests
2. Made test code more reliable

I didn't see this till just now..my bad.
I changed the test a bit just to create a new room (unique name) everytime just so I could get this republished to nuget.  Which obv caused a conflict with your updated test code

OK - bad luck.
Can you do the merge or will I have to modify the code in my fork?
Frank
Von<EMAIL_ADDRESS>Gesendet: Freitag, 30. Januar 2015 04:25
An: KyleGobel/Hipchat-CS
Cc: Frank Neumann
I didn't see this till just now..my bad.
I changed the test a bit just to create a new room (unique name) everytime just so I could get this republished to nuget. Which obv caused a conflict with your updated test code
—
Reply to this email directly or view it on GitHub.

I merged it :).  Thanks Frank

Thanks. Are you going to create a new version for Nuget as well?
I would like to notify my students to use the new version.

Best regards,

Frank

Gesendet: Freitag, 30. Januar 2015 um 14:23 Uhr
Von: "Kyle Gobel"<EMAIL_ADDRESS>An: KyleGobel/Hipchat-CS<EMAIL_ADDRESS>Cc: frankneumann<EMAIL_ADDRESS>Betreff: Re: [Hipchat-CS] Added ViewRecentRoomHistory + tests (#19)
I merged it :). Thanks Frank
—
Reply to this email directly or view it on GitHub.

yup new version is pushed.
We'll have to figure out something soon about the tests, might just have to disable all of them for the build server.  If i test it locally, and then try to run a build on myget I get rate limited exceptions...have to wait out the cooldown and rebuild


[Removal]: Striped Garnet Tarantula  - What should be that "sufficient evidence" ????

Modifications
No
Extra forwarders
No
Extra antennas
No
Additional Information
"This hotspot was bulk added to the denylist along withe the whole town. Even 4 weeks later , today the hotspot is appearing with "On Denylist". Why is that ? I provided all the needed information and it is clearly flagged incorrectly. Thanks"
"We do not have sufficient evidence to remove these hotspots from the denylist. This is an automated comment due to the volume of addition and removal requests. A future tool is in the works to provide more insight into the analysis approach made by the Helium team and community members that maintain this list."

@jthiller Hi,i thought they were being honest,not game hotspots.
Maybecan check them.


Bad link to prism-react-renderer's list of default languages

Have you read the Contributing Guidelines on issues?

[X] I have read the Contributing Guidelines on issues.

Description
On this page:
https://docusaurus.io/docs/markdown-features/code-blocks#supported-languages
It says:
By default, Docusaurus comes with a subset of commonly used languages.
That link is broken
Self-service

[ ] I'd be willing to address this documentation request myself.

I'd like to work on this


How to disable input but not select?

When working with tablett or Phone there are cases where we don't want show keybord (qwerty), but just want user to select from dropdown. How to do that?
Another similar question: how to open there numeric keypad, not qwerty?

[noAutoComplete]="true"
Does it help you?


Fix template function in TagBoundEncoder

Should take argument with perfect forwarding, otherwise non-copyable
arguments don't work.
Problem
Can't encode non-copyable/non-movable type.
Change overview
Fix template function signature.
Testing
Built, ran cert tests.

fast track: trivial fix


Rewrite try/protect in Lua, support table errors

Merged  #162 that implements it in C.

Do

Probably not, there are a few implementations available on Lua wiki and elsewhere already.

Would it be worth it to leave those links here for posterity?

I think so. Particularly because of the new idea of checking the private metatable to make sure the error was thrown by try(). Do these other Lua implementations use the same idea?

In my library i add special unique key to error wrap table.


[W.I.P] [DO NOT MERGE] fix: change flows

This Draft is open to clarify a few issues on how this example should be implemented.
Changes introduced
I think that urbandict is not working as it is supposed.
These changes inser the flow I think it should follow. Although I have not been able to make it work b/c of problems at DocIndexer. I can't make it use matches as traverse key, and I do not know how to make it use parent_id as key for the Indexer query.
The idea can be summarized:
Index time
!Flow
pods:
  splitter:
    uses: yaml/craft-split.yml
    **explanation**: First, at doc-level-0, we extract the word and put it into **meta_info**. and keep the definitions as a comma-separated set of **definitions**. In the next step (**CompoundExecutor**), it uses a **Sentencizer** to generate from this `level-0-depth document`, a set of chunks (`level-1-depth documents`) (as many chunks as number of descriptions (; separated))
  encoder:
    **explanation**: Here, encode every description (every chunk (`level-1-depth document`)). This is why IndexRequest has depth_range: [1, 1]
  chunk_indexer:
    **explanation**: Put every embedding to a vector Index. As a CompundIndexer, keep a key-value index, with the key being the chunk id (`level-1-depth document`) and the value its PB content.
  doc_indexer:
    uses: yaml/index-doc.yml
    needs: gateway
    **explanation**: Take directly the document (`level 0`) and index its document. It would even be cooler to split `craft` into 2 pods, and have it need the `splitter` instead of the `gateway`. This way it could keep the text as the set of descriptions and the word in the meta-info (So it can be extracted like this at query time).
  join_all:
    uses: _merge
    needs: [doc_indexer, chunk_indexer]
    read_only: true

Search time
!Flow

pods:
  encoder:
    uses: yaml/encode.yml
    **explanation**: Since the input will be a simple description, no need to craft it. So we need to encode directly the level-0 document. This is why **SearchRequest** has depth_range: [0, 0]
  chunk_indexer:
    uses: yaml/index-chunk.yml
    **explanation**: Here it needs to search for the closest vectors in the **vector indexer**. It needs to search based on the 0-depth document. Afterwards, from the **matches** (that is why matches is used as traverse key), extract the values from the key-value index.
  ranker:
    uses: MinRanker
  doc_indexer:
    uses: yaml/index-doc.yml
    **explanation**: I need to traverse on `matches` and extract from key-value pair based on the parent_id of the matches. Does it make sense? We would need something like this (Although I think we need a different driver for it) (see PR link)

Linked to https://github.com/jina-ai/jina/pull/758

yes, it makes sense. One will need to traverse matches and find the parent_id

And how one would implement this pattern with the actual codebase?

So maybe we need to change in  jina-ai/jina#758 to have a new driver. If we extract from parent_id we do not want to merge with the document maybe, we just want to extract what is in there? or merge and then exclude some fields?

I see. The point is that there is no way to track the parent id all the way back to the top level.

Should work after issue https://github.com/jina-ai/jina/issues/763 is solved

I guess after closing https://github.com/jina-ai/jina/issues/763 we should need to add Chunk2DocRankerDriver explicitly in the yaml file?

Hi Joan, #148 I think these changes can avoid problems and make urbandict working normally. Although the broken query result is still un-fixed, this has no urgent now.

Nope, we still need changes on the jina side. The more discussion is at
https://github.com/jina-ai/jina/issues/799

Yes, but besides this the changes of the new commit are also needed?

Current logs of an example with 5 documents indexed.
urbandict-issue-message-tracing.txt

This PR is ready for merge once in core this PR is merged:  https://github.com/jina-ai/jina/pull/851

Ready to merge


Vulnerability

You can escape the sandbox using something like this:
const compiler = require('@nx-js/compiler-util');

const code = compiler.compileCode('} { return global.process.env.LOGNAME');
const user = code({});
console.log(user);

Hi,
as possible workaround you could clean the source string before to put into the compiler:
const compiler = require('@nx-js/compiler-util');

const sourceCode = '} { return global.process.env.LOGNAME';
const cleanSourceCode = sourceCode.replace(/^([\n\r\t\s;]*}[\n\r\t\s;]*{)*/, '').trim();
const code = compiler.compileCode(cleanSourceCode);
const user = code({});
console.log(user);

@mgesmundo That will not be enough, you could use '1 + 1 } { return global.process.env.LOGNAME' (or any other preceding valid JavaScript before the first closing curly bracket) to bypass it.
To prevent this, you'll have to parse all the code and balance curly brackets.

@saltimbanc this updated RegEx /^([\w\W\d\s]*}.*{)*/ removes all code before the dangerous couple of curly brackets } {.

Would transpiling code from typescript to JavaScript help mitigate such attacks?

@mgesmundo That RegExp matches valid code, something like function one(){return "something"}function two(){return "another thing"}, also, you can break it by using a new line between the opening and closing brackets, }\n{ because . doesn't match new lines if you don't add the s flag to the RegExp.
I don't think you can fix this with regex, you could use /^([^{]*}.*{)*/gms but even this can be broken by '{'}{, you could extend the regex by making sure the opening curly bracket is not inside a string literal but I don't know if there is any reliable way to do that in RegExp.
RegExp is not my forte, I think you could check for { characters inside string literals with something like this: /^((('||"){\1)|[^{]}.{)*/gmsbut you can it this using''{'}{` (escaping the quote inside the literal).

@natarius Probably. This exploit uses invalid syntax so I guess the code will not be transpiled.

yeah, seems like typescript transpiring catches this!
here are the options I used:
`const compilerOptions = {
allowJs: true,
target: ts.ScriptTarget.ES2020,
lib: ["ES2020"],
module: ts.ModuleKind.CommonJS,
noEmitOnError: true,
};
     const jsCode = ts.transpileModule(tsCode, {compilerOptions});
     return jsCode.outputText;

`

The TipeScript solution is interesting. The updated RegEx /^([\w\W\d\s]*}[\w\d\s\n]*{)*/ cover the @saltimbanc issue. My previous RegEx was wrong because the . handles every chars, included the bracket. The vulnerability depends on the first closed bracket followed by an opened bracket. All code before this first (closed) bracket and the second (opened) should be removed anyway. The code after the second opened bracket will be safe again. I don't see IMHO any reason to keep some valide code inside a dangerous bracket couple } { or before the first one only.
It would be interesting to write some test case.

@mgesmundo will you commit the regex to the repo?

@solkimicreb do you like a PR with this proposed fix and its tests? Do you have some other solution other than the RegEx and TypeScript?

@mgesmundo You can break that RegExp using "{";}"";{

@saltimbanc For my needs I use an async wrapper:
const compiler = require('@nx-js/compiler-util')

const sleep = (milliseconds) => {
  return new Promise(resolve => setTimeout(resolve, milliseconds))
}

compiler.expose = [ 'console', 'Promise' ]

const test = async (src) => {
  // Fix vulnerability: https://github.com/nx-js/compiler-util/issues/12
  const cleanSrc = src.replace(/^([\n\r\t\s;]*}[\n\r\t\s;]*{)*/, '').trim();
  const source = `
    const $wrapperFunction = async () => {
      ${cleanSrc}
    }
    return $wrapperFunction()
  `

  const code = compiler.compileCode(source);

  return code({}, { sleep });
}

const source1 = `
  } {
  return async ms => {
    await sleep(ms);
    return 'Hello World!';
  }
`
const source2 = `"{";}"";{ return global.process.env.LOGNAME`

const getResult = async () => {
  try {
    const r = await test(source1);
    console.log('>>> result is', await r(1000));
  } catch (e) {
    console.log('>>> invalid source code')
  }
  try {
    const r = await test(source2);
    console.log('>>> result is', await r(1000));
  } catch (e) {
    console.log('>>> invalid source code')
  }
  const code = compiler.compileCode(source2);
  console.log('>>> vulnerable', code({}))
}

getResult();

// after 1s
// >>> result is Hello World!
// >>> invalid source code
// >>> vulnerable marcello

Wouldn't it be possible to prevent this vulnerability by simply changing the "wrapper" to include a return statement ?
Right now what the wrapper does is essentially with (sandbox) { code }.
But it could be changed to with (sandbox) { return code }
This way the code block can't inject a closing curly brace to escape the sandbox and then execute other code, because the other code would never be reached due to the return statement.
Obviously this will change the semantics a bit, because to execute multiple statements you would have to wrap them in an immediately invoked function expression (IIFE), but it sounds like a minor price to pay for having a less vulnerable sandbox.
If the worry is breaking existing code that relies on the existing behavior, a new "safer" method can be added that implements this, and people can choose which one to use (simpler + less safe) or (more boilerplate + safer)

Also wouldn't it be also possible to escape the sandbox by simply getting hold of any function, reaching its constructor (i.e. Function), then using it to create a new un-sandboxed evaluation.
See this for example: https://runkit.com/feralgeometry/5f7dcaf7cc4527001aed3119


using "runtimeVersion" overloads console on launch

Type: Bug
We run NodeJS servers and in our launch.json file we generally like to put the runtimeVersion. What happens though is that it will dump the PATH properties to the console, which exceeds the character limit of a command, and then the debugger never starts. The workaround is to remove "runtimeVersion" from the launch properties.
Here is a sample from the terminal window when I tried to launch, and what you will notice is that the text ends with "and" which was supposed to be "android"
/usr/bin/env ENVIRONMENT_NAME=local 'PATH=/Users/nicholas/.nvm/versions/node/v12.14.1/bin:/Users/nicholas/.nvm/versions/node/v12.14.1/bin:/Users/nicholas/.pyenv/shims:/Users/nicholas/Library/pnpm:/Users/nicholas/.yarn/bin:/Users/nicholas/.config/yarn/global/node_modules/.bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Users/nicholas/.nvm/versions/node/v18.16.0/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/usr/local/git/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Xamarin Workbooks.app/Contents/SharedSupport/path-bin:/opt/local/bin:/opt/local/sbin:/Users/nicholas/Development/SDKs/android-sdk-macosx/platform-tools/:/Users/nicholas/Development/SDKs/and~/Development/CCHome/cchome-config-server:  /usr/bin/env ENVIRONMENT_NAME=local 'PATH=/Users/nicholas/.nvm/versions/node/v12.14.1/bin:/Users/nicholas/.nvm/versions/node/v12.14.1/bin:/Users/nicholas/.pyenv/shims:/Users/nicholas/Library/pnpm:/Users/nicholas/.yarn/bin:/Users/nicholas/.config/yarn/global/node_modules/.bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Users/nicholas/.nvm/versions/node/v18.16.0/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/usr/local/git/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Xamarin Workbooks.app/Contents/SharedSupport/path-bin:/opt/local/bin:/opt/local/sbin:/Users/nicholas/Development/SDKs/android-sdk-macosx/platform-tools/:/Users/nicholas/Development/SDKs/and
VS Code version: Code 1.84.2 (Universal) (1a5daa3a0231a0fbba4f14db7ec463cf99d7768e, 2023-11-09T10:52:33.687Z)
OS version: Darwin arm64 23.1.0
Modes:

System Info

Item
Value

CPUs
Apple M1 Max (10 x 24)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: enabled_ondirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: enabled

Load (avg)
5, 5, 4

Memory (System)
32.00GB (0.13GB free)

Process Argv
--crash-reporter-id 8a8708a2-857f-4160-b1d7-75d09a14ed9d

Screen Reader
no

VM
0%

Extensions (59)

Extension
Author (truncated)
Version

logcat
abh
0.0.7

android-dev-ext
ade
1.3.2

cloudformation-yaml-validator
cha
0.3.15

npm-intellisense
chr
1.4.5

aws-cloudformation-yaml
Dan
0.2.2

vscode-eslint
dba
2.4.2

fileheadercomment
doi
0.0.5

vscode-jest-runner
fir
0.4.69

flow-for-vscode
flo
2.2.1

gc-excelviewer
Gra
4.2.58

todo-tree
Gru
0.0.226

duk-debug
Har
0.5.6

vscode-test-explorer
hbe
2.21.1

rest-client
hum
0.25.1

vscode-hexdump
ili
1.8.2

vscode-peacock
joh
4.2.2

mdhelper
jos
0.0.11

vscode-position
jtr
1.1.2

vscode-jest-test-adapter
kav
0.8.1

vscode-cfn-lint
kdd
0.25.1

vscode-docker
ms-
1.28.0

csharp
ms-
2.10.28

vscode-dotnet-runtime
ms-
2.0.0

vscode-edge-devtools
ms-
2.1.3

isort
ms-
2023.10.1

python
ms-
2023.20.0

vscode-pylance
ms-
2023.11.10

jupyter
ms-
2023.10.1100000000

jupyter-keymap
ms-
1.1.2

jupyter-renderers
ms-
1.0.17

vscode-jupyter-cell-tags
ms-
0.1.8

vscode-jupyter-slideshow
ms-
0.1.5

remote-containers
ms-
0.321.0

azure-account
ms-
0.11.6

cpptools
ms-
1.18.5

js-debug-nightly
ms-
2023.11.1317

makefile-tools
ms-
0.7.0

test-adapter-converter
ms-
0.1.8

vscode-js-profile-flame
ms-
1.0.8

vscode-typescript-next
ms-
5.4.20231116

vsliveshare
ms-
1.0.5892

vsliveshare-pack
ms-
0.4.0

debugger-for-edge
msj
1.0.15

vscode-jest
Ort
5.2.3

vscode-code-outline
pat
0.2.1

java
red
1.24.0

json-to-js
ren
0.2.0

bash-debug
rog
0.3.9

code-spell-checker
str
3.0.1

vscode-stylelint
sty
1.3.0

intellicode-api-usage-examples
Vis
0.2.8

vscodeintellicode
Vis
1.2.30

vscode-java-debug
vsc
0.55.0

vscode-java-dependency
vsc
0.23.2

vscode-java-pack
vsc
0.25.15

vscode-java-test
vsc
0.40.1

vscode-maven
vsc
0.43.0

vscode-icons
vsc
12.6.0

vscode-import-cost
wix
3.3.0

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
py29gd2263cf:30880073
vscaac:30438847
vsclangdc:30486549
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
2e4cg342:30602488
f6dab269:30613381
showlangstatbar:30737416
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
welcomedialog:30887143
pythonnosmt12:30797651
pythonidxpt:30866567
pythonnoceb:30805159
synctok:30869157
dsvsc013:30795093
dsvsc014:30804076
dsvsc015:30845448
pythontestfixtcf:30871695
pythonregdiag2:30871582
pyreplss2:30886141
pythonmypyd1:30879173
pythoncet0:30885854
2e7ec940:30885897
pythontbext0:30879054
accentitlementsc:30887149
dsvsc016:30886110
dsvsc017:30886112
dsvsc018:30886114
aa_t_chat:30882232

@guycnicholas could you verify that this is fixed in the latest Insiders build?
or also @connor4312 if we could get verification steps?

Have a node.js launch config with something in the env key, and console: "integratedTerminal"
Run it
Verify you no longer see a massive dump of the environment when the terminal is created

@justschen Yepper, it looks glorious!  Thanks

Thanks for confirming!

@connor4312 Argggg....there is a problem, and I am not sure how our team can work around it.  I was going to log a bug, but I think it is this fix causing it. Scenarios:
With the newest 1.88 release the debugging/console behavior changed a bit and broke.

launch with "console": "integratedTerminal" - result breakpoints do not get hit, output goes to terminal window.
launch with "console": "internalConsole" - BPs get hit, output goes to Debug console
launch with no console item - BPs get hit, output goes to Debug console

The 1.87.2 release hits BPs on all 3 scenarios, and the output behaves the same as 1.88
The issue is, we typically run from the integrated terminal as we paste our AWS Klam credentials there so the app can pick them up.  The other terminal options do not allow for that as I am not sure how to get to them.
I didn't even consider something like this, so my testing was limited to launching some of my servers.  I then didn't continue using the early build as it didn't pickup the settings from my released version, so I swtiched back.
Are there any insights you can provide?

It sounds like there is some behavioral difference in your application regarding whether it's run with a PTY or not. I would need more details to know how to recommend a fix.

VSCodeTest.zip
I have recommended to my teammates that they keep the 1.87 version around until I fix happens, which I assume would not happen until the April release at a minimum.
The bug being the lack of hitting any BPs when using the integrated term. As to "workaround", I was more hoping there could be a way to keep the debugger attached.
Presently, it looks like it connects and disconnects immediately. Here is the output from my sample.
/Users/nicholas/.nvm/versions/node/v16.16.0/bin/node ./test.js
Debugger attached.
Waiting for the debugger to disconnect...
~/Development/VSCodeTest:  /Users/nicholas/.nvm/versions/node/v16.16.0/bin/node ./test.js
9
27

This does work for me, it's possible the environment variables are not getting applied in your terminal correctly:

Which shell are you using?
Can you console.log(process.env.NODE_OPTIONS) and let me know if that prints everything?

I think we are having a Miss-Q on this. The problem isn't that the integrated terminal doesn't have env vars.
The problem is the debugger doesn't attach. Try to put a breakpoint in the code and see if you get there.

The debugger uses environment variables to set up its attachment to the program. Breakpoints work for me, using zshell on macos. If there's some setup in your environment which prevents those from getting through (either a shell that might not accept environment variables, or a configuration that overwrites NODE_OPTIONS in your dotfiles) attachment won't work. You can see this if you log process.env.NODE_OPTIONS in a correctly-attached debug sessions.

Oh...sorry. I get:
 --require /Users/nicholas/.vscode/extensions/ms-vscode.js-debug-nightly-2024.3.2217/src/bootloader.js  --inspect-publish-uid=http

hm, okay, that should work find. Can you grab a log?

If you're able to, add "trace": true to your launch.json and reproduce the issue. The location of the log file on your disk will be written to the Debug Console. Share that with us.
⚠ This log file will not contain source code, but will contain file paths. You can drop it into https://microsoft.github.io/vscode-pwa-analyzer/index.html to see what it contains. If you'd rather not share the log publicly, you can email it to<EMAIL_ADDRESS>

I don't get much in the file, the attached image shows what file I am looking at, and it is the newest, but it's contents are only:
{"cwd":"/Users/nicholas/Development/VSCodeTest","processId":45115,"nodeVersion":"v16.16.0","architecture":"x64"}

Ohhh, I see what's happening. This is unfortunate.
You have nvm in your dotfiles, and it runs an npm command to do 'something' at some point, which is when the debugger attaches. So the debugger attaches, it runs, and the debugger thinks the session is over and cleans up its mess (including the unix socket used to communicate with the debugee.) Then the real program runs, sees the socket is gone, and thinks that it must no longer be in debug mode.
I'm going to back this change out in a patch release and figure out how to reimplement it. Sorry for the hassle.

Thanks for your effort and glad you figured it out.
Setting the runtimeVersion is there because it is desirable to run each server using the version contained in its Docker container, which then requires us to have multiple versions on our machines...which of course, leads to NVM.

VS Code 1.88.1 has been released with the fix

I can still reproduce this issue on Macos.

VSCode info
Version: 1.91.0
Commit: ea1445cc7016315d0f5728f8e8b12a45dc0a7286
Date: 2024-07-01T18:53:23.353Z
Electron: 29.4.0
ElectronBuildId: 9728852
Chromium: 122.0.6261.156
Node.js: 20.9.0
V8: 12.2.281.27-electron.0
OS: Darwin arm64 23.5.0

Not sure if relevant, my terminal is zsh.
launch.json
{
          "type": "node",
          "request": "launch",
          "name": "Test integratedTerminal",
          "skipFiles": [
              "<node_internals>/**"
          ],
          "program": "${workspaceFolder}/index.js",
          "console": "integratedTerminal",
          "runtimeVersion": "18"
      }

Checking the Terminal logs I can see that the generated command is 1437 characters long (due to long PATH in env)
However only the first 1024 characters are sent correctly and parsed

Removing the runtimeVersion works (using the same Node v18.20.3)
The command generated is only 671 characters long (strangely it does not contain PATH?)
Even more strangely, if I keep this terminal open in vscode, add back the runtimeVersion and run the program again, now it works with 1437 character command!
Closing the terminal, and running again fails.
So the first command in a new terminal is limited to 1024 characters ??

Likewise. I am on 1.91.1 and I tried all 3 consoles, integrated and external both write a terminated path to the terminal and fail to continue to launch. When set to internal, it just launches and quits.

Same, using nvm.
So far my solution is to:

remove "runtimeVersion" from launch.json
make sure I run nvm use to load the correct node version for the project.

same, but the point of having the version in there is that we then match it to the version in the docker container we build with, and then all devs are ensured of running with the correct version without having to think about it.

I cloned the VS Code repo and ran it from the source code to try and debug the issue, but I couldn't reproduce the problem in debug mode.
I'm not sure if the issue is fixed in the latest version on the main branch or if there is a difference between running a production build of VS Code versus a development build.
The development build seems slower, so if the issue is related to timing or a race condition, it might not occur in the slower version.
I'll try running from the 1.91.0 tag later, to see if that works or not

This issue is not fixed, it's still open

I checked out 1.91.0 and ran it from source code.
With this version I can reproduce the issue 8/10 times.
If I add a breakpoint and pause for a second before sendText, it works. So I can't inspect it to see what fails.
If I add a 1s timeout before sendText here, it seems to work every time
https://github.com/microsoft/vscode/blob/1.91.0/src/vs/workbench/api/browser/mainThreadTerminalService.ts#L208
Might also be related with https://github.com/microsoft/vscode/issues/38137

I don't think there's any solution to this, if someone is smarter and comes up with an idea please let me know.


Inconsistency with 2.4.4: the behavior of v-model + v-on on the same element

Version
2.5.2
Reproduction link
https://jsfiddle.net/8gorckx3/
Steps to reproduce
Click the checkbox once (or several times, no matter)
What is expected?
Properties "on" and "ison" to end up having the same value
What is actually happening?
Properties "on" and "ison" end up having the opposite values

In<EMAIL_ADDRESS>(at least) the same experiment results in values of "on" and "ison" being the same, which seems more natural to me, as the new state of checkbox is exactly what I'm interested in inside the event handler.
This seems like a trivial use case so I'm not sure if this is a regression bug or a feature :) Either way, the fact that backward compatibility with 2.4.4 is broken is surely a bad thing, especially since npm installs 2.5.2 when given "^2.4.4" version range.

This is an intentional change to fix a couple bugs caused by event firing orders. In fact if you try 2.4.4 in Firefox, you will see that it has the behavior of 2.5.x in Chrome.
2.5.x has consistent behavior in all browsers, so this is considered a fix.


Treat logger.ApplicationError as Critical

https://github.com/aspnet/Hosting/blob/e62ceb852827d386b492abc9cc6801b1a8c70263/src/Microsoft.AspNetCore.Hosting/Internal/HostingLoggerExtensions.cs#L52
Wouldn't it be better to treat startup exceptions as LogLevel Critical?
/// <summary>
/// Logs that describe an unrecoverable application or system crash,
/// or a catastrophic failure that requires immediate attention.
/// </summary>
Critical = 5,

If you have a web application that doesn't start up, it's as bad as it can get in my opinion. Using Critical would make it easier to raise alerts etc.

This makes sense to me. @Tratcher let me know what you think.

Sounds good.


Improves visualisation

This PR improves visualisation:

Show version beside map marker.
Fix issue that map doesn't zoom at first open.
Increase map height
Show lat, lon in table
Show version which is invisible in red (removed)

Demo:

https://aleung.github.io/osm-deep-history/#/node/3098740415
https://aleung.github.io/osm-deep-history/#/node/1

It should have fixed #7

This PR is outdated. See #14


About loading weights on docker

I would like to load weights on the provided docker, but it is not working, so I have a question.
Here is a detailed description of how I did it.
First, I started the container and logged in as root.
docker pull deepdrive/spc
docker run -it --gpus all deepdrive/spc
At this time, the nvidia-smi command worked fine.

I then git cloned spc2 and placed the downloaded weights in /home/carla/spc2/pretrain/. (I renamed the file to model_only_fpn50.pth.)
Enter the spc virtual environment.
conda activate spc
Open a separate terminal and start carla.
docker exec -it <container ID> bash
(spc) carla@325657a8b89c:~/carla_0.8.4$ bash run.sh
Confirm that the message "Waiting for the client to connect..." is displayed.
Tried to start training in the previous terminal.
(spc) root@325657a8b89c:~/spc2/scripts# bash train_carla.sh
Here are the steps I followed so far. However, I could not start as follows.
Please let me know if there is something wrong. Thank you very much.
(spc) root@325657a8b89c:~/spc2/scripts# bash train_carla.sh  No model to resume! turn to the base pretrain model: pretrain/model_only_fpn50.pth Traceback (most recent call last):   File "main.py", line 95, in <module>     main()   File "main.py", line 91, in main     train_policy(args, env)   File "/home/carla/spc2/train.py", line 334, in train_policy     trainer = Trainer(args, env)   File "/home/carla/spc2/train.py", line 124, in __init__     self.model, self.optim, self.epoch, self.exploration, self.num_steps = init_models(self.args)   File "/home/carla/spc2/models/model.py", line 238, in init_models     train_net, epoch, step = load_model(args, args.save_path, train_net, resume=args.resume)   File "/home/carla/spc2/utils/util.py", line 327, in load_model     net.load_state_dict(state_dict)   File "/home/carla/anaconda3/envs/spc/lib/python3.5/site-packages/torch/nn/modules/module.py", line 845, in load_state_dict     self.__class__.__name__, "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for ConvLSTMMulti: 	Missing key(s) in state_dict: "conv_lstm.feature_map_predictor.identify.bias", "conv_lstm.feature_map_predictor.identify.weight", "conv_lstm.feature_map_predictor.stair.bias", "conv_lstm.feature_map_predictor.stair.weight", "conv_lstm.detector.coll_head.0.bias", "conv_lstm.detector.coll_head.0.weight", "conv_lstm.detector.coll_head.2.bias", "conv_lstm.detector.coll_head.2.weight", "conv_lstm.detector.coll_head.4.bias", "conv_lstm.detector.coll_head.4.weight", "conv_lstm.detector.coll_head.6.bias", "conv_lstm.detector.coll_head.6.weight", "conv_lstm.detector.coll_head.8.bias", "conv_lstm.detector.coll_head.8.weight", "conv_lstm.detector.thrd_head.confidence_head.0.bias", "conv_lstm.detector.thrd_head.confidence_head.0.weight", "conv_lstm.detector.thrd_head.confidence_head.2.bias", "conv_lstm.detector.thrd_head.confidence_head.2.weight", "conv_lstm.detector.thrd_head.confidence_head.4.bias", "conv_lstm.detector.thrd_head.confidence_head.4.weight", "conv_lstm.detector.thrd_head.confidence_head.6.bias", "conv_lstm.detector.thrd_head.confidence_head.6.weight", "conv_lstm.detector.thrd_head.confidence_head.8.bias", "conv_lstm.detector.thrd_head.confidence_head.8.weight", "conv_lstm.detector.thrd_head.residual_head.0.bias", "conv_lstm.detector.thrd_head.residual_head.0.weight", "conv_lstm.detector.thrd_head.residual_head.2.bias", "conv_lstm.detector.thrd_head.residual_head.2.weight", "conv_lstm.detector.thrd_head.residual_head.4.bias", "conv_lstm.detector.thrd_head.residual_head.4.weight", "conv_lstm.detector.thrd_head.residual_head.6.bias", "conv_lstm.detector.thrd_head.residual_head.6.weight", "conv_lstm.detector.thrd_head.residual_head.8.bias", "conv_lstm.detector.thrd_head.residual_head.8.weight", "conv_lstm.detector.thrd_head.dimension_head.0.bias", "conv_lstm.detector.thrd_head.dimension_head.0.weight", "conv_lstm.detector.thrd_head.dimension_head.2.bias", "conv_lstm.detector.thrd_head.dimension_head.2.weight", "conv_lstm.detector.thrd_head.dimension_head.4.bias", "conv_lstm.detector.thrd_head.dimension_head.4.weight", "conv_lstm.detector.thrd_head.dimension_head.6.bias", "conv_lstm.detector.thrd_head.dimension_head.6.weight", "conv_lstm.detector.thrd_head.dimension_head.8.bias", "conv_lstm.detector.thrd_head.dimension_head.8.weight", "conv_lstm.detector.thrd_head.center_head.0.bias", "conv_lstm.detector.thrd_head.center_head.0.weight", "conv_lstm.detector.thrd_head.center_head.2.bias", "conv_lstm.detector.thrd_head.center_head.2.weight", "conv_lstm.detector.thrd_head.center_head.4.bias", "conv_lstm.detector.thrd_head.center_head.4.weight", "conv_lstm.detector.thrd_head.center_head.6.bias", "conv_lstm.detector.thrd_head.center_head.6.weight", "conv_lstm.detector.thrd_head.center_head.8.bias", "conv_lstm.detector.thrd_head.center_head.8.weight", "conv_lstm.depth_head.0.bias", "conv_lstm.depth_head.0.weight", "conv_lstm.depth_head.2.bias", "conv_lstm.depth_head.2.weight", "conv_lstm.depth_head.4.bias", "conv_lstm.depth_head.4.weight", "conv_lstm.depth_head.6.bias", "conv_lstm.depth_head.6.weight", "conv_lstm.guide_layer.fc11.bias", "conv_lstm.guide_layer.fc11.weight", "conv_lstm.guide_layer.fc22.bias", "conv_lstm.guide_layer.fc22.weight", "conv_lstm.guide_layer.fc33.bias", "conv_lstm.guide_layer.fc33.weight", "conv_lstm.coll_layer.fc11.bias", "conv_lstm.coll_layer.fc11.weight", "conv_lstm.coll_layer.fc22.bias", "conv_lstm.coll_layer.fc22.weight", "conv_lstm.coll_layer.fc33.bias", "conv_lstm.coll_layer.fc33.weight", "conv_lstm.offroad_layer.fc11.bias", "conv_lstm.offroad_layer.fc11.weight", "conv_lstm.offroad_layer.fc22.bias", "conv_lstm.offroad_layer.fc22.weight", "conv_lstm.offroad_layer.fc33.bias", "conv_lstm.offroad_layer.fc33.weight", "conv_lstm.offlane_layer.fc11.bias", "conv_lstm.offlane_layer.fc11.weight", "conv_lstm.offlane_layer.fc22.bias", "conv_lstm.offlane_layer.fc22.weight", "conv_lstm.offlane_layer.fc33.bias", "conv_lstm.offlane_layer.fc33.weight", "conv_lstm.speed_layer.fc11.bias", "conv_lstm.speed_layer.fc11.weight", "conv_lstm.speed_layer.fc22.bias", "conv_lstm.speed_layer.fc22.weight", "conv_lstm.speed_layer.fc33.bias", "conv_lstm.speed_layer.fc33.weight".  	Unexpected key(s) in state_dict: "conv_lstm.detector_pred.loc_head.0.weight", "conv_lstm.detector_pred.loc_head.0.bias", "conv_lstm.detector_pred.loc_head.2.weight", "conv_lstm.detector_pred.loc_head.2.bias", "conv_lstm.detector_pred.loc_head.4.weight", "conv_lstm.detector_pred.loc_head.4.bias", "conv_lstm.detector_pred.loc_head.6.weight", "conv_lstm.detector_pred.loc_head.6.bias", "conv_lstm.detector_pred.loc_head.8.weight", "conv_lstm.detector_pred.loc_head.8.bias", "conv_lstm.detector_pred.cls_head.0.weight", "conv_lstm.detector_pred.cls_head.0.bias", "conv_lstm.detector_pred.cls_head.2.weight", "conv_lstm.detector_pred.cls_head.2.bias", "conv_lstm.detector_pred.cls_head.4.weight", "conv_lstm.detector_pred.cls_head.4.bias", "conv_lstm.detector_pred.cls_head.6.weight", "conv_lstm.detector_pred.cls_head.6.bias", "conv_lstm.detector_pred.cls_head.8.weight", "conv_lstm.detector_pred.cls_head.8.bias", "conv_lstm.coll_vehicle_layer.conv1.weight", "conv_lstm.coll_vehicle_layer.conv1.bias", "conv_lstm.coll_vehicle_layer.conv2.weight", "conv_lstm.coll_vehicle_layer.conv2.bias", "conv_lstm.coll_vehicle_layer.conv3.weight", "conv_lstm.coll_vehicle_layer.conv3.bias", "conv_lstm.coll_vehicle_layer.conv4.weight", "conv_lstm.coll_vehicle_layer.conv4.bias", "conv_lstm.coll_vehicle_layer.fc1.weight", "conv_lstm.coll_vehicle_layer.fc1.bias", "conv_lstm.coll_vehicle_layer.fc2.weight", "conv_lstm.coll_vehicle_layer.fc2.bias", "conv_lstm.coll_vehicle_layer.fc3.weight", "conv_lstm.coll_vehicle_layer.fc3.bias", "conv_lstm.coll_other_layer.conv1.weight", "conv_lstm.coll_other_layer.conv1.bias", "conv_lstm.coll_other_layer.conv2.weight", "conv_lstm.coll_other_layer.conv2.bias", "conv_lstm.coll_other_layer.conv3.weight", "conv_lstm.coll_other_layer.conv3.bias", "conv_lstm.coll_other_layer.conv4.weight", "conv_lstm.coll_other_layer.conv4.bias", "conv_lstm.coll_other_layer.fc1.weight", "conv_lstm.coll_other_layer.fc1.bias", "conv_lstm.coll_other_layer.fc2.weight", "conv_lstm.coll_other_layer.fc2.bias", "conv_lstm.coll_other_layer.fc3.weight", "conv_lstm.coll_other_layer.fc3.bias", "conv_lstm.guide_layer.fc1.weight", "conv_lstm.guide_layer.fc1.bias", "conv_lstm.guide_layer.fc2.weight", "conv_lstm.guide_layer.fc2.bias", "conv_lstm.guide_layer.fc3.weight", "conv_lstm.guide_layer.fc3.bias", "conv_lstm.coll_layer.fc1.weight", "conv_lstm.coll_layer.fc1.bias", "conv_lstm.coll_layer.fc2.weight", "conv_lstm.coll_layer.fc2.bias", "conv_lstm.coll_layer.fc3.weight", "conv_lstm.coll_layer.fc3.bias", "conv_lstm.offroad_layer.fc1.weight", "conv_lstm.offroad_layer.fc1.bias", "conv_lstm.offroad_layer.fc2.weight", "conv_lstm.offroad_layer.fc2.bias", "conv_lstm.offroad_layer.fc3.weight", "conv_lstm.offroad_layer.fc3.bias", "conv_lstm.offlane_layer.fc1.weight", "conv_lstm.offlane_layer.fc1.bias", "conv_lstm.offlane_layer.fc2.weight", "conv_lstm.offlane_layer.fc2.bias", "conv_lstm.offlane_layer.fc3.weight", "conv_lstm.offlane_layer.fc3.bias", "conv_lstm.speed_layer.fc1.weight", "conv_lstm.speed_layer.fc1.bias", "conv_lstm.speed_layer.fc2.weight", "conv_lstm.speed_layer.fc2.bias", "conv_lstm.speed_layer.fc3.weight", "conv_lstm.speed_layer.fc3.bias".

I checked the uploaded weights and fixed the key naming issues you reported.
The previous naming incompatibility issue is raised because of the version naming shift, I believe the backbone weights in this new version of pertained weights should be fine.
I'm closing this issue as the issue should be fixed now. But please feel free to reopen it or open a new one if you encounter any further issues.

Thank you very much.
However, I just downloaded the updated readme weight file, and it consists of the following, and I can't find the .pth filel.
Do I need to do something to rename the file?
archive
├── data
│   ├──<PHONE_NUMBER>49472
│   ├──<PHONE_NUMBER>52976
....................................................
│   └──<PHONE_NUMBER>87728
├── data.pkl
└── version

I just checked the link (https://drive.google.com/file/d/1O7u_ifm0HZE1mBCEgnRW7aoyrX7WgEoZ/view?usp=sharing), it should just contain a .pth file.  Could you double check your link?

The previous file is in zip format, but it will not load.
When you unzip it, it will be in a directory format, as described in issues.

Sorry, I didn't get your point. I checked the history, the previous version should be a "pretrained.pth" file?
But, anyways, how about the current version provided by the link above? Does it work now?

I'm sorry for the misunderstanding.
I downloaded the link above and it was in zip format. When unzipped, the directory structure looks like this.
archive ├── data │   ├──<PHONE_NUMBER>49472 .................................................... │   └──<PHONE_NUMBER>87728 ├── data.pkl └── version

That's strange, I triple checked the link: https://drive.google.com/file/d/1O7u_ifm0HZE1mBCEgnRW7aoyrX7WgEoZ/view?usp=sharing. It should just indicate to a .pth file, not a zip file. Have no idea why that happened to you.


vmware: do not create vm datastore when attach an existing volume on solidfire

Description
This PR fixes #7148

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Bug Severity

[ ] BLOCKER
[ ] Critical
[x] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

@blueorangutan package

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 5481

Trillian Build Failed (tid-6081)

@blueorangutan test rocky8 vmware-70u3

@weizhouapache a Trillian-Jenkins test job (rocky8 mgmt + vmware-70u3) has been kicked to run smoke tests

Trillian Build Failed (tid-6085)

@blueorangutan package

@rohityadavcloud a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 6040

@blueorangutan test rocky8 vmware-67u3

@shwstppr a Trillian-Jenkins test job (rocky8 mgmt + vmware-67u3) has been kicked to run smoke tests

Trillian test result (tid-6529)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 54843 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7160-t6529-vmware-67u3.zip
Smoke tests completed. 106 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_01_deploy_vm_on_specific_host
Error
18.67
test_vm_deployment_planner.py

test_02_deploy_vm_on_specific_cluster
Error
3602.74
test_vm_deployment_planner.py

test_03_deploy_vm_on_specific_pod
Error
1.32
test_vm_deployment_planner.py

test_04_deploy_vm_on_host_override_pod_and_cluster
Error
2.31
test_vm_deployment_planner.py

test_05_deploy_vm_on_cluster_override_pod
Error
2.30
test_vm_deployment_planner.py

test_09_expunge_vm
Failure
424.61
test_vm_life_cycle.py

@weizhouapache will we merge this? it is not harming anything as far as we can see, and only solidfire users can test it for us.

@DaanHoogland
I am ok with merging this. but you know, I am the author :-)

@weizhouapache will we merge this? it is not harming anything as far as we can see, and only solidfire users can test it for us.

@DaanHoogland I am ok with merging this. but you know, I am the author :-)

yes, but you kept it in draft untill now, so I am checking with you anyway.

ah, moved it to "ready for review".
I will merge this if there is no objection until the end of Friday

@weizhouapache will we merge this? it is not harming anything as far as we can see, and only solidfire users can test it for us.

@DaanHoogland I am ok with merging this. but you know, I am the author :-)

yes, but you kept it in draft untill now, so I am checking with you anyway.

ah, moved it to "ready for review". I will merge this if there is no objection until the end of Friday

ok, let's merge as you suggested but keep the issue open as @rohityadavcloud suggested until a solidfire user confirmed.

@weizhouapache I've tested this fix backported to a 4.17 env with Solidfire and VMware; it fixes the issue of previously failured where it would attempt to create datastore on an already created datastore for a solidfire iSCSI target. However, it doesn't address the issue https://github.com/apache/cloudstack/issues/7148 completely.


[RELEASE] Cut v1Beta1 of the AdminNetwork policy API

This issue will track the actual release of v1beta1 of the AdminNetworkPolicy and BaselineAdminNetworkPolicy resources.  The Beta tracking project will show the granular tracking effort.

/remove-lifecycle stale

/remove-lifecycle rotten

/remove-lifecycle rotten

@npinaeva @tssurya @danwinship @fasaxc what is missing here?
I really like to be able to move to beta with AdminNetworkPolicy this year, being permanently in alpha can lose the momentum ... we don't need all best and whistles for the first core API, we can later new features as DNS?
do we have an idea of what will be the bare minimum that we want for this API?

originally we wanted to complete everything in this project https://github.com/orgs/kubernetes-sigs/projects/32
but we may need to update the list with the latest changes. The biggest TODO seems to be around more test converage

@npinaeva @tssurya @danwinship @fasaxc what is missing here?
I really like to be able to move to beta with AdminNetworkPolicy this year, being permanently in alpha can lose the momentum ... we don't need all best and whistles for the first core API, we can later new features as DNS?
do we have an idea of what will be the bare minimum that we want for this API?

yea coincidentally I was speaking with @danwinship about this this earlier last week, The way I see it, its these 3:

Implementation status: We have Calico (https://github.com/projectcalico/calico/issues/7578) and Cilium (https://github.com/cilium/cilium/issues/23380) implementing the core/standardized fields in the API - wanted to wait a bit more to hear feedback from them, @fasaxc has been bringing up some nice nuances in upstream meetings; - (OVN-Kubernetes and Antrea have already finished those phases and implemented the standardized fields last year -we had some bug fixes from those already done)
API Status: I wanted to move Networks and Nodes egress peers into standard (will bring this up in this week's community meeting); Like you said we don't need to include DomainNames in the first run till we get some implementation feedback - we wanted to cut https://github.com/kubernetes-sigs/network-policy-api/pull/235 minor version and then make that as beta last we spoke with @astoycos. Only remaining thing which we need to settle on is tenancy - https://github.com/kubernetes-sigs/network-policy-api/pull/178 ; being able to express tenancy using same/notsame-labels was part of the original ANP enhancement so unsure if we can beta the API without including the new API replacement for tenancy - https://github.com/kubernetes-sigs/network-policy-api/pull/178#issuecomment-2313003403 - When we met in Chicago KubeCon we wanted to at least address the most common case in english "allow to yourself deny to others"
Tests: Core parts are already done mostly, we have some more enhancements to test coverage we want to get to: https://github.com/kubernetes-sigs/network-policy-api/issues?q=is%3Aissue+label%3Aarea%2Fconformance+is%3Aopen -> OVN-Kubernetes/Antrea are already running these tests, but I wanted to get some feedback from Calico/Cilium as well on the test suite and running them downstream and see how it looks. Kube-Network-Policies has also adopted it for CI coverage - so overall we seem to be good. I will try to spend some time going through the open issues for conformance and see where we stand.


Turkish language support

Hey there, thank you very much for the contribution! :) I'll have a look a bit later!

Great! Looks good now.
I haven't tested it though, it seems like we need @xou816 to approve running the spot-snapshots workflow to get a build artifact. I don't feel like building it locally from source right now.

Thanks a lot Ondras, well both of you :) workflow should run!

Great! Seems to work just fine.

Great! Seems to work just fine.

happy to see that :D

Looks good to me, merging! :)


Fix docs about JSON format query

The format=json indeed works ;)

Codecov Report

Merging #224 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #224   +/-   ##
=======================================
  Coverage   75.52%   75.52%
=======================================
  Files          32       32
  Lines         380      380
=======================================
  Hits          287      287
  Misses         93       93

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 194746b...eb0f8fa. Read the comment docs.


remote ssh broken in 86.1

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: 1.86.1
OS Version: MacOS Sonoma 14.3.1

Steps to Reproduce:

Open VSCode
Connect to remote ssh that had previously worked prior to 1.86.0.

I understand there was an issue with 1.86.0 and glibc version etc - the thread is closed due to heated discussion. So good for me that I can now connect to a server with an older version of glibc - great !
HOWEVER
It starts processes and rouges on the remote not working as before 1.86.0
I'm sure you're all experiencing this.
AND !!!
I had installed an earlier version 1.85.x to workaround accessing servers with old version of glibc and DISABLED updating. It had not updated since 1.86.0 was released which was by intention.
When I opened VSC today - IT HAD UPDATED BY ITSELF to 1.86.1

Yes, remote access is broken in 1.86.1, even when connecting to a host with glibc >= 2.28.  I cannot confirm, but it was working on 1.86.0 (remote with glibc 2.31), but I'd need to revert to 1.86.0 to check, and this does not appear to be easy to do.

Can you share the log from the Remote-SSH output channel so I can hopefully see what isn't working?
You can grab 1.86.0 if you want from here, but right click, copy link, change the version in the url to 1.86.0: https://code.visualstudio.com/updates/v1_86

Thanks for looking @Rob, however, my issue is that I am working with Servers that do not make the cut for 1.86.0 - old GLIBC version. The story is that I installed 1.85 so that I could access these servers (third party hosting) until they upgrade their OS/GLIBC. I had disabled upgrade for extensions and vscode version and everything was fine.

why'd the vscode upgrade itself to 1.86.1 (from 1.85 that had upgrade disabled) (without my/human intervention) ?
1.86.1 connects to these (legacy) servers giving a warning about old GLIBC which is great, however, processes on the remote server skyrocket and cause DoS - it's basically unusable ...

why'd the vscode upgrade itself to 1.86.1 (from 1.85 that had upgrade disabled) (without my/human intervention) ?

Are you sure that it hadn't already downloaded the upgrade by the time you disabled updating, and sure you set the right setting? You can try downgrading again and see whether it happens again.

however, processes on the remote server skyrocket and cause DoS - it's basically unusable ...

Sorry, not sure what is happening here without more details.

Thanks for looking @rob, however, my issue is that I am working with Servers that do not make the cut for 1.86.0 - old GLIBC version. The story is that I installed 1.85 so that I could access these servers (third party hosting) until they upgrade their OS/GLIBC. I had disabled upgrade for extensions and vscode version and everything was fine.
1. why'd the vscode upgrade itself to 1.86.1 (from 1.85 that had upgrade disabled) (without my/human intervention) ?

2. 1.86.1 connects to these (legacy) servers giving a warning about old GLIBC which is great, however, processes on the remote server skyrocket and cause DoS - it's basically unusable ...

Not sure.

Hey @roblourens, this issue might need further attention.
@theAusMenace, you can help us out by closing this issue if the problem no longer exists, or adding more information.

Sorry for the slow resopnse but is there still an issue here with all the latest versions?

All good now please close


Added desktop file for Linux

Hi!
I recently downloaded Nault from the AUR, which works nicely but doesn't include a desktop file. Rather than updating the AUR repo I thought it'd be more useful upstream so that everyone uses the same desktop file. There's a couple things you might want to modify, like the name or the comment. Also, I don't think the mime type exists yet but I thought it'd be a good idea to make it future proof.

Do we need anything else or is this just the extra file?
I'm not that familiar with arch linux packages.
I'm asking because of that code that is commented out: https://github.com/agilob/nault-bin/blob/master/PKGBUILD#L25

I just saw that the latest AppImage already includes a nault.desktop file:
https://github.com/Nault/Nault/releases/download/v1.9.4/Nault-1.9.4-Linux.AppImage

Oh! Sorry, I didn't know appimages already included desktop files and all. That should work, yes. The only issue I'm facing is that after extracting the appimage the only contents in usr/share/icons/ is hicolor/0x0/apps/nault.png, but 0x0 doesn't make much sense as a resolution

The current .desktop file get's probably generated by electron-builder.
Maybe some options are missing here: https://github.com/Nault/Nault/blob/master/package.json#L129
https://www.electron.build/configuration/linux

I'll try that out. I'm new to the project though, how can I build the appimage? There are no instructions in https://github.com/Nault/Nault/blob/master/DEVELOPMENT.md (AFAIK)

Please refer to the "desktop builds" section: https://github.com/Nault/Nault/blob/master/DEVELOPMENT.md#desktop-builds
Running npm run desktop:build-local under Linux should result in your AppImage file at desktop-app\build.

OK I figured it out! All that I needed to fix was the generic name and adding the icons, which I converted to the required sizes & filenames with imagemagick. Nault can be packaged for Linux easily now (after the next release with this change).

Don't know how to verify but code change looks good to me.

Looks good @marioortizmanero , thanks for your contribution!

1.9.7 with these changes is now in AUR @marioortizmanero  :)

Great!! 🚀🚀

@BitDesert shall I open a PR indicating that Nault is on the AUR on the README?

@marioortizmanero Sure, maybe somewhere in the Desktop App section here? https://github.com/Nault/Nault/blob/master/README.md#desktop-app


Add video support, continued ( #24 )

I haven't seen anyone working on video support since #24 , I thought I'd give it a shot. This PR might be a bit rough around the edges, but hopefully it's able to be added given some polish.

[x] Animation control now affects the video (including stepping)
[x] IE11 support
[x] onLoad readded (mostly, see below)
[x] FlagGroupBar/FlagGroupPanel kept as "Select image"
[x] References to "Image" changed to "Element"
[x] Other minor changes requested in the original PR

The major remaining issue is IE support.

buildRectangularFlagFromElement.js - I've been working on IE support for a while, but I just can't seem to get it to work. VideoTexture isn't supported by IE, and the only workaround I could find was drawing the video to a canvas and using the canvas as the texture. However, I can't seem to get IE to draw the video to the canvas. It'll draw images fine, but refuses to draw videos. This demo mostly replicates how the canvas is drawn in this workaround, as seen below. I'm not sure if this is an issue shared by others (I couldn't find anything else about it online) or if it just doesn't work in this specific case. Any help would be appreciated in that regard, if you know anything about it. If everything else is compatible, videos should be as well!

If we can't find a solution, it would be better to remove the workaround and its corresponding code in Flag.js to just leave it as a black flag. Maybe a warning toast or something?

Other minor notes:

Flag.js/AnimationModule.js - For some reason every reference to the <video> element turns into a dict that contains null texture data { data: [255, 255, 255], width: 1, height: 1 }. this.imageElement and this.isVideo are both defined properly on construction and reset/play (video tag, true), but is set to the above Object when the video is playing. I assume this is just a side effect of passing the video element into Three.js's VideoTexture, but I got around this by adding the video to the DOM and grabbing it with document.getElementById every time. The only place that this doesn't work is with IE support when loading one video after another, as it grabs the wrong video element.
BuildAsyncFlagFromElement.js - URL input is checked here- I found it was easiest to check it just by a simple string suffix match. File input, however, is checked with options.type. It might be better to enforce consistency across both input methods, but this way doesn't involve checking/setting the media type in multiple places (on both input dialogues, from URL parameters, etc). I'm also unsure if you would have wanted to just string match or do something else. The original PR didn't support loading videos from a URL.
FileInput.js - Why are onChange and onLoad both needed again? I feel like the original author removed onLoad because it loaded the video twice. I've commented out this onLoad argument so that it doesn't do that, but I'm not sure if you needed it for something.

Let me know what you think!

Hey there, thanks for all your work on this. I took a look and wrote down some feedback, though I don't think I quite understand how everything is supposed to work yet so nothing is set in stone.

I wouldn't worry too much about IE support, it's just a nice to have. Especially if it means adding messy workarounds. I'd be fine with just defaulting to the white flag and displaying an error message. Something like "Browser feature not supported." I just want to make sure errors are actually being caught and handled so none of these new features causes IE to completely crash, since everything else in FlagWaver is otherwise supported. Also fun fact but the canvas textures were actually implemented before for other reasons before I got rid of it, I might end up bringing it back to address an unrelated an issue.

I would like the video element to be referenced directly instead of calling documentById. On the off chance that I get around to adding multiple flags, I don't want to be relying on a global ID like that. This is also causing some weird issues I noticed when I tested.

Found a bug where multiple video elements are added each time the input is changed. Looks like this happens if you enter a web URL that doesn't resolve.

I believe I was using File.type to check for valid file types, I thought it would be easier than managing a list of file extensions, but I agree it should be consistent.

Maybe there should be separate buildAsyncFlagFromImage and buildAsyncFlagFromVideo functions, then in app/components/FlagGroup.js the type check runs and chooses the correct function. FlagGroup.js can access options.file.type so we wouldn't need to manually pass type property in every setOptions call.

There was a reason for separate onChange/onLoad, though I don't remember exactly what it was. I had to fiddle around with those pesky file inputs quite a bit to get it to work right. Something to do with the file loading being asynchronous and needing to wait for the file to finish loading. I think some parts of the app checks if the url is an empty string to determine if it's a web file input or not.

Nitpicky stuff

Add a line break before/after if statements if there is other code above/below it.
Also do this with const/let declarations
Wondering if it makes sense to rename buildAsyncFlagFromElement to buildAsyncFlagFromHTMLElement
Minor bug: Stop button should reset the video. It seems like it's already doing this, just need to trigger a re-render.
flagwaver/subjects/Flag.js: Instead of having two separate properties this.element and this.isVideo, can this be one property this.textureVideo and have it be null for non-videos.

Thanks for the quick reply on my PR! I've made some changes:

Removed WIP IE support. You're right that it's probably too much of a hassle right now. Plus, I don't think there's anywhere the app can crash using IE (it's just a rendering problem with OpenGL). A warning toast with "Browser feature not supported." now displays.
Whoops! It turns out I (or the original PR, I guess) didn't realize options.texture was something that the Flag had access to on construction. This is always the <video> element (whereas this.mesh.material.map.image is not), which fixes our issue completely.

AnimationModule.js has also changed to reflect this. Expanding to multiple flags in the future shouldn't be as difficult now. getFlag() probably doesn't need to be called on every play() / pause() / step() / reset(), but it's flagGroupModule doesn't exist on construction or init(), and needs to change as the flag changes.

Split buildAsyncFlagFromElement into buildAsyncFlagFromImage and buildAsyncFlagFromVideo and they are now both called from FlagGroup using a cleaner matching function.
I didn't modify this since I'm still unsure of its use and it doesn't seem to do anything but cause problems when it's enabled.

Improved formatting (line breaks!)
The stop button resets the <video>'s position, but I can't seem to trigger a rerender (it's not app.render() or something like app.update(0)?).

Hi, thanks again for your work on this. I'll take a deeper look later tonight but for now I just took a quick look.
src/assets/js/app/components/FlagGroup.js
For isVideo, are you able to check using type.match('video.*')? I believe options.file.type should be available to access here.
What is window.document.documentMode? Is that a browser sniffing test for IE? I would prefer to detect whatever error or missing feature in IE is causing the issue. If that's not possible, let's make it more explicit that this is checking for IE.
Also make sure to always fall back to the default white flag for any kind of error.
This is roughly what I'm thinking.
buildAsyncFlagFromVideo(src, options)
    .then((flag) => {
        if (supportsVideoTexture) {
        // if (!isBrowserIE11) {
            this.updateFlag(flag);
        } else {
            addToast({ ... })

            buildAsyncFlagFromImage(DEFAULT_FLAG_IMAGE_PATH, options)
                .then((flag) => {
                    this.updateFlag(flag);
                });
        }
    });

src/assets/js/flagwaver/app/modules/AnimationModule.js
Can the play/pause/step methods be handled entirely in Flag.js? If not, I still want to make sure that this still works regardless of if there is one or multiple flags. I think there should be something we can use for that...
src/assets/js/flagwaver/helpers/buildAsyncFlagFromVideo.js
The function name here should match the filename buildAsyncFlagFromVideo.js
Also can the supportsVideoTexture/isBrowserIE11 checks I mentioned earlier be handled by the promise in this function? This would be more consistent with how the other errors are handled.
src/assets/js/flagwaver/helpers/buildRectangularFlagFromElement.js
Let's rename this to buildRectangularFlagFromMedia
src/assets/js/flagwaver/subjects/Flag.js
I think THREE.VideoTexture inherits from THREE.Texture so it's not necessary to specify both in {THREE.Texture||THREE.VideoTexture}
Also can you un-nest that first double nested if statement.
misc
Lastly, try wrapping the video re-render call inside as setTimeout(() => {}, 0). It could be that the HTML video element view doesn't refresh immediately so the render call needs to be deferred

Addressed most of the above issues:

FlagGroup.js - The reason I check for both file extension and file type is that options.type isn't set for URL input (I mention this in BuildAsyncFlagFromElement.js in the original PR message). options.file isn't accessible here, since only src and type are set from FlagGroupBar and FlagGroupPanel.

window.document.documentMode is an IE sniffing check, which I've made more clear. I don't think it's possible to check if the texture isn't rendering, since it doesn't throw any errors. I do check this twice in this commit, which may not be ideal (see below).

AnimationModule.js - I don't think these can be handled entirely in Flag.js itself, since the UI only calls play/pause/step in AnimationModule. I've modified this to get all flags from every FlagGroupModules and play/pause/step each one, in case you have multiple flag modules. It would be ideal if all video playback code could be handled in Flag.js, but I don't know if that's completely possible without messier code.
buildAsyncFlagFromVideo.js - You mention that you want the check to be handled inside the Promise- I'm not sure if you'd rather have just one IE check in FlagGroup.js and a generic "Video could not be loaded" message for everything, a specific error message passed in the reject call, or something else. It should be consistent between buildAsyncFlagFromVideo and buildAsyncFlagFromImage, but I don't know about having two IE checks. I could be overthinking it though.

Also oops, forgot to change the function name haha

buildRectangularFlagFromElement.js renamed to buildRectangularFlagFromMedia.js
Flag.js - Fixed.
I still can't get it to work, even with a setTimeout... I'm also unsure how I would trigger a rerender from inside Flag.js, since I don't think it has access to App (I've been trying under AnimationModule's reset() function, calling a reset and setting a timeout for a rerender).

I just took a closer look and here is the rest of my feedback. I think it's mostly minor stuff at this point, this is looking pretty close.
app/components/FileInput.js

I think the type property can be removed in these function calls.
Do you know why window.URL.createObjectURL was added and the reader load event removed? It looks like this is the reason onLoad/onChange was changed in the other components. I would prefer not having to change all of this unless there is a good reason to.

app/components/FlagGroupBar.js and FlagGroupPanel.js

Passing type property here should no longer be necessary
I don't think the changes to handleFilePickerChange would've been necessary if FileInput.js hadn't been changed.

flagwaver/helpers/buildRectangularFlagFromImage and buildRectangularFlagFromVideo

import buildRectangularFlagFromElement... should fit onto one line now that it's not more than 80 char long

flagwaver/helpers/buildRectangularFlagFromElement

Add line break at L19

flageaver/helpers/loadVideo.js

Move video.src - src; below the other properties

flagwaver/subjects/Flag.js

Is it necessary/is there a benefit to checking this.videoElement.paused before calling paused() and play() ?

I see you just posted a another update, thanks! I'll take another look

I see the what you mean about checking the type in FlagGroup.js. I've found a solution to that, just pass fileRecord to FlagGroup. Probably a good idea to make fileRecord a member of Flag.options, but that's a problem for another day.
app/components/FlagGroup.js:
renderModule() {
    const { fileRecord, options, addToast } = this.props;
    const src = options.src || DEFAULT_FLAG_IMAGE_PATH;

    const { file } = fileRecord;
    const isVideo = file && file.type.match('video.*');

    console.log('fileRecord',fileRecord)
    console.log('isVideo',!!isVideo)

app/containers/FlagGroupContainer.js:
const mapStateToProps = state => ({
    fileRecord: state.fileRecord,
    options: state.flagGroup
});

As for the promise error catch, I didn't realize this meant the same function had to handle both types of errors, but this can be solved by having the promise return multiple values.
src/assets/js/flagwaver/helpers/buildAsyncFlagFromVideo.js:
    if (isBrowserIE11) {
        reject({
            flag: buildFlag(options),
            msg: 'Browser feature not supported.'
        });
    }

    loadVideo(
        src,
        (element) => {
            resolve(
                buildRectangularFlagFromMedia(element, options)
            );
        },
        () => {
            reject({
                flag: buildFlag(options),
                msg: 'Video could not be loaded.'
            });
        }
    );

app/components/FlagGroup.js:
        .catch(({ flag, msg }) => {
                this.updateFlag(flag);

                addToast({
                    status: 'error',
                    message: msg
                });
            });

This commit should address all of these.
Some comments:

FlagGroup.js - We still need the extension check in isVideo for internet video, since it doesn't have a file or file type. I put mp4|mov there as some common formats, but I'm not sure if you wanted to add any more or check the URL in a different way (HEAD request for Content-Type?)

DEFAULT_FLAG_IMAGE_PATH isn't the white flag returned by a rejected Promise from buildAsyncFlagFromVIdeo/Image, but I suppose it's okay

Flag.js - The video needs to check if it's paused for the simulate() function, which calls play() every time. It also prevents some console errors from showing if the video is played while it is already playing, or vice versa.

I just realized some if statements are redundant, so I've modified those to just call the play/pause functions

I've made a few edits and addressed some issues. I think this is close to the finish line, but I still want to take some time to review and test before merging.

Still need to do browser testing.
Does the Step button automatically loop when the end of the video is reached? (need to test).
Large videos cause the computer to lag horribly, but I don't think there's much that can be done about that.
Adding a mute button to AnimationControlBar would be a nice to have.

I've started to do some browser testing, and unfortunately found a major bug in Safari (14.1.2) where the entire page will crash when the end of a video is reached.

Awesome work! Unfortunately I don't have access to Safari so I can't do any testing on that front, but it looks like you're handling things well.
You mentioned that large videos cause a lot of lag- wouldn't it be easier on performance to just get the file with URL.createObjectURL instead of FileReader.readAsDataURL? I'm fairly sure readAsDataURL reads the whole file into memory as a base64 encoded string, which is a lot more taxing than generating a URL from the file itself. You might need to revoke the URL later during deconstruction, but it would probably lag less. I'm not sure if there was a problem with this solution other than code cleanliness, but it might be better performance-wise.

I was testing using the web URL input when I noticed the lag. Since the web URL input doesn't go through the FileReader, I don't think changing that will help. For reference, this was the video I used for testing.
If you do end up finding a solution to the lag, I think it would be worth looking into as a separate PR.
For now, I consider the main priority to be ensuring cross browser support. I'm okay with making an exception for IE, but not if multiple browsers are having trouble. The Safari bug is the only major bug I'm currently aware of, but I also haven't done a lot of testing yet.

Just curious, any updates over the past few months? I feel like this branch is pretty stable.
I was able to get access to a Mac with Safari (14.1.2) through a friend, but I couldn't seem to reproduce the crashing error you were experiencing with Safari... the video seemed to play back fine and repeat perfectly at the end without issue. Were you using a local file? (perhaps it crashed trying to reload the whole video, which is a different issue?)
I've tested other browsers and they seem to work perfectly (Chrome 101.0.4951.67, Edge 101.0.1210.53, Firefox 101.0), along with the mobile devices I own (iPad 15.4.1: Safari, Chrome 102.0.5005.67, Android 12: Chrome 101.0.4951.61)
Unfortunately I don't have any older versions of these browsers/OSes to test, but it seems to work relatively flawlessly for me.

Hi there,

I tested on Safari 15.1 for MacOS 12.0.1
I used this video after first downloading it to my computer as an .mp4

I looked at it again today and am still seeing the same freezing issue, BUT I noticed that if I let my computer idle for several minutes, the animation will eventually start up again and keep going. Which is still unbelievably bad for an 8 second video, but it's not completely stopping like I had originally believed. Let me know if you are seeing the same.
Safari in general just seems to have some pretty terrible performance. From what I can gather, newer versions of Safari have some kind of issue that is affecting all WebGL apps. So I'm thinking that it may not be an actual issue with the code or the video, though I'm not quite sure how to handle this type of issue. What really needs to happen is for Apple/Webkit to fix their broken browser :/
I'll take your word that it works fine in the other browsers, it looks like you got a decent variety of browsers/devices in your tests, so thank you for that. We can mark that complete at least.

Update: Some good news and bad news.
I think I found a band-aid for the Safari problem. Not a real solution, just something to make the app at least barely usable until Webkit fixes their WebGL problems.
It's quite simple. FlagWaver already caps the number of max updates per frame to 90. I can just set this to a smaller number like, 2. Now Safari can maintain a stable rate at a speedy 7 FPS 😂😂😂 ...a significant improvement over 0 FPS!
Bad news is I did some more browser testing and found another bug. In Opera, the step forward button doesn't work properly.

I believe this PR is in good shape and have merged it to the dev branch. There are a few things I wish to take care of before I deploy it however.

This feature is now live as of v2.1.0.


Expose new line ending option in VS

Expose the line ending option in VS, as discussed in Issue #1693 and PR #2921

@paulvanbrenk is this about project property pages?
Is so, and since you brought this up :), can you list the options we need to show? We have not touched the project property pages in a while and we have added a lot of new flags since. It would be great if you can look through the list of flags and see if we need to expose them or not? Or possibly expose all flags.

Not necessarily, this is mostly about the msbuild properties, since I would argue this is an advanced feature.

Ahh.. If this is the case we need another issue for adding --rootDir, --noEmitHelpers as well as support for --module umd and --module system.

I'll make a list and update this bug.

I have the list covered in #3034


Support connecting signals for a Builder

There is no means of connecting signal handler functions to named signals in a builder description. This was initially mentioned in #9.
Use of gtk_builder_connect_signals_full is probably the way to go.
gotk3's approach in https://github.com/gotk3/gotk3/pull/9 provides some inspiration.
Unlike gobbi's normally type-safe approach to connecting signal handlers this is probably going to have to connect the handler functions without regard to type-safety.

@sunshine69 I've just open #11 which implements a new api for connecting signals. I'd appreciate it if you could take a look at the documentation and see if the api looks reasonable to you.
I realise from https://github.com/gotk3/gotk3/issues/457 that you're unlikely to use this this functionality, so I understand if you don't want to spend too much time on it.


v0.2.2 breaks search

I rebuilt my site with the latest version (v0.2.2) and the search didn't seem to work. I changed my Gemfile.lock to build it with v0.2.1 and the search started working again. I changed it back, rebuilt and it was broken again.
I haven't had time to look into what changes were made, so I will update this when I have some time this weekend.

@harper357 please try v0.2.3 and let me know if you're still having problems.

I tried building a new site with v0.2.3 and it worked.
I uninstalled just-the-docs from my old site then reinstalled it and the search still wasn't working. It must be something I customized that I will have to debug. I will close this.


__has_feature may not be accurate

For example, Qt will define this macro even if it was not defined.
/usr/include/qt5/QtCore/qcompilerdetection.h:516:#ifndef __has_feature
/usr/include/qt5/QtCore/qcompilerdetection.h:517:#  define __has_feature(x)             0
/usr/include/qt5/QtCore/qcompilerdetection.h-518-#endif

And looks like gcc version 9.3.0 (Gentoo 9.3.0-r1 p3) does not have __has_feature pre-defined.
So if immer headers are included after Qt headers, immer will deem that there is no exception supported, even if I use -fexceptions.
I guess we should at least allow user to always use exceptions (which is important for things like lager::lenses::at because it uses exception to fill in the optional).

@chuim, what do you think the best solution is here?

Thanks for looping me in. I have no experience on cross compiler support so I have no suggestion to propose. I can try to research a solution once I'm back from the holidays, early next year.

Cool, thanks!

GCC defines the __cpp_exceptions symbol if exceptions are enabled [1, 2]. I don't have a GCC setup I work with so I'd rather ask someone else to try updating the exception detection code in immer/config.hpp to also check for it.
Also, that snippet of code from Qt seems odd as it decides to override that macro with an always-false implementation what might mislead decisions from other macros in the code.

GCC defines the __cpp_exceptions symbol if exceptions are enabled [1, 2]. I don't have a GCC setup I work with so I'd rather ask someone else to try updating the exception detection code in immer/config.hpp to also check for it.
Also, that snippet of code from Qt seems odd as it decides to override that macro with an always-false implementation what might mislead decisions from other macros in the code.

Hi @chuim, Thanks for investigating the issue!
Regarding the GCC setup, if you use the nix-shell as advised in the project README it is super easy to get one without polluting your environment. If you are on a Linux the default setup will use a GCC, on Mac you can try nix-shell --argstr toolchain gnu-9.

Hi @chuim, Thanks for investigating the issue!
Regarding the GCC setup, if you use the nix-shell as advised in the project README it is super easy to get one without polluting your environment. If you are on a Linux the default setup will use a GCC, on Mac you can try nix-shell --argstr toolchain gnu-9.


Products card looks different

Description

Make sure the products card are looking the same as the design that was provided by the UI/UX team.
Acess the figma file here

Screenshots
How it should look :

How it looks :

Additional information
No response
🥦 Browser
Brave
👀 Have you checked if this issue has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Contributing Guidelines?

[X] I have read the Contributing Guidelines

Are you willing to work on this issue ?
No response

Can you assign me this issue?

Go for it @Kishore-abhimanyu !

Can you assign me this issue

Can you assign me this issue

Always check this before asking for assignment.

@Kishore-abhimanyu Are you working on this ?

Will create a new pr with PR details soon


Housekeeping: FIGMA

As a frontend developer
I want to have an up to date design document
Because it makes planning, communicating and actually developing the frontend easier.
NOTES:

This is a repeating housekeeper task.

Acceptance Criteria:

[ ] There are no major deviations between the design document and the actual frontend.

awaiting review from @Straczi

Looks good to me, can't spot any significant differences between the current product and the figma demos :)


setting request headers

Hey,
I'm having difficulties setting headers in my requests (browser.get('/path');)
Is there a way to do so ?
Thanks !

Custom headers would be more than nice. :+1:

+1 using headers to enable things in non-prod env is very common. Please add this feature.

+1
In my case the browsers in prod env have to go through proxy and authentication servers, which set a number of HTTP headers. Currently our solution is to proxy all requests from protractor through connect.

+1; We use SSO / Kerberos and I would like to imitate another user than me. Hence I need to set the Authorization header.

+1 on adding headers

+1 for modifying headers

+1 on adding headers. In my case, I want to be able to control different responses from the same user action coming back from a mocked API.

+1 We don't want to log allways... we have a test case testing it. We need to bypass them. We want setting headers!

+1

We'd need webdriver to implement this feature, which they've already rejected doing.  You could try contacting them on their github but I don't think it will get you anywhere.

The selenium issue you linked does not seem to be the same thing we are asking for. People in that thread are asking to be able to test response headers going to protractor. We are asking to be able to set request headers coming from protractor to the server at the start of the test. This is to do things like simulate a logged in user w/o having to take the extra time to explicitly log in every time. Or in our case to bypass an annoying resource-consuming overlay which sets a cookie when the user closes it not to show up again.

Sorry, sounds like I linked to the wrong issue.  This has been raised with WebDriver though and they have rejected it.  @juliemr is probably more able to find the exact issue, but you could just make a new issue on their github

+1 for having the possibility to set request headers
@jackspaniel for setting a cookie you can use the browser.manage().addCookie() method.

+1 for setting/modifying request headers.
BTW, why this issue has been closed ?

It was closed because we'd need webdriver to implement this feature, which they've already rejected doing.  You could try contacting them on their github but I don't think it will get you anywhere.

To add, though it does not seem possible to arbitrarily set request headers, if you just want to set cookies, that is at least possible with browser.manage().addCookie('foo', 'bar', '/')

It's been a while but if I recall correctly that doesn't actually send a cookie to the server with the request - it just registers in the browser that it received a cookie.

If you support having the ability to set request headers on tests, please go to this ticket and add +1 to the request. I just created it as it inherently has and is affecting the way i have to create tests.
https://github.com/SeleniumHQ/selenium/issues/2131

+1 for modheaders - we use modheaders to access our app during a release... so this would make that easy.

+1

+1

workaround, just put the "header" in the url
GET localhost:6969?my_headers={"a":"b"}

our app uses two-factor authentication and sends you an sms. It would be hard (impossible?) to login as user would normally login.

our app uses two-factor authentication and sends you an sms. It would be hard (impossible?) to login as user would normally login.

Just use Twilio number (they have even free one). Using Twilio API get an sms with the code. That's not difficult. https://www.twilio.com/docs/usage/api


Avoid race conditions in raft-kv-memstore example

The raft-kv-memstore example made some inaccurate assumptions, which this PR aims to resolve by making locking more strict and thereby eliminating the following race conditions:

The fact that build_snapshot would not be invoked concurrently by the OpenRaft runtime is currently correct, but might not remain true in the future. If this were to happen in the future, this could constitute a race condition.
build_snapshot and install_snapshot might be invoked concurrently on a follower that has log compaction enabled. This could also constitute a race condition, even under the current runtime.

Also in this PR:

Wrote a test for snapshot transfer between state machine stores, since this was missing from the test suite and the PR alters the snapshotting logic
Use AtomicU64 rather than Mutex for snapshot_idx

Checklist

[x] Updated guide with pertinent info (may not always apply).
[x] Squash down commits to one or two logical commits which clearly describe the work you've done.
[x] Unittest is a friend:)

This change is

BTW, there are several lint errors in the CI.

Ran cargo fmt now, hopefully that fixes it.


Bug when deleting element of the array in React 18

It works perfectly well with some sipmle element like P or Li. But when I tried to implement it to more complex components with style and structure like these:
.squares {
margin-top: 2rem;
display: grid;
grid-template-columns: repeat(4, 1fr);
gap: 2.5rem;
}
.square {
padding: 1rem;
border: 0.2rem solid $border-grey;
border-radius: 0.3rem;
position: relative;
display: grid;
grid-template-rows: repeat(7, 1fr);
row-gap: 1rem;
z-index : //calculated from JSX props
}

  {squares.map(square => {
    return (

    )
  }}

It works ok when I try to add new element to array - animation works properly, but only for the last element. If I add a new element to the start of the array it animate only the last element on rerender.
And I got simmilar bug when try to delete any element from array - it looks that only the last element moves and has animation on rerender.

@Alexey-Timakov hi! Have you managed to solve this issue? I have the similar issue and don't know what the cause of the problem

@Alexey-Timakov hi! Have you managed to solve this issue? I have the similar issue and don't know what the cause of the problem

Unfortunately no, i didn't. Just decided to not use it at all

try adding key to each div in squares.map(...)


Added support for nested collections.

Fixes issue #45.

@clegoz thanks for the contribution.  I'll review over the next week or so and likely push a release with this shortly thereafter.