fix: broken decoding

Thanks to very important find by @mattsse. All enrs with for example "eth2" kv-pair have failed decoding.

Fixes decoding of enr with an custom kv pair

A few things to note here.
The bug identified here relates solely to applications that are attempting to RLP-encode lists as ENR values and decode them. ENRs currently cannot decode values that are stored as RLP lists or recursive RLP lists.
The specific key you are referring to, "eth2" is used in Ethereum's consensus protocol is unaffected by this, because that protocol doesn't store ENR values as RLP lists. The protocol always uses straight bytes encoded as ssz.
This PR as it stands actually breaks the tests and prevents decoding of other fields that are not RLP lists. In its current form we cannot merge this PR.
I've tried pushing fixes to this branch, but appear not to have permissions. As this appears urgent for your use case, i'm going to open other PR with the changes.


Unable to load shared library ... serialport_native.node

I'm flummoxed.  I think I have everything set up correctly, but I keep getting an error when I try to load the serialport module.  (I'm using Ubuntu 11.10)
jeff@mega:/storage/projects/tempbrain$ npm rebuild serialport

<EMAIL_ADDRESS>install /storage/projects/tempbrain/node_modules/serialport
cd serialport_native;node-waf configure build;cp build/Release/serialport_native.node ../

Checking for program g++ or c++          : /usr/bin/g++
Checking for program cpp                 : /usr/bin/cpp
Checking for program ar                  : /usr/bin/ar
Checking for program ranlib              : /usr/bin/ranlib
Checking for g++                         : ok
Checking for node path                   : ok /home/jeff/local/lib
Checking for node prefix                 : ok /home/jeff/local
'configure' finished successfully (0.213s)
Waf: Entering directory /storage/projects/tempbrain/node_modules/serialport/serialport_native/build' Waf: Leaving directory /storage/projects/tempbrain/node_modules/serialport/serialport_native/build'
'build' finished successfully (0.054s)
<EMAIL_ADDRESS>/storage/projects/tempbrain/node_modules/serialport
jeff@mega:/storage/projects/tempbrain$ cat barebones.js
var serialport = require("serialport");
jeff@mega:/storage/projects/tempbrain$ node barebones.js
node.js:201
throw e; // process.nextTick error, or 'error' event on first tick
^
Error: Unable to load shared library /storage/projects/tempbrain/node_modules/serialport/serialport_native.node
at Object..node (module.js:463:11)
at Module.load (module.js:351:31)
at Function._load (module.js:310:12)
at Module.require (module.js:357:17)
at require (module.js:368:17)
at Object. (/storage/projects/tempbrain/node_modules/serialport/serialport.js:11:28)
at Module._compile (module.js:432:26)
at Object..js (module.js:450:10)
at Module.load (module.js:351:31)
at Function._load (module.js:310:12)
jeff@mega:/storage/projects/tempbrain$ node --version
v0.6.1
jeff@mega:/storage/projects/tempbrain$ npm --version
1.1.1
I'm not sure what else to try.  Is there some other part of my environment that's not set up correctly?  Any ideas would be greatly appreciated.

The drive I was using was mounted noexec... argh.  Sorry if I wasted anyone's time.

I'm having the same issue on Mac OS 10.7.3. Node 0.6.11. Npm 1.1.1. Any ideas? Exact same npm install output. Builds successfully and I can see the serialport_native.node in the serialport directory.

Try creating a script in that directory and then running it to verify that you have run permissions there.  That's the problem I was having.

Thanks for a quick response!
I created a quick one-line test file in 'node_modules/serialport'. Permissions seem to be fine.
var serialport_native = require('./serialport_native');
Same error. I'm running in 64-bit mode. Could that be the issue? Restarted Lion in 32-bit, rebuilt the package and same error.

Downgraded to 0.6.10 version of node (npm 1.1.0-3) and it all works smoothly again.

FYI, I just upgraded to node 0.6.11 (npm 1.1.1) and don't have any problems.

Same problem with node 6.11
11:06:54 > uname -a
Darwin Woodman.local 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64
11:06:07 > node --version
v0.6.11
11:06:09 > npm --version
1.1.1
11:06:12 > npm install serialport@latest
npm http GET https://registry.npmjs.org/serialport
npm http 304 https://registry.npmjs.org/serialport

><EMAIL_ADDRESS>install /Users/sebastian/Dropbox/Coding/node_modules/serialport
> cd serialport_native;node-waf configure build;cp build/Release/serialport_native.node ../

Checking for program g++ or c++          : /usr/bin/g++
Checking for program cpp                 : /usr/bin/cpp
Checking for program ar                  : /usr/bin/ar
Checking for program ranlib              : /usr/bin/ranlib
Checking for g++                         : ok
Checking for node path                   : ok /usr/local/lib/node
Checking for node prefix                 : ok /usr/local
'configure' finished successfully (0.114s)
Waf: Entering directory `/Users/sebastian/Dropbox/Coding/node_modules/serialport/serialport_native/build'
[1/2] cxx: serialport_native.cc -> build/Release/serialport_native_1.o
[2/2] cxx_link: build/Release/serialport_native_1.o -> build/Release/serialport_native.node
Waf: Leaving directory `/Users/sebastian/Dropbox/Coding/node_modules/serialport/serialport_native/build'
'build' finished successfully (0.535s)
<EMAIL_ADDRESS>./node_modules/serialport
11:06:22 > node
> var serialport = require("serialport");
Error: Unable to load shared library /Users/sebastian/Dropbox/Coding/node_modules/serialport/serialport_native.node
    at Object..node (module.js:472:11)
    at Module.load (module.js:348:31)
    at Function._load (module.js:308:12)
    at Module.require (module.js:354:17)
    at require (module.js:370:17)
    at Object.<anonymous> (/Users/sebastian/Dropbox/Coding/node_modules/serialport/serialport.js:11:28)
    at Module._compile (module.js:441:26)
    at Object..js (module.js:459:10)
    at Module.load (module.js:348:31)
    at Function._load (module.js:308:12)

Agreed.  Same issue here.  Not a permissions problem, something is broken for node 0.6.11 under os/x (at least).
fresh:shack garth$ uname -a
Darwin fresh.local 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64
fresh:shack garth$ node --version
v0.6.11
fresh:shack garth$ node

require('serialport')
Error: Unable to load shared library /Users/garth/code/shack/node_modules/serialport/serialport_native.node

I'm having the identical issue with node 0.6.11 under Snow Leopard but with the "midi" library (I found this open issue when Googling the same error ;-)). So it's probably not node-serialport specific.

It will be fixed in the 0.6.12 node .pkg installer by distributing a fat binary.


The api keys wont load

I get a PHP error when I want to pay my cart:
[2020-04-02 15:26:32] request.CRITICAL: Uncaught PHP Exception TypeError: "Return value of Kiener\MolliePayments\Setting\MollieSettingStruct::getTestApiKey() must be of the type string, null returned" at /var/www/custom/plugins/MolliePayments/src/Setting/MollieSettingStruct.php line 47 {"exception":"[object] (TypeError(code: 0): Return value of Kiener\\MolliePayments\\Setting\\MollieSettingStruct::getTestApiKey() must be of the type string, null returned at /var/www/custom/plugins/MolliePayments/src/Setting/MollieSettingStruct.php:47)"} []
I've entered both keys on the /admin#/sw/plugin/settings/MolliePayments page.
Am I doing somthing wrong?

I can confirm that this is also occuring in our setup.
Shopware 6.1.5

It broke due to this commit: https://github.com/mollie/Shopware6/commit/0321b296b912874ae8714b2a19d0b0e85e278868#diff-4d4c2f990fa8293eb30b3d45a5c1b91eR78.
The sales_channel_id is empty by default.

The sales channel id can be null in the current release. Thanks for the input 😊

I still have this problem in 1.0.7 in case I don't explicit set the settings for the specific sales channel.


Use the official aws-sdk instead of jet3t

Fixes #4289.
In this PR, the version of the below libraries are changed.

Hadoop => 2.9.0 to match library versions
AWS-SDK => 1.11.29 (https://github.com/druid-io/druid/issues/4456)
Jackson => 2.8.10
HttpClient => 4.5.3 (https://github.com/druid-io/druid/blob/master/common/pom.xml#L160-L169, https://issues.apache.org/jira/browse/HTTPCLIENT-1727)

I've done the below tests.

Indexing

Hadoop indexing with Hadoop 2.9.0
Hadoop indexing with Hadoop 3.0
Local indexing

With prefetch
Without prefetch

insert-segment-to-db tool: this tool finds segments under the given directory of deep storage
Moving segments with moveTask
Killing segments with killTask

This change is

@jihoonson does this means we are dropping support for Hadoop < 2.9.0 ?

@b-slim yes. I tested with 2.7.5, but it didn't work because of the mismatched version of aws-sdk library.

I added WIP label because this PR needs more testing on larger clusters. My tests were done only in my laptop.

Also, there are some TODO notes I left which should be addressed.

Is it possible to keep the same version of aws-java-sdk as we have now, and do this patch without changing the Hadoop version? Or is there a reason they both need to be upgraded?
I am just asking because people struggle with upgrades every time we change the Hadoop version, so I was hoping it wouldn't be necessary (I was hoping the next upgrade after 2.7.3 would be to 3.x, sometime in the future after its deployment becomes more widespread).

@gianm yes, there is a reason. To use aws-java-sdk, we need to match its version to that used by Hadoop as well as other libraries' versions like jackson, httpclient, and so on. (Unfortunately, I lost my note listing the libraries needing version change.) From my testing, 2.9.0 was the oldest working version.

@gianm yes, there is a reason. To use aws-java-sdk, we need to match its version to that used by Hadoop as well as other libraries' versions like jackson, httpclient, and so on. (Unfortunately, I lost my note listing the libraries needing version change.) From my testing, 2.9.0 was the oldest working version.

I don't understand why we need to use aws-java-sdk from Hadoop at all. I thought that when we run Hadoop jobs, we use its FileSystem implementations both for reading (possibly from S3) and writing (to a possibly S3 deep storage) and do not call jets3t directly. Could we solve this by excluding the java sdk from the main classloader when we are running Hadoop jobs, meaning we only load the one provided in hadoop-dependencies?

I don't understand why we need to use aws-java-sdk from Hadoop at all.

This should say: I don't understand why we need to use Druid's version of aws-java-sdk from Hadoop at all.

FYI am all in to move to 3.0 but I really think we should be able to support hadoop 2.7 line (if possible) 2.9 is not really popular and I don't think it will be, the most likely, the users will stick with 2.7 line or do the leap and move to 3.0.
@jihoonson can you please elaborate around https://github.com/druid-io/druid/pull/5382#issuecomment-365062725 ? what is the test that failed and what kind of dependency is breaking? Thanks

I don't understand why we need to use aws-java-sdk from Hadoop at all. I thought that when we run Hadoop jobs, we use its FileSystem implementations both for reading (possibly from S3) and writing (to a possibly S3 deep storage) and do not call jets3t directly.

Yes, this is correct. What I meant is the old implementation of S3AFileSystem uses some deprecated APIs of aws-sdk which are not in the recent versions. This is not just an issue of aws-sdk, but also some other libraries like jackson.

@b-slim when I tested with hadoop 2.7.5, I could see the below error.
2018-02-13T05:02:23,061 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_lineitem_2018-02-13T05:02:00.954Z, type=index_hadoop, dataSource=lineitem}]
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:222) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:184) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:450) [druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:422) [druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:219) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	... 7 more
Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287) ~[?:?]
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667) ~[?:?]
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93) ~[?:?]
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2701) ~[?:?]
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2683) ~[?:?]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:372) ~[?:?]
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) ~[?:?]
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:500) ~[?:?]
	at org.apache.hadoop.mapreduce.lib.input.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:110) ~[?:?]
	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301) ~[?:?]
	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318) ~[?:?]
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) ~[?:?]
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) ~[?:?]
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_161]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) ~[?:?]
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) ~[?:?]
	at io.druid.indexer.DetermineHashedPartitionsJob.run(DetermineHashedPartitionsJob.java:119) ~[druid-indexing-hadoop-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexer.JobHelper.runJobs(JobHelper.java:362) ~[druid-indexing-hadoop-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:325) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:219) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	... 7 more

This error means that S3AFileSystem of 2.7.5 is using a constructor of TransferManager which is not in the recent version of aws-sdk. So, I added an older version (1.7.4) of aws-sdk to the classpath, and changed the dependency order in the classpath for aws-sdk 1.7.4 to be appeared earlier. This caused another error shown below.
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
2018-02-12T23:37:31,308 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_lineitem_2018-02-12T23:36:37.550Z, type=index_hadoop, dataSource=lineitem}]
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:231) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:184) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:450) [druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:422) [druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:228) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	... 7 more
Caused by: io.druid.java.util.common.ISE: Job[class io.druid.indexer.DetermineHashedPartitionsJob] failed!
	at io.druid.indexer.JobHelper.runJobs(JobHelper.java:384) ~[druid-indexing-hadoop-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:325) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:228) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	... 7 more

This error came from the mismatched version of Jackson, so I also added an older version of Jackson to the classpath as well and adjusted the order. This causes the below error.
2018-02-12T23:56:18,757 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_lineitem_2018-02-12T23:56:12.573Z, type=index_hadoop, dataSource=lineitem}]
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:231) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:184) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:450) [druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:422) [druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:228) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	... 7 more
Caused by: java.lang.ExceptionInInitializerError
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:311) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:228) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	... 7 more
Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error in custom provider, java.lang.VerifyError: Cannot inherit from final class
  at io.druid.jackson.JacksonModule.jsonMapper(JacksonModule.java:46)
  at io.druid.jackson.JacksonModule.jsonMapper(JacksonModule.java:46)
  while locating com.fasterxml.jackson.databind.ObjectMapper annotated with interface io.druid.guice.annotations.Json
  while locating com.fasterxml.jackson.databind.ObjectMapper
    for the 1st parameter of io.druid.guice.JsonConfigurator.<init>(JsonConfigurator.java:67)
  at io.druid.guice.ConfigModule.configure(ConfigModule.java:40)
  while locating io.druid.guice.JsonConfigurator
    for the 2nd parameter of io.druid.guice.JsonConfigProvider.inject(JsonConfigProvider.java:188)
  at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:133)

2) Error in custom provider, java.lang.VerifyError: Cannot inherit from final class
  at io.druid.jackson.JacksonModule.jsonMapper(JacksonModule.java:46)
  at io.druid.jackson.JacksonModule.jsonMapper(JacksonModule.java:46)
  while locating com.fasterxml.jackson.databind.ObjectMapper annotated with interface io.druid.guice.annotations.Json
  while locating com.fasterxml.jackson.databind.ObjectMapper
    for the 1st parameter of io.druid.guice.JsonConfigurator.<init>(JsonConfigurator.java:67)
  at io.druid.guice.ConfigModule.configure(ConfigModule.java:40)
  while locating io.druid.guice.JsonConfigurator
    for the 2nd parameter of io.druid.guice.JsonConfigProvider.inject(JsonConfigProvider.java:188)
  at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:133)

2 errors
	at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:176) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) ~[guice-4.1.0.jar:?]
	at com.google.inject.Guice.createInjector(Guice.java:99) ~[guice-4.1.0.jar:?]
	at com.google.inject.Guice.createInjector(Guice.java:73) ~[guice-4.1.0.jar:?]
	at io.druid.guice.GuiceInjectors.makeStartupInjector(GuiceInjectors.java:60) ~[druid-processing-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:106) ~[druid-indexing-hadoop-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:311) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:228) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	... 7 more
Caused by: java.lang.VerifyError: Cannot inherit from final class
	at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_161]
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_161]
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_161]
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) ~[?:1.8.0_161]
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73) ~[?:1.8.0_161]
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368) ~[?:1.8.0_161]
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362) ~[?:1.8.0_161]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361) ~[?:1.8.0_161]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_161]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_161]
	at com.fasterxml.jackson.datatype.guava.GuavaModule.setupModule(GuavaModule.java:55) ~[jackson-datatype-guava-2.8.10.jar:2.8.10]
	at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:524) ~[jackson-databind-2.8.10.jar:2.8.10]
	at io.druid.jackson.DefaultObjectMapper.<init>(DefaultObjectMapper.java:47) ~[druid-processing-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.jackson.DefaultObjectMapper.<init>(DefaultObjectMapper.java:35) ~[druid-processing-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.jackson.JacksonModule.jsonMapper(JacksonModule.java:46) ~[druid-processing-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.jackson.JacksonModule$$FastClassByGuice$$fa1bdb70.invoke(<generated>) ~[druid-processing-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:82) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:93) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:80) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:80) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.Initializer$InjectableReference.get(Initializer.java:223) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.Initializer.injectAll(Initializer.java:132) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:174) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) ~[guice-4.1.0.jar:?]
	at com.google.inject.Guice.createInjector(Guice.java:99) ~[guice-4.1.0.jar:?]
	at com.google.inject.Guice.createInjector(Guice.java:73) ~[guice-4.1.0.jar:?]
	at io.druid.guice.GuiceInjectors.makeStartupInjector(GuiceInjectors.java:60) ~[druid-processing-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:106) ~[druid-indexing-hadoop-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:311) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:228) ~[druid-indexing-service-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
	... 7 more

I guess the last error is because of the mismatched Jackson version when the peon tries to initialize some variables using Guice injection which means a newer version of Jackson is required. I'm not sure how the second and the third errors can be fixed (maybe by shading?).
I understand what @gianm and @b-slim are concerned with. I'm doing further testing to figure out what's the best way. I'll post the result here once my testing is done.

@jihoonson does this deprecate https://github.com/druid-io/druid/blob/druid-0.12.0-rc1/pom.xml#L1170-L1176 the spark2 profile?

Related: https://github.com/druid-io/druid/pull/5288

Please add the removal of AWSSessionCredentialsAdapter to the master PR comment with justification (i.e. does the aws sdk do this already?)

@jihoonson does this deprecate https://github.com/druid-io/druid/blob/druid-0.12.0-rc1/pom.xml#L1170-L1176 the spark2 profile?

Yes, maybe. I didn't test yet.

Please add the removal of AWSSessionCredentialsAdapter to the master PR comment with justification (i.e. does the aws sdk do this already?)

AWSSessionCredentialsAdapter was used by only S3StorageDruidModule to pass credentials from  aws-java-sdk's AWSCredentialsProvider to jet3t's Rest3Client. We don't need it anymore.

@drcrallen thanks! I'm doing further tests to support older versions of Hadoop (especially 2.7 line) as well as resolving my TODOs. I'll update this PR once it's ready.

I am ok with this solution in principle. How would we do it though?

What is the tolerance for suggesting hadoop 2.9 or higher?

My sense is that Hadoop 2.9.0 was only released in Nov 2017 so it's too new to make the minimum version. People don't tend to update their Hadoop clusters very often.

I am ok with this solution in principle. How would we do it though?

We may fork the hadoop-aws module as a project under druid-io, and make some changes to support hadoop 2.7 line. And we may release the customized hadoop-aws.jar along with our next release (0.13). Does it make sense?

@drcrallen would you please answer for my question here?

@jihoonson answered

@gianm We may fork the hadoop-aws module as a project under druid-io, and make some changes to support hadoop 2.7 line. And we may release the customized hadoop-aws.jar along with our next release (0.13). Does it make sense?

Will this work with a remote cluster? I thought - usually - the remote cluster prefers its own Hadoop jars rather than any Hadoop jars that user code (i.e. Druid) might ship. I guess it's fine since on the remote cluster, Druid probably won't be using aws-java-sdk directly, so there won't be an opportunity to have problems there?

@jihoonson thanks for doing this,  are you planning on shading/relocation as well? can you please elaborate more on what part will be forking if you have to?

I've done some more tests and updated the PR description. One thing related to Hadoop support is that this PR works with Hadoop 2.7.5 if S3NFileSystem is set to use. The below is a quote from Hadoop Wiki.

Unmainteained: S3N FileSystem (URI scheme: s3n://)
S3N is the S3 Client for Hadoop 2.6 and earlier. From Hadoop 2.7+, switch to s3a
A native filesystem for reading and writing regular files on S3.With this filesystem is that you can access files on S3 that were written with other tools. Conversely, other tools can access files written using Hadoop. The S3N code is stable and widely used, but is not adding any new features (which is why it remains stable).
S3N requires a compatible version of the jets3t JAR on the classpath.
Since Hadoop 2.6, all work on S3 integration has been with S3A. S3N is not maintained except for security risks —this helps guarantee security. Most bug reports against S3N will be closed as WONTFIX and the text "use S3A". Please switch to S3A if you can -and do try it before filing bug reports against S3N.

If only new features are not being added to S3N, I think it's fine. But, if there are some bugs, using S3N for Hadoop 2.7 won't work. @b-slim do you have any idea about this?

I found that the current master branch also doesn't work with Hadoop 2.7.5 S3AFileSystem. I think this issue should be handled separately.

S3A issue relates to #4841.

Ready for review.

@drcrallen any other comments?

@jihoonson just https://github.com/druid-io/druid/pull/5382/files#r175774341 similar to https://github.com/druid-io/druid/blob/89215382510a6eceac898d14022b76a065fd7cdf/aws-common/src/main/java/io/druid/common/aws/AWSCredentialsConfig.java#L34

@drcrallen ok, I agree it's needed. But, I would like to do in a follow-up PR because this PR easily conflicts with others. If you think its' fine, then I'll raise an issue for it. If not, I'll separate the proxy configuration from this PR and make another one. What do you think?

@drcrallen thanks. Raised https://github.com/druid-io/druid/issues/5507.

@jon-wei thanks for the review. Addressed your comments.

I'm going to merge this PR shortly. @b-slim @drcrallen @jon-wei thanks for the review!

@jihoonson i see this PR is merged so what is the story now? are we dropping support for Hadoop < 2.9 if, that is the case did we have a round of vote at mailing list? is this suppose to be 0.13.0 ?

@b-slim we still support Hadoop 2.7 line. I tested Hadoop 2.7.3 and 2.7.5. When Hadoop 2.7 is used with s3 deep storage together, S3N still works but S3A doesn't work which is same for the master before this PR. To use S3A, they need a custom hadoop-aws which shades all conflicted libraries like aws, guava, and jackson.

@b-slim

-First as per this thread we are suppose to have support from 2.6.0 and higher, https://groups.google.com/forum/#!topic/druid-development/vFjR0pdq83U. So am not sure if you have done testing with that as well or we are dropping support for any version less than 2.7.3?

I tested Apache Hadoop 2.6.0, 2.7.1, 2.7.3, 2.7.5, 2.8.3, 2.9.0, 3.0.0, Cloudera 5.7.0, and HDP 5.7. But, it would be great if you can test more Apache Hadoop and HDP versions as well.

-Second, am wondering, have you tested that with HDFS as deep storage or only S3?

I've tested four cases when deep storage is HDFS, S3A via HDFS, S3N via HDFS, and direct S3.

-Third if it works with 2.7.X why changing the version Hadoop to 2.8.3?

Well, at first, I wanted to use Hadoop 2.9 to use more recent libraries especially Jackson. But, it broke the support for Hadoop 2.7, so I needed to change it again. IMO, it doesn't make sense to use Hadoop 2.7.3 as our default Hadoop version because we don't support the whole features for that version like S3A.


Use zero redundancy or zero replica shards for single node ES cluster

Follows https://github.com/jaegertracing/jaeger-operator/pull/531#discussion_r304010332

done in #539


[Feature request]: allow the default model to be set with env variables

What do you need?
allow the default model to be set with env variables

Done

Added as —changeDefaultModel. All the available models are listed in —listmodels. It’s also added to —setup


create_communicator_grouped2 may trigger uninit value memory issue(randomly crash) when you train more iterations.

Container:
nvcr.io/nvidia/pytorch:24.05-py3

Machine:
x86 CPU with A100 node
Reproduce:
python -m torch.distributed.run --nproc-per-node=2 examples/pytorch/comm_gemm_overlap/ln_mlp_with_overlap.py --num-iters=1000

It'll probably crash or not because at line 102, the operator= will trigger destructor to delete the old value std::function of _alloc_copy_allgather, which is actually an uninit value.
see also the definition of struct communicator
struct communicator {
...
std::function<void(void **, void *, size_t, ExtComm)> _alloc_copy_allgather; //will not be initialized by malloc
std::function<void(ExtComm)> _barrier; //will not be initialized by malloc
std::function<void(void *)> _free; //will not be initialized by malloc

Hope this hint helps the progress.

The commit on this fork will fix this.
https://github.com/denera/TransformerEngine/commit/7a9522bdbbe28d2682567ea450f10d87cc68d03a

@anderson101866 This should be fixed now in TE/main as of PR #1087. Could you check and close the issue if resolved?

Yes, it’s resolved. Thanks for your greatly help!


Update Eran Stiller's Bio

Docs Build status updates of commit c18ae0f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

rd/eran-stiller.yml
:white_check_mark:Succeeded
View

rd/media/profiles/eran-stiller.jpg
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

@estiller
Your change has been merged and published to RD website. Please check it.

Thanks! Looks good.


Update set-up-your-development-environment.md

Q
A

content fix?
yes

New article?
no

Related issues?
fixes #X, partially #Y, mentioned in #Z

What's in this Pull Request?
Fixed link to required node version so it points to correct url

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/spfx/set-up-your-development-environment.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Since we did add support for Node 8 as part of the 1.4.1 release, will close this. This PR was pushed just before the 1.4.1 release, so completely understand the suggested edit. Will close this, for now, due obvious reasons.


Feature following

The way the following_list view was done was a bit repetitive. We require just a single view.

My bad, i'm sorry i did not delete it before sending
On Tue, Nov 17, 2015 at 10:54 PM, Chidiebere Nnadi<EMAIL_ADDRESS>
wrote:

The way the following_list view was done was a bit repetitive. We require
just a single view.
—
Reply to this email directly or view it on GitHub
https://github.com/andela/codango/pull/53#issuecomment-157521199.

I have also removed the line of code
On Tue, Nov 17, 2015 at 11:02 PM, Issa Jubril<EMAIL_ADDRESS>wrote:

My bad, i'm sorry i did not delete it before sending
On Tue, Nov 17, 2015 at 10:54 PM, Chidiebere Nnadi <
<EMAIL_ADDRESS>wrote:

The way the following_list view was done was a bit repetitive. We
require just a single view.
—
Reply to this email directly or view it on GitHub
https://github.com/andela/codango/pull/53#issuecomment-157521199.


Code release requirement

Dear Bochen,
Thanks for your great work! I am wondering when will the code be released?
Best,
Yixuan

Any updates?

Dear Bochen,
Thanks for your great work! I am wondering when will the code be released?
Best, Yixuan

Dear Yixuan,
Thank you for your interest in our work! The source code of this project has been released. For project guidelines and model weights, I'll upload them in the next few days.
Besides, Happy Mid-Autumn Festival!

"Since the original papers of EDCNet-S2D, CMX, and CMNeXt do not evaluate these two datasets, we re-implement them based on the source code with a 3-channel voxel grid [15] as event representation."
May I ask if you also use DSEC-Semantic train from scratch in reproducing the three networks EDCNet-S2D, CMX, and CMNeXt?

 "Since the original papers of EDCNet-S2D, CMX, and CMNeXt do not evaluate these two datasets, we re-implement them based on the source code with a 3-channel voxel grid [15] as event representation." May I ask if you also use DSEC-Semantic train from scratch in reproducing the three networks EDCNet-S2D, CMX, and CMNeXt?

@Candy-Crusher Thank you for your inquiry. Not from scratch. At the training stage, the encoders of these three models are loaded with ImageNet pre-trained weights. As for the decoders, they are randomly initialized.


How to take all files of project for compilation in ceedling

Hi I have started using Ceedling in my project fot making test coverage.
My Project is containing too many files, third party code as well.
If I take one source C file, create Test_source file and compile test_source c file then it gives linker Error because source c file contains different API calls, and These APIs are defined in other .c Files, there are many such Dependent .c files for source .c File to be tested.
Currently, I have to add .c files in compilation using TEST_FILE(); in test_source file because header files of similar name of source file is not available in project.
But there are many files dependent c source files So it is taking more amount of time for adding dependent c files manually. and it increases size of test_source file, because there is a large number of dependent source files.
I had tried with setting label  :auto_link_deep_dependencies: TRUE  but still it gives compilation error.
If we use makefile then we can find source .c files and also we can compile all source files using several operations.
Is there any way in which first I can compile all source files of the project so I don't need to provide dependent files manually to Test_source file?

Hi Mark,
Yes, I need to add real APIs instead of mock APIs.
In my project, I want to only mock the Low-level APIs which contain hardware dependency.

That makes sense when testing the units that use those low-level api's... in fact, you would only need to include a mock of the headers that a particular unit actually talks to.
Each unit gets compiled into its own test executable, therefore most often you will only want to include a single source file (the one being tested) with mocks of anything that this module requires. Done any other way, it quickly gets to a point where you can't successfully test it (or at least not UNIT test it... systems tests work in this way).


Sound and visual bug when player is die

1.12.2 minecraft forge + DynamicTrees

Go to Single
Open to LAN with cheats
/kill (x3 times for example)
Try to destoy block

I hear the block break many times. Both sound and visualization are broken.
In multiplayer too

What version of Forge and Dynamic Trees are you running? Do you have any other mods running at the same time?

What version of Forge and Dynamic Trees are you running? Do you have any other mods running at the same time?

UPD: Break ANY block (not only tree's blocks)
lastest on https://files.minecraftforge.net/ (DynamicTrees-1.12.2-0.9.4, DynamicTreesPHC-1.12.2-1.4.2, DynamicTreesTC-1.12.2-1.4.1e)
lastest on https://minecraft.curseforge.com/ (forge-1.12.2-<IP_ADDRESS>36-universal)
Yes, I have other mods, but it working (without DynamicTrees), and when I add DynamicTrees this problem appears
My mods:

this is probably the issue from #220

this issue is resolved in Dynamic Trees 9.0.5

update your Dynamic Trees (helps to try that before coming here too)

update your Dynamic Trees (helps to try that before coming here too)

thanks!


chore: update grafana dashboards and add backup instructions

Description
Resolves #3 by removing severity filters from Grafana dashboards.
Changes Made

Added backing up instructions under docs/DASHBOARD_EXPORT_GUIDE.md
Updated submodules to latest version.
Updated dashboards in provisioning to align with production state.
Adapted Grafana version in Dockerfile for consistent local testing.
Use aliased index in datasource provisioning (see SRGSSR/pillarbox-monitoring-transfer#16)

Checklist

[x] I have followed the project's style and contribution guidelines.
[x] I have performed a self-review of my own changes.
[x] I have made corresponding changes to the documentation.
[x] I have added tests that prove my fix is effective or that my feature works.

Maybe documenting the manual way to backup a dashboard could be valuable as well? (Share > Export > Save to file)


feat(ChatWindow): chat window

Overview

PR Checklist

[ ] Related to designs:
[ ] Related to JIRA ticket: [ABC-123]
[ ] I have run this code to verify it works
[ ] This PR includes unit tests for the code change
[ ] This PR includes testing instructions tests for the code change
[ ] The alpha package of this PR is passing end-to-end tests in all relevant Codecademy repositories

Testing Instructions

Don't make me tap the sign.
PR Links and Envs

Repository
PR Link
PR Env

Monolith
Monolith PR
Monolith Env

Portal
Portal Link
Portal Env

Another Repo
Another Link
Another Env

📬Published Alpha Packages<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://64527fcd6718892f021e1c4a--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages<EMAIL_ADDRESS>
📬Published Alpha Packages<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://645295837a18373ef286947f--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://6452ad3187b58f0e91fc555a--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://6454063703ca603684912083--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://6454217abe49bf11e3b39cbd--gamut-preview.netlify.appDeploy Logs


Fixed #10045 -- Corrected docs about .annotate()/.filter() ordering.

https://code.djangoproject.com/ticket/10045
Since it's not easily in the diff, the first query is
>>> Publisher.objects.annotate(num_books=Count('book')).filter(book__rating__gt=3.0)

The linked ticket calls out that a filter after an annotate will not reuse the join. @akaariai does that have to be so, or are we allowed to reuse that join? The annotate will reuse the join introduced by filter but not the other way around.
Actually, has anyone tested that this is still the case in 1.8/1.9? Perhaps (but unlikely) the expressions changes have changed the behaviour.

I confirmed it's still an issue on master. Here's the query:
SELECT "t10045_publisher"."id", "t10045_publisher"."name",
   "t10045_publisher"."num_awards", COUNT("t10045_book"."id") AS "num_books"
FROM "t10045_publisher"
LEFT OUTER JOIN "t10045_book" ON ("t10045_publisher"."id" = "t10045_book"."publisher_id")
INNER JOIN "t10045_book" T3 ON ("t10045_publisher"."id" = T3."publisher_id")
WHERE T3."rating" > 3.0
GROUP BY "t10045_publisher"."id", "t10045_publisher"."name", "t10045_publisher"."num_awards"

This is a feature. .filter(friends__age__lt=30).annotate(Avg('friends__age')) means "give the average age of friends under 30". .annotate(Avg('friends__age')).filter(friends__age__lt=30) means give average age of friends for those people who have friends younger than 30.
I don't like this feature. This is one of the cases where even those who know Django really well don't know how this actually works. In addition, the annotate() + m2m filtering issues causes the latter query to return wrong results if there are multiple friends younger than 30 for single person.

I added some more examples. Please let me know if you think anything should be added, changed, or deleted.

:+1: big improvement.


[BUG] unable to load Steam.dll

Describe the bug
sempre que eu inicio um jogo nao online aparece isso
Steps to Reproduce
1.instalo o jogo
2.extraio o jogo
3.aparece o bug
Expected behavior
abri o jogo normalmente
Screenshots

Operating System
W 10
Hydra Version
2.0.1
Additional Information
No response

We can't give support on how to install games. Hydra is just a launcher with a download manager.
You can join the discord server managed by the community, they have a lot more experience installing games and will be able to help you for sure.
https://discord.gg/TSxymg7JTJ


Instagram Login Result

InstagramLoginResult(super=StatusResult(status=fail, message=, spam=false, lock=false, feedback_title=null, feedback_message=null, error_type=null, checkpoint_url=null), logged_in_user=null,
why all variables are null? shouldnt they have some value so we can indicate the cause of  a failed login?
for example if it requires sms code, or it was me?

Can you paste the raw response that you get from Instagram? It should be printed on the log if you enable DEBUG level.

hello, how exactly do i enable debug level? all i get in the console is this (reegarding my issue)
the fail status is because i have two factor on,
and then i relogin with the code provided by sms, my concern is that those variables should have some value so we can know the exact error..

i really dont know how to solve this issue.. i get NPE when assigning value to login result variable from instagram.login
login_res = instagram.login();
and i also get null pointer exception when i try to read  login_res.getMessage()
how can i catch the cases when login fails due to it was me or sms?


intenger overflow in pdMS_TO_TICKS (IDFGH-2132)

Hello,
the ticket is for 2 issues.
a. The version of projdefs.h ( IDF v 3.3) is out of syn from the FREERTOS's one
2. The pdMS_to_ticks overflow the integer when the time is in hours.
documented here

Hi @az-z ,
As discussed on the thread, changing the type here won't prevent an overflow if using this macro to convert >1.2 hours into ticks. The better solution in this case is to divide by pdMS_TO_TICKS.
However we can still update the macro to cast the operand to TickType_t.

@projectgus ,
thank you for your feedback.
what if:
#define pdMS_TO_TICKS( xTimeInMs ) (  ( TickType_t ) ( xTimeInMs ) * ( configTICK_RATE_HZ  / ( TickType_t ) 1000 ) )

( configTICK_RATE_HZ / ( TickType_t ) 1000 ) = 0 for configTICK_RATE_HZ < 1000

combine into one?
#if configTICK_RATE_HZ < 1000
#define pdMS_TO_TICKS( xTimeInMs ) ( ( TickType_t ) ( xTimeInMs ) * ( configTICK_RATE_HZ / ( TickType_t ) 1000 ) )
#else
#define pdMS_TO_TICKS( xTimeInMs ) ( ( TickType_t ) ( xTimeInMs ) * ( configTICK_RATE_HZ / ( TickType_t ) 1000 ) )

That only works if tick rates greater than 1000 are also multiple of 1000. Usually max is 1000 anyway so it is not helpful.
#define pdMS_TO_TICKS( xTimeInMs ) ( ( TickType_t ) ( xTimeInMs ) / ( ( TickType_t ) 1000 / configTICK_RATE_HZ ) ) works best for configTICK_RATE_HZ ≤ 1000 as long as 1000 % configTICK_RATE_HZ = 0

Then what would be a good solution?
On Wed, Nov 6, 2019, 03:42 negativekelvin<EMAIL_ADDRESS>wrote:

That only works if tick rates greater than 1000 are also multiple of 1000.
Usually max is 1000 anyway so it is not helpful.
#define pdMS_TO_TICKS( xTimeInMs ) ( ( TickType_t ) ( xTimeInMs ) / ( (
TickType_t ) 1000 / configTICK_RATE_HZ ) ) works best for
configTICK_RATE_HZ ≤ 1000 as long as 1000 % configTICK_RATE_HZ = 0
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/espressif/esp-idf/issues/4292?email_source=notifications&email_token=ABGHHXLXEDFE2RI3XDUZVITQSJ7PJA5CNFSM4JJO5KG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDFXSPA#issuecomment-550205756,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABGHHXIOCIUU65ST2Y52X2LQSJ7PJANCNFSM4JJO5KGQ
.

As was suggested, don't use it.  https://github.com/espressif/esp-idf/search?q=portTICK_period_MS&unscoped_q=portTICK_period_MS

not sure what you meant with the url... the search for pdm_to_ticks returns a number of results too.

Yes they all show how to calculate a delay without using pdMS_TO_TICKS

the issue with overflow has been raised with the freertos team before .
Their answer was :

You should not need to edit any of the core FreeRTOS source files, just define pdMS_TO_TICKS() however you want in FreeRTOSConfig.h. The default definition appears as below, so will only be used if pdMS_TO_TICKS is not defined elsewhere.

#ifndef pdMS_TO_TICKS
#define pdMS_TO_TICKS( xTimeInMs ) ( ( TickType_t ) ( ( ( TickType_t ) ( xTimeInMs ) * ( TickType_t ) configTICK_RATE_HZ ) / ( TickType_t ) 1000 ) )
#endif
shouldn't the projdefs.h be updated with the proper version from upstream?
thank you.

Yes they all show how to calculate a delay without using pdMS_TO_TICKS. You are welcome to try fixing it in a PR and upstreaming it to freertos if you want but avoiding it is easier.

got it. yeah, they didn't seem to be interested to fix it before. (see my post above)

Right, to fix it you have to avoid the multiply and avoid fractional intermediate division results. It becomes a bit complicated to support arbitrary tick rates but that is already a problem for portTICK_period_MS calculation.

do i need to do anything to update the  projdefs.h from the upstream?

The definition of portTICK_RATE_MS (and other macros) from the newer upstream version FreeRTOS 10 is now in our preview branch and will be updated in master when this branch is merged and we update the master branch to FreeRTOS 10:
https://github.com/espressif/esp-idf/blob/feature/freertos_upstream_plus_tlsf_allocator_preview/components/freertos/include/freertos/projdefs.h#L41
I'm going to close this issue now though, as not using the FreeRTOS 10 projdefs.h is not a bug in the current version (which is based on FreeRTOS 8).

@projectgus I don't understand your answer.  The only change I see between the FreeRTOS 8 and FreeRTOS 10 macros are that they added a bunch of TickType_t casts which have no effect on the esp32 case (since TickType_t is uint32_t, and normal type promotions rules do the same thing as those added casts).
It doesn't address the real issue, which is that you're taking a uint32_t, multiplying, then dividing.  The multiply overflows even though the result would not.  An intermediate cast to uint64_t could fix it, or special case values like HZ=1000, etc?  It seems weird to say "Don't use this macro" when the macro could be fixed.


TF BERT dense einsum patch

Also cc @jplu

Thanks a lot @jlei2 for these inputs!!
What I see is that you are using the layer proposed in the official BERT repo, but recently they have deprecated it in favor of https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/EinsumDense Can please try to use this one instead and see if you still get the same results? I will be very useful for us to know.

Hi @jplu,
I tried using the new EinsumDense API and observed same results (PR: https://github.com/jlei2/transformers/pull/2) .
Tensorflow: 2.3.0
Tensorflow Serving: 2.3.0
1 GPU-V100 Cuda 10.1
model w/ EinsumDense

model  w/o EinsumDense

Big difference in GPU idle time:
model w/ EinsumDense

model  w/o EinsumDense


Statie: Lack of perex and feed

This is more of a question than an issue. I want my blog to have an rss feed. I tried to do this by:
{% for post in posts|slice(0,10) %}
      <item>
        <title>{{ post.title | xml_escape }}</title>
        <description>{{ post.content | xml_escape }}</description>
        <pubDate>{{ post.getDateInFormat('r') }}</pubDate>
        <link>{{ site.url }}{{ post|link }}</link>
        <guid isPermaLink="true">{{ site.url }}{{ post|link }}</guid>
      </item>
    {% endfor %}

However by the time this is being rendered post.content is actually the html for the entire post webpage rather than just the contents of the post (i.e. the html conversion of my markdown file). It appears that Statie just keeps replacing the content in place as it generates the page
I see some other people using statie have avoided this issue by using a "perex" in their frontmatter and including that, I'd rather not have to go and add that to each of my posts by hand though.
Do you have any suggestions?

Hi, the content is in the HTML form by design.
So you need the HTML to be text back again?
I use strip tags, see my website rss: https://github.com/TomasVotruba/tomasvotruba.cz/blob/master/source/rss.twig

Ideally I'd have the HTML as text again, yes. Given that post.content is a complete HTML document with title and site navigation and so forth, then just strip tagging won't work as I'll get the contents of the title element and so forth.
Perhaps I need to generate a post.perex, either automatically or as a one-off.

If you want to add content_raw property to the post object in PR, I'll guide you.

Is it necessary? AbstractGeneratorFile already has rawContent / getRawContent which is the markdown content converted to HTML and then strip tagged. I can then just use post.rawcontent in my twig file.
Perhaps I'm missing something or you prefer content_raw, I'm happy to help :)

Well, have you tried it?

Sorry, yes I gave it a quick try and seems to be fine.


feat(linter/eslint): implement sort-vars rule

This is good work. Ping me when you're ready for another review.


[Suggestion] Experience Transfer Tool

It would be good if you there was a slight texture change when swapping between modes to save having to check your bag to see.

Forgive my poor attempts, it was just a mock up 😜

good idea, but which way should be the Icons?
Xp flow from Tank view:

Player to Tank (+)
Tank to Player (-)

or tho other way around?
Xp flow from Player view:

Player to Tank (-)
Tank to Player (+)

the item glow gives already a hint which mode is current active.

Yes, fair point.
I'll leave that decision to you :D


ci: submodule trigger for - v<IP_ADDRESS>

Reason for Change:

submodule trigger for - v<IP_ADDRESS>
Issue Fixed:

Requirements:

[ ] uses conventional commit messages

[ ] includes documentation
[ ] adds unit tests
[ ] relevant PR labels added

Notes:

Closing this as the release for v<IP_ADDRESS> is done.


[tracking] Outstanding tasks to MVP light client contract

Problem Statement
Various rollups on EVM L1s may have a need for using Celestia as a DA layer, while keeping fraud/validity proof verification in the EVM. To that end,  two concrete possibilities exist: the Gravity bridge and a native EVM light client. This issue summarizes how each will work, any changes necessary for our usecase, and pros and cons.
Gravity Bridge
The Gravity bridge is a project initiated in the Cosmos ecosystem: https://github.com/cosmos/gravity-bridge. Various forks (maybe as many as a dozen at this point) exist of this protocol, with minor changes. Non-exhaustively

https://github.com/PeggyJV/gravity-bridge (Rust off-chain code)
https://github.com/InjectiveLabs/peggo (Go off-chain code)

Method of Operation
The Gravity bridge has two components: an on-chain component written in Solidity and an off-chain component (the orchestrator) written in Rust (Peggy) or Go (Injective). A high-level explanation of the Solidity contracts can be found here.
The orchestrator runs alongside a Tendermint node (potentially as an SDK module) that causes validators (i.e. block producers) to, in addition to regular Tendermint votes, also sign EVM-compatible messages that mirror:

validator set changes, and
arbitrary events in blocks.

Specifically, messages are signed in accordance with ERC-191. These signatures are batched and sent to an on-chain contract. The contract keeps track of the validator set as it changes, and verifies that the arbitrary events are signed off by a supermajority of voting power.
Updates to the validator set must occur every time

The validator set changes too much (currently, 5% is recommended, though this parameter may be tweaked up to 33% I think), or
No other updates have occurred within the unbonding window. For safety, probably 1/2 the unbonding window should be used here.

To update the contract's view of the validator set, updateValset is called. The orchestrator will construct an EVM-compatible transaction with batched signatures here. The contract will simply verify that:

The provided previous validator set matches the known hash of the validator set, and
A supermajority of the previous validator set has voted correctly for the new validator set.

Then update its saved validator set hash.
I estimate this operation is ~500k gas each time. As an example, Ethereum is an EVM-compatible chain and at its current gas prices this would cost ~$400. It would be prohibitive to update the validator set every block, hence the above threshold of 5%. With only occasional updates, this cost should end up not being too significant.
Batches of arbitrary messages are submitted to the EVM chain with submitBatch. This requires verification of the current validator set and a supermajority of signatures. Note that arbitrary messages does not necessarily imply single blocks; properly implemented, a single batch of arbitrary messages can be made across as many blocks as can fit in the gas limit.
Required Changes

Numerous gas optimizations can and should be made. Just as an example of how immature the implementation is: when updating the validator set, the new validator set is provided. But this is actually not needed under an honest majority assumption; a commitment to the new validator set is sufficient.
The submitBatch function only works for transfers of fungible tokens across the bridge. This will have to be modified for our purposes, namely a list of (block height, namespace ID, message commitment) tuples.
The orchestrator exists mostly as a separate module for now. It will have to be fully integrated into either the SDK or the node software directly.
An audit of the contracts will be required.

Pros

A majority of the of the code already exists. This will probably be the fastest option.
Potential for Cosmos branding and association.

Cons

Realistically, there is 0% chance of being able to upstream changes, barring us owning the canonical implementation of Gravity bridge. This is because there is no specification or reference implemention of the Gravity bridge protocol, so a dozen projects each have their own completely independent fork, with more popping up. This will have to be developed in isolation of the broader Cosmos ecosystem solely for our purposes.
Cost of verifying signatures for each message batch is non-trivial. Verifying 50 signatures would be ~200k gas, while simply posting them would be ~50k.
Requires validators to run a module that introduces a logic dependence on the EVM, and EIP-191. In addition, this will requires computational overhead in the form of each validator needing to run a full node for the EVM chain (or trust a centralized service provider to provide them confirmation of transaction inclusion and effect).
One of the key requirements for PoS is accountability: being able to identify which validators of a dishonest majority were in fact dishonest, and burning their stake with social coordination. Validators that submit a bad batch or validator set update to the EVM chain must be penalized, otherwise the bridge isn't making and honest-but-accountable-majority assumption (good), but an honest-and-unaccountable-majority assumption (bad). To that end, non-validator nodes must also be able to understand the EVM and EIP-191 in order to be able to understand that the validator submitted something they should not have, and penalize them.
A separate instance of the Gravity bridge module is required to handle each chain that is bridged to. This is not yet implemented properly, but even if it were, it would be onerous in a future where potentially many chains would leverage Celestia as a DA service.

EVM Light Client Bridge
The EVM light client bridge (this repo) is a Celestia light client written in Solidity. It accepts new blocks headers as they are in Celestia, and follows the chain of block headers under an honest majority assumption.
Method of Operation
The EVM light client bridge has two components: an on-chain component written in Solidity and an off-chain component (which we'll call the orchestrator), yet to be written.
The on-chain contract will accept new Celestia block headers in protobuf, along with addition metadata, such as the validator set (diff). It can then update its view of the validator set similar to the Gravity bridge. Again, the actual updating of the validator set is only required when a threshold of the voting power changes.
To parse the protobuf-encoded header, the following libraries implement deterministic protobuf:

https://github.com/celestiaorg/protobuf3-solidity
https://github.com/celestiaorg/protobuf3-solidity-lib

The final piece of the puzzle is that verification of the header and signatures must be done optimistically, by making the header chain an optimistic rollup. This is necessary since parsing protobuf every time would be too expensive. This would significantly reduce gas costs vs the Gravity bridge: verifying 50 signatures would be ~200k gas, while simply posting them would be ~50k. This similarly applies to processing messages. In other words, gas costs can be reduced by somewhere around a factor of 4 with an optimistic EVM light client bridge vs the Gravity bridge.
Required Changes
Significant changes are required.

Actually use protobuf to encoding/decode headers, etc. Library is implemented but is unused.
NMT proof verification, transaction and share parsing, etc.
Optimistic verification and rollbacks. This is one of the easier things to implement on this list, since the author is moderately familiar with how optimistic rollups work.
Off-chain orchestrator will have to be implemented, but significant portions of the Gravity bridge can be re-used.
Gas optimizations for the protobuf library.
An audit of the contracts will be required.

Pros

Since the contract accepts Celestia block headers directly, there is no need for validators to run any separate module. In addition, validators do not even need to run anything that connects to another chain in any ways. A single relayer can be tasked with posting all updates to the relevant EVM chain(s).
There is no logical dependence on the EVM or EIP-191. The bridge operates under an honest-but-accountable-majority assumption.
With optimistic verification, gas costs can be reduced significantly vs the Gravity bridge.
Potential to maintain potentially widely-used optimistic Tendermint light client bridge and reference implementation of deterministic protobuf codec Solidity library.

Cons

Large parts of the Solidity code are missing, including actually using the protobuf codec, NMT proof verification, transaction and share parsing, etc.
An orchestrator will have to be implemented to actually send transactions to Ethereum. The Gravity orchestrator can be re-used mostly.
With optimistic verification, a delay is required to finalize the submitted block headers. This may be unacceptable for zk rollup projects, but is fine for optimistic rollups.

Discussion
Don't forget to like, comment, and subscribe 👇

An audit of the contracts will be required.

How come an audit is only necessary in the gravity bridge case but not for the EVM Light Client case?
I think we should expand more on the required changes so we can properly estimate the amount of time both would take. My intuition is that the gravity bridge changes are less involved and a matter of a few weeks max? While the evm client work is rather months not weeks? Is that about right?
But before that we should clarify if the lack of accountability a show-stopper for using the gravity bridge.

How come an audit is only necessary in the gravity bridge case but not for the EVM Light Client case?

Oh, an audit would be necessary for both, I just forgot the add it for the latter.

is that the gravity bridge changes are less involved and a matter of a few weeks max?

About right, yes.

But before that we should clarify if the lack of accountability a show-stopper for using the gravity bridge.

It is not IMO.

Decision is to modify the Gravity bridge as above.


Fix for relative asset lookup (shouldUseRelativeAssetPaths)

Preface
The issue to import e.g. fonts in css #5162 was closed by "Correctly lookup assets when using a relative build directory" #5163. The fix adds this line, to decide whether the feature should be used or not.
https://github.com/facebook/create-react-app/blob/7973c81b912145350e8da68a3b91a17075b8b4fd/packages/react-scripts/config/webpack.config.prod.js#L33
There are comments spread that recommend to set "homepage": "." or "homepage": "./" in package.json. See documentation.
Issue
The application is served under different domains prefixed with a path ("/mypath").
Setting "homepage": "./mypath" was working fine, although I'm not sure if this is supported.
But the fix for assets urls in (s)css will not be applied, because of the condition:
https://github.com/facebook/create-react-app/blob/7973c81b912145350e8da68a3b91a17075b8b4fd/packages/react-scripts/config/webpack.config.prod.js#L33
Fix
We could change the condition and paths will work in the described case.
const shouldUseRelativeAssetPaths = publicPath && publicPath.startsWith('./');

This is how I work around this issue.

Make sure the application is served under /mypath/
Redirect /mypath to /mypath/
Use "homepage": "."

I'll let the issue open to clarify, what the intended behavior should look like and decide if the fix should address the issue, or documentation could be updated.

The same problem appears if react-router is used, so that the application is server under a path, but there is an additional path by the router.

Any feedback?


failed to download package docs

After updating, sidekick is not working for public packages. Sidekick still works for local functions. Uninstall + reinstall of the package and changing the cache directory did nothing.

Traced it back to a dependency in elm-package.json that isn't on http://package.elm-lang.org/ (contains Native code). It looks like failing to find that package's documentation caused all the other ones to fail.

Hi @feldman4, I think your hunch is correct.  Right now, elmjutsu is using Task.sequence for that part.  Thanks for reporting this issue!


Ret ixmp

fix of the retixmp package that allow to load on ixmp any kind of dataframe and array
added retixmp installation in intall.bat
expanded the README, explaining the requirements for using retixmp

Please confirm that this PR has done the following:

[ ] Tests Added
[x] Documentation Added
[ ] Description in RELEASE_NOTES.md Added

Adding to RELEASE_NOTES.md
Please add a single line in the release notes similar to the following:
(should I do it after submitting the PR?)
- (#68)[https://github.com/iiasa/ixmp/pull/68] bring retixmp available to other users

I couldn't find a .stickler.yml file in this repository. I can make one for you, or you can create one by following the documentation.

I couldn't find a .stickler.yml file in this repository. I can make one for you, or you can create one by following the documentation.


Playground: selecting broken examples in a particular order causes a crash with a weird error message

Steps to reproduce (assuming #172 is not fixed):

Select "Draw an image"
Select "Moving rectangle"
Select "Hello world (using font.um)"

Result:
error: /playground_main.um (12, 39): Incompatible types Font and { Font Err }
error: /window.um (128): Pointer is null

	Stack trace:
		/window.um:000128: cycle
status: Exception thrown, see JavaScript console

Program terminated with exit(-1)

Expected:
error: /playground_main.um (12, 39): Incompatible types Font and { Font Err }

Since #172 is (almost) fixed, I can no longer reproduce the current issue.


Latest changes on master in develop

Hello,
could you please merge the latest fixes on master to the develop branch?
Thanks in advance,
Christoph

Done.


docs(architecture): move diagram up; elab on svcs; add LC Hooks links

Closed by bf29846f3e8f0b7ee2fd6253318f516151a2ab37


Unable to import JSX types, Error: "no interface JSX.IntrinsicElements exists", although it exists

TypeScript Version:  4.1.2

Search Terms:
no interface JSX.IntrinsicElements exists
Code
This works:
namespace JSX {
    export interface IntrinsicElements {
        div: {[k: string]: any}
    }
}

const d =
 // <-- no error
type test = JSX.IntrinsicElements // <-- no error

playground
But this does not work:
import type {JSX} from 'solid-js'

const d =
 // Error: JSX element implicitly has type 'any' because no interface 'JSX.IntrinsicElements' exists.

// But JSX is in scope, just that JSX expressions do not see it.
// As an example, the following non-JSX code works fine:
type test = JSX.IntrinsicElements // <-- no error

playground
Expected behavior:
It should see the definition of JSX.IntrinsicElements which is clearly present due to the import statement.
Actual behavior:
The JSX definition in the second example (imported) appears to be invisible to TS.
As you can see from both examples, only a locally-defined JSX works, but not one that is imported.

If libraries could export non-global versions of JSX types, then projects could very easily have a mix of files where each file may have a different type of JSX (f.e. React in some files, Preact in others, and Solid.js in others, assuming those libs export non-global JSX types).

namespace JSX {
    export interface Element {
//  ^^^^^^
        foo: number
    }
    export interface IntrinsicElements {
//  ^^^^^^
        div: HTMLDivElement
    }
}

const d =

If libraries could export non-global versions of JSX types, then projects could very easily have a mix of files where each file may have a different type of JSX (f.e. React in some files, Preact in others, and Solid.js in others, assuming those libs export non-global JSX types).

They can. preact does this.

Ahh, thanks! I never use namespace; I totally overlooked export.

Turns out it actuall doesn't work when JSX is imported, as any person organizing their code will want to do:
This does not work:
// WHILE THIS WORKED
// namespace JSX {
//     export interface Element {
//         foo: number
//     }
//     export interface IntrinsicElements {
//         div: {foo: number, bar: string}
//     }
// }

// THIS DOES NOT
import type {JSX} from 'solid-js'

const d =

 // JSX element implicitly has type 'any' because no interface 'JSX.IntrinsicElements' exists.

playground

Please note, the JSX type from solid-js is in a .d.ts file, and in that case TypeScript apparently does not require export inside the namespace JSX definition. TS requires export inside the namespace when the file is a module (the inconsistency is confusing).
I also tried converting that file to a .ts file, and adding export for each type inside the JSX definition, but the error remains the same (the consumer code can not find the JSX.IntrinsicElements interface).

You wanna use the @jsx pragma to change the factory function used from React.createElement to whatever the factory for your library is. The JSX factory is looked up relative to the factory specified. So if your factory is h (eg, via a /* @jsx h */) we look for h.JSX.

Thing is, I don't want any jax pragma. I'm using preserve, and transpile JSX with a custom step.
But still, JSX types are in scope, and TS is not picking it up. I don't think there should be any difference between an inline declaration and an import with regards to visibility. JSX shouldn't have special treatment.

We have an existing import syntax, and I think it is very intuitive for that to work.
There is a new /* jsxImportSource ... */ syntax specifically for importing JSX types coupled with a factory. That is not intuitive: people don't assume to import all types with import except one special type (JSX) that requires comment syntax.
I think that
import {JSX} from `somewhere`
// or
import type {JSX} from `somewhere`

Should just work. I really feel there isn't a need to make JSX a hacky unique type when it comes to making it visible in a scope.

You can still use the pragma with preserve, I believe - we need the name of the factory function to know under which non-global namespace to look for the JSX namespace.

(Also, we didn't "invent" the comment pragma syntax - we just copied it from babel)

jsxImportSource requires a jsx-runtime.d.ts file at the root of a package. This coupling of JSX types to a specific filesystem structure is not ideal at all.
We should be able to

write "jsx": "preserve" in tsconfig.json (or specify a jsxFactory for compile output)
import type {JSX} from 'anywhere' in any file (or import it from a file that makes it global, which already works), or specify "jsxImportSource": "anywhere" (without the implied jsx-runtime path) to have it auto-imported in all files.

and be done.
I can't think of a good reason to have this stuff coupled to specific files or coupled to React.
The "runtime" in jsx-runtime doesn't make any sense for libraries that want to use jsx:preserve, as there isn't even a runtime for TypeScript to care about.
After fixing jsxImportSource to not require some particular file path, it will make more sense because:

if jsx:preserve is used along with jsxImportSource, then one can expect that it will import only type definitions (which apparently is not documented anywhere to begin with)
if jsx:react or jsxFactory or similar are used, then the runtime can be expected to also be imported from jsxImportSource

But it would be best to simply allow import {JSX} (or import {JSX, jsx}, or import {JSX, h}, depending on tsconfig options) in any file, allowing users to use import instead of jsxImportSource, being more closely aligned with standards and conventions.

TypeScript playground does not install dependencies for jsxImportSource. playground example
If we could instead rely on import, the playground example (which will install solid-js based on the import) would just work.

Besides when @jsxImportSource comments aren't working (I posted a reproduction there),
I realized that we have to import JSX twice in order to use it in both normal code, and with JSX expressions.
So in order for the following code to work (try it here live in your browser, requires free Gitpod account, or see the repository), we must import JSX in the same file using both syntaxes:
/* @jsxImportSource solid-js */ // first import of JSX
import { createState, onCleanup, Component, JSX } from "solid-js"; // second import of JSX
import { render } from "solid-js/web";

const App: Component = () => {
  const [state, setState] = createState({ count: 0 }),
    timer = setInterval(() => setState("count", (c) => c + 1), 1000);
  onCleanup(() => clearInterval(timer));

  // The @jsxImportSource comment is needed for this JSX expression to recognize the
 intrinsic element.
  return
{state.count}
;
};

render(() => <App />, document.getElementById("app"));

// The `import {JSX} from "solid-js"` is  needed in order to use the JSX types in non-JSX expressions
interface Foo extends JSX.HTMLAttributes<HTMLDivElement> {}

I hope by now you may be convinced that import {JSX} for both cases is more intuitive, more DRY, and probably less likely to not work.

@weswigham I updated the OP to make it clear based on the new findings regarding import.

I appreciate your consideration on this issue. At this point I'm fairly used to having most of the ecosystem assume JSX === VDOM factory function because Babel/TypeScript ship with a default JSX transform. But the spec for JSX makes no mention of what the output should be only indicates the syntax. I'm happy to have any solution at all. Even it is simply from the fact that React decided one day to have a new version that doesn't rely on a pragma import. The pre 4.1.1 solutions were horrendous, so I will happily follow predefined file paths for types if it accomplishes the goal. I understand since you did not invent the syntax for jsxImportSource you can't really change its function, but importing JSX working seems like it could be reasonable.

We do two things to find JSX types:

Look for <root identifier of JSX factory entity name>.JSX - IMO, this is preferred, as it is local to a file, and customizable on a per-file, and per-JSX package basis. However, it is also newer (and was not in our original JSX implementation), so fewer packages/authors support it.
Look for a global JSX namespace. This was our original implementation behavior, done as such because all JSX types are essentially compiler-intrinsic changing types that set global behaviors. We had no concept of file-local JSX behavior.

We have never looked for a locally scoped namespace named JSX, nor do I think we should further diversify what strange things you can do to typecheck JSX. We already have a mechanism for locally-sourced JSX definitions - if you can, I recommend using it.

We do two things to find JSX types:

Look for <root identifier of JSX factory entity name>.JSX - IMO, this is preferred, as it is local to a file, and customizable on a per-file, and per-JSX package basis. However, it is also newer (and was not in our original JSX implementation), so fewer packages/authors support it.
Look for a global JSX namespace. This was our original implementation behavior, done as such because all JSX types are essentially compiler-intrinsic changing types that set global behaviors. We had no concept of file-local JSX behavior.

We have never looked for a locally scoped namespace named JSX, nor do I think we should further diversify what strange things you can do to typecheck JSX. We already have a mechanism for locally-sourced JSX definitions - if you can, I recommend using it.

Right, Global though is basically a mess. You can't have multiple JSX providers in the same project using Global they attack each other and it is completely broken. In those scenarios it is basically a non-starter.
I strongly dislike the assumption JSX has a single factory function. It is very narrow view that isn't true. From Inferno's precompiled VDOM, Solid's custom DOM transformations.. It is not a coincidence these also happen to be the most performant JSX libraries. You will see more optimizations of this nature in the future for performance reason. I would expect similar from Vue at some point and I wouldn't count React out in the future.
There are plenty of things that JSX can generate and do. So jsxImportSource not being attached to a factory is a huge step forward to remove the last of the global JSX as it gets adopted. I'm fine with that generally but there are some obvious inconsistencies which @trusktr brings up. I'm not sure the best way though since the default would be to apply it across the project without any per file intervention. The challenge is that jsxImportSource only means something terms of the new Babel translation so coming from a clean slate perspective it's not obvious why. The next time Babel changes (like they just did) so will TypeScript.
So it could be beneficial to have a native TypeScript solution, but I do understand if the plan is just continue keeping this perspective and follow Babel's lead. Interestingly Babel itself does a pretty good job following the spec in terms of their syntax support, whereas TypeScript has historically tuned the implementation to match React.

Right, Global though is basically a mess. You can't have multiple JSX providers in the same project using Global they attack each other and it is completely broken. In those scenarios it is basically a non-starter.
I strongly dislike the assumption JSX has a single factory function. It is very narrow view that isn't true. From Inferno's precompiled VDOM, Solid's custom DOM transformations.. It is not a coincidence these also happen to be the most performant JSX libraries. You will see more optimizations of this nature in the future for performance reason. I would expect similar from Vue at some point and I wouldn't count React out in the future.
There are plenty of things that JSX can generate and do. So jsxImportSource not being attached to a factory is a huge step forward to remove the last of the global JSX as it gets adopted. I'm fine with that generally but there are some obvious inconsistencies which @trusktr brings up. I'm not sure the best way though since the default would be to apply it across the project without any per file intervention. The challenge is that jsxImportSource only means something terms of the new Babel translation so coming from a clean slate perspective it's not obvious why. The next time Babel changes (like they just did) so will TypeScript.
So it could be beneficial to have a native TypeScript solution, but I do understand if the plan is just continue keeping this perspective and follow Babel's lead. Interestingly Babel itself does a pretty good job following the spec in terms of their syntax support, whereas TypeScript has historically tuned the implementation to match React.

I strongly dislike the assumption JSX has a single factory function

I agree. This discludes other forms of JSX that do not have factory functions (like @ryansolid's Solid.js is quite amazing), and it means they would have to define a fake type for something that doesn't actually exist.

The next time Babel changes (like they just did) so will TypeScript.

Disregarding that, I really don't see why we can't just have jsx:preserve along with import {JSX} from 'anywhere', which would be intuitive and following standards that the ES community already has, and be on our way.
If import worked, I wouldn't have spent multiple days fiddling to try and get JSX segregation working.
@weswigham Have you used more than one form of JSX in a single project? It is a real big pain.
I believe import {JSX} from 'anywhere' should at least be supported, regardless how the other stuff changes. It makes so much sense.

I strongly dislike the assumption JSX has a single factory function

I agree. This discludes other forms of JSX that do not have factory functions (like @ryansolid's Solid.js is quite amazing), and it means they would have to define a fake type for something that doesn't actually exist.

The next time Babel changes (like they just did) so will TypeScript.

Disregarding that, I really don't see why we can't just have jsx:preserve along with import {JSX} from 'anywhere', which would be intuitive and following standards that the ES community already has, and be on our way.
If import worked, I wouldn't have spent multiple days fiddling to try and get JSX segregation working.
@weswigham Have you used more than one form of JSX in a single project? It is a real big pain.
I believe import {JSX} from 'anywhere' should at least be supported, regardless how the other stuff changes. It makes so much sense.

The @jsxImportSource feature is also lacking.

There is no hovering on it to see any intellisense tooltips.
With import we can hover on the imported identifiers and see information.
With import we can Go To Definition on imported identifiers or the module specifier.

It really makes much sense to support import {JSX}.

nor do I think we should further diversify what strange things you can do to typecheck JSX.

You should honestly remove the strange things. But even if you keep them, import works in an obvious way, and it isn't strange, and people know what to expect.
The very first thing I thought was: I'll just import JSX.
It makes so much sense!

The @jsxImportSource feature is also lacking.

There is no hovering on it to see any intellisense tooltips.
With import we can hover on the imported identifiers and see information.
With import we can Go To Definition on imported identifiers or the module specifier.

It really makes much sense to support import {JSX}.

nor do I think we should further diversify what strange things you can do to typecheck JSX.

You should honestly remove the strange things. But even if you keep them, import works in an obvious way, and it isn't strange, and people know what to expect.
The very first thing I thought was: I'll just import JSX.
It makes so much sense!

Note, Babel's new jsx-runtime feature (which also does not follow ES Module standards by virtue of requiring a particular import path) is specifically for libraries with runtime factory functions, and largely guided by React.
This is unfair to the rest of the JSX ecosystem, and TypeScript is making the choice to support the unfairness (or, at least not providing easier options for other forms of JSX).

Note, Babel's new jsx-runtime feature (which also does not follow ES Module standards by virtue of requiring a particular import path) is specifically for libraries with runtime factory functions, and largely guided by React.
This is unfair to the rest of the JSX ecosystem, and TypeScript is making the choice to support the unfairness (or, at least not providing easier options for other forms of JSX).

@trusktr Can you please log a new concrete suggestion issue for supporting that? Piling on more comments to something that was filed as a bug is very confusing for us; if we want to have a real discussion on that point it's very awkward that we'd have to scroll down n comments to understand what the issue was really about.

@trusktr Can you please log a new concrete suggestion issue for supporting that? Piling on more comments to something that was filed as a bug is very confusing for us; if we want to have a real discussion on that point it's very awkward that we'd have to scroll down n comments to understand what the issue was really about.

@RyanCavanaugh I think the OP describes the issue fairly clearly. I'm not sure what I can provide in a new issue.

@weswigham @RyanCavanaugh Mind re-opening the issue? The OP describes the issue exactly, there is no need for a new one. But I can make a duplicate if you confirm that is what you want me to do.


Diaganostics settings policy is out of preview, update text & screenshots

The diagnostics policy is out of preview, the references to [Preview] in the text and screenshots should be removed.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5fe7e618-4080-5f13-27fc-8f4f16ea67ba
Version Independent ID: eb10e12f-553d-41b6-67de-4b7756cbfac8
Content: Configure Vault Diagnostics settings at scale - Azure Backup
Content Source: articles/backup/azure-policy-configure-diagnostics.md
Service: backup
GitHub Login: @v-amallick
Microsoft Alias: v-amallick

Thanks for bringing this to our attention. Your feedback has been shared with the content owner for further review.

Hi @shanecastle-osharp
Thank you for letting us know. Currently, looking into the issue. Will get back soon.

Hi @shanecastle-osharp, we have updated the article and screenshots. These will reflect in the article soon. Thank you.

#please-close


Changelog

Would you mind to add a CHANGELOG.md to the repo?

I should, I'm version bumping a tad too aggressively and I imagine that makes keeping up to date difficult. I'll get one included tomorrow.

I have a first draft of the changelog up here.
I'll leave this Issue open for a few days while I touch it up and record changes from earlier versions.

great 👍

I feel like the changelog is in a decent state.
Closing this issue now.


Add Another Way

Can you add these too?
Link: https://polarnotion.github.io/starwarsintro/
https://user-images.githubusercontent.com/4999783/136720590-d399b2e0-46aa-4ec4-bce8-233b2c266f63.mp4

I added a 2nd journal option with this CSS for the starwars crawl. I've left the old version in place for legacy users as well.


Add a few typical characters to the notify regex

Just a small update to the regex for notifyall. Currently, it breaks for anyone with characters or other non-alphabetical characters. This isn't comprehensive, but it will catch some of the most common ones.

(As  I suggested in jira): let's stop using regular expressions when split would work just fine


Possible to start the line chart from left?

Had a doubt: If it's possible to start the line chart from the left instead of right (default behavior)?
And then start to reverse the x-axis when the line chart reaches the maximum across.

Try to change in:
const config = {
  options: {
    scales: {
      x: {
      reverse: true,
      },
     ....
    },
  }
};

To make it reverse, then try to make async call to change it back to reverse: false  when the time will equal to ttl.

Hi,
Did you find any solution?

Ah I dont have a code reference to show, but I remember what i did, I think I just assigned a negative value to the x-axis, something like that, don't quite remember now.