January Publish #3

Docs Build status updates of commit f9a6148:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/psexec.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit f9a6148:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/psexec.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit b9eee2a:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/psexec.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit b9eee2a:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/psexec.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit a8ac891:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/psexec.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit a8ac891:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/psexec.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 645468f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/psexec.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 645468f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/psexec.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Problems trying to visualize Network Training with the Deeplearning4j Training UI

I tried to follow the guide here
https://deeplearning4j.org/docs/latest/deeplearning4j-nn-visualization
but during training there is apparently nothing published at
http://localhost:9000/train
I tried to apply the guide to an existing and working project with the following initial pom.xml dependencies:
		<dependency>
			<groupId>org.nd4j</groupId>
			<artifactId>nd4j-native-platform</artifactId>
			<version>1.0.0-beta6</version>
		</dependency>
		<dependency>
			<groupId>org.deeplearning4j</groupId>
			<artifactId>deeplearning4j-core</artifactId>
			<version>1.0.0-beta6</version>
		</dependency>

Following the deeplearning4j-nn-visualization page I tried to add this dependency
    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-ui_2.10</artifactId>
        <version>1.0.0-beta4</version>
    </dependency>

But simply doing, when executing line
UIServer uiServer = UIServer.getInstance();
I get the following error
Exception in thread "main" java.lang.IllegalArgumentException: Don't know how to bind argument of type class net.jodah.typetools.TypeResolver$Unknown
	at play.routing.RoutingDsl.pathBindableFor(RoutingDsl.java:233)
	at play.routing.RoutingDsl.with(RoutingDsl.java:190)
	at play.routing.RoutingDsl.access$000(RoutingDsl.java:66)
	at play.routing.RoutingDsl$PathPatternMatcher.build(RoutingDsl.java:362)
	at play.routing.RoutingDsl$PathPatternMatcher.routeTo(RoutingDsl.java:298)
	at org.deeplearning4j.ui.play.PlayUIServer.runMain(PlayUIServer.java:179)
	at org.deeplearning4j.ui.api.UIServer.getInstance(UIServer.java:67)
	at org.deeplearning4j.ui.api.UIServer.getInstance(UIServer.java:45)

Then I tried to add
		<dependency>
			<groupId>org.deeplearning4j</groupId>
			<artifactId>deeplearning4j-ui</artifactId>
			<version>1.0.0-beta6</version>
		</dependency>

with this, there are no errors, but if during the training I open browser (as indicated in the guide) at
http://localhost:9000/train
the page is unreachable.
There is some other dependency to be added?
I am using the following versions:

Deeplearning4j version:  1.0.0-beta6
Platform information (OS, etc): Linux Arch 64bits
CUDA version: not used
NVIDIA driver version: not used

Thank you very much
Best regards
Vincenzo

First - don't use mixed versions, everything (all ND4J/DL4J/DataVec etc dependencies) should always be the same version. You probably got a warning about that.
(But yes, that page is outdated listing 1.0.0-beta4 deeplearning4j-ui_2.10 instead of 1.0.0.-beta6 deeplearning4j-ui, cc @treo )
As for not being able to see anything: I think that's probably code related.
The UI server doesn't initialize (i.e., no web server) until you do this:
UIServer uiServer = UIServer.getInstance();

and you won't have any data until you do this: (you'll see the empty graphs/layout though)
uiServer.attach(statsStorage);

I'm pretty sure being unable to access the server is due to no initialization of the UI server. If you still can't get it working, post your code too and we can re-open this issue and help you more.

Hi Alex,
I am now using 1.0.0-beta6 for all dependencies.
I followed exactly the code in the guide, that is:
    //Initialize the user interface backend
    UIServer uiServer = UIServer.getInstance();

    //Configure where the network information (gradients, score vs. time etc) is to be stored. Here: store in memory.
    StatsStorage statsStorage = new InMemoryStatsStorage();         //Alternative: new FileStatsStorage(File), for saving and loading later

    //Attach the StatsStorage instance to the UI: this allows the contents of the StatsStorage to be visualized
    uiServer.attach(statsStorage);

    //Then add the StatsListener to collect this information from the network, as it trains
    net.setListeners(new StatsListener(statsStorage));

but things doesn't change.
I also was able to reproduce the behavior cloning repo
https://github.com/eclipse/deeplearning4j-examples.git
and executing class
org.deeplearning4j.examples.userInterface.util.UIExample.java
The execution hangs at line
net.fit(trainData);
where the console shows
Feb 10, 2020 12:34:02 AM org.nd4j.util.OneTimeLogger info
INFO: Starting MultiLayerNetwork with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]
Feb 10, 2020 12:34:03 AM org.deeplearning4j.ui.VertxUIServer attach
INFO: StatsStorage instance attached to UI: FileStatsStorage(/tmp/ui-stats.dl4j)

and address
http://localhost:9000/train
is unreachable.
Is the address right? What could cause this block?
Just a note: if I comment those UIServer-related lines of code all is working right (no freeze)

http://localhost:9000/train is unreachable. Is the address right?

Yes, that's the correct address.
That's unexpected, there are no known issues with freezing during training, with or without the UI.
Furthermore even if there was an issue during training, it should not make any difference what-so-ever to the UI web server - they are relatively independent systems.
Can you confirm that the examples were a fresh clone, completely unmodified?
And is this being run in an IDE or via maven command line?
We'll also need more info about your system. OS, basic hardware, etc.
Finally, can run it until it freezes, and then give us a thread dump?
If you're using Intellij, you can use "Dump Threads" in the run tool window - https://www.jetbrains.com/help/idea/run-tool-window.html
If you're on command line, you can use jps and jstack - https://helpx.adobe.com/au/experience-manager/kb/TakeThreadDump.html

Platform information (OS, etc): Linux Arch 64bits

I'm pretty sure those builds are not up and running yes @sshepel?
Nothing seeing anything here, for example:
https://jenkins.konduit.ai/blue/organizations/jenkins/eclipse%2Fdeeplearning4j/detail/master/77/pipeline

cc @raver119 - any thoughts?
Looks like it might be a c++ deadlock on Nd4jCpu.execSummaryStatsScalar, but I can't see how/why that would happen (never seen this on any of our machines, or CI machines).
Java version and linux variant shouldn't make any difference AFAIK.
@vincenzocaselli what exactly is the processor model? I saw a couple of different possibilities for the Dell 5530

@vincenzocaselli so, you're using example as is, and it deadlocks for you?

@raver119 That's my understanding, yes, unmodified latest version of dl4j-examples (1.0.0-beta6)
I've tried that same example on my machines, and it's fine.
(We also run it as part of pre-release QA anyway)

Java version and linux variant shouldn't make any difference AFAIK.

As a side note I tried both with
OpenJDK Runtime Environment (build 13.0.2+8)
OpenJDK 64-Bit Server VM (build 13.0.2+8, mixed mode)

and
Java(TM) SE Runtime Environment (build 13.0.1+9)
Java HotSpot(TM) 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

with same results.

@vincenzocaselli what exactly is the processor model? I saw a couple of different possibilities for the Dell 5530

Intel Core i7-8850H (Six Core 2.60GHz, 4.30GHz Turbo, 9MB 45W)

@vincenzocaselli so, you're using example as is, and it deadlocks for you?

yes
Let me know if I can make other tests: different JVMs, change some env variable, ...

Can you please export OMP_NUM_THREADS=1 and run again? Things will get slower, but we'll rule out parallelism.

Can you please export OMP_NUM_THREADS=1 and run again? Things will get slower, but we'll rule out parallelism.

@raver119  done, but unfortunately the behavior is exactly the same

Hm. That doesn't sound realistic to me. Can you please disable UI stuff, and put regular score listener there with frequency of 1?

@raver119 by skipping UI stuff it works like a charm
    public static void main(String[] args){
        //Get our network and training data
        MultiLayerNetwork net = UIExampleUtils.getMnistNetwork();
        DataSetIterator trainData = UIExampleUtils.getMnistData();

//        //Initialize the user interface backend
//        UIServer uiServer = UIServer.getInstance();
//
//        //Configure where the network information (gradients, activations, score vs. time etc) is to be stored
//        //Then add the StatsListener to collect this information from the network, as it trains
//        StatsStorage statsStorage = new FileStatsStorage(new File(System.getProperty("java.io.tmpdir"), "ui-stats.dl4j"));
//        int listenerFrequency = 1;
//        net.setListeners(new StatsListener(statsStorage, listenerFrequency));
//
//        //Attach the StatsStorage instance to the UI: this allows the contents of the StatsStorage to be visualized
//        uiServer.attach(statsStorage);

        net.setListeners(new ScoreIterationListener(1));

        //Start training:
        net.fit(trainData);

        //Finally: open your browser and go to http://localhost:9000/train
    }

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/vincenzo/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-simple/1.7.25/8dacf9514f0c707cbbcdd6fd699e8940d42fb54e/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/vincenzo/.m2/repository/ch/qos/logback/logback-classic/1.1.7/logback-classic-1.1.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
[main] INFO org.nd4j.linalg.factory.Nd4jBackend - Loaded [CpuBackend] backend
[main] INFO org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for linear algebra: 1
[main] WARN org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory - *********************************** CPU Feature Check Warning ***********************************
[main] WARN org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory - Warning: Initializing ND4J with Generic x86 binary on a CPU with AVX/AVX2 support
[main] WARN org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory - Using ND4J with AVX/AVX2 will improve performance. See deeplearning4j.org/cpu for more details
[main] WARN org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory - Or set environment variable ND4J_IGNORE_AVX=true to suppress this warning
[main] WARN org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory - *************************************************************************************************
[main] INFO org.nd4j.nativeblas.Nd4jBlas - Number of threads used for OpenMP BLAS: 1
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Backend used: [CPU]; OS: [Linux]
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Cores: [12]; Memory: [3.8GB];
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Blas vendor: [OPENBLAS]
[main] INFO org.deeplearning4j.nn.multilayer.MultiLayerNetwork - Starting MultiLayerNetwork with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 0 is 2.4596229357207373
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 1 is 2.3847859273214307
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 2 is 2.3786044546229355
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 3 is 2.2900389586334073
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 4 is 2.2641599509936485
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 5 is 2.248210364763382
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 6 is 2.055237813208146
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 7 is 2.020907858175258
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 8 is 1.9386540538316195
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 9 is 1.8051564138398883
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 10 is 1.6514443572336142
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 11 is 1.572192444812278
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 12 is 1.4436239291466388
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 13 is 1.0696110265790622
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 14 is 1.1613952072641363
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 15 is 0.8913948249016883
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 16 is 1.1365792501164045
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 17 is 0.9962026023624351
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 18 is 0.8859268674927635
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 19 is 0.8205417612492297
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 20 is 0.708769747733592
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 21 is 0.7118142633597823
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 22 is 0.6540580576891457
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 23 is 0.6822972891144343
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 24 is 0.7132086500773225
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 25 is 0.7072300808610156
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 26 is 0.6134240954977738
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 27 is 0.7122743342806406
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 28 is 0.7319481317854911
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 29 is 0.7265425042849846
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 30 is 0.5727326479706499
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 31 is 0.5091023160649354
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 32 is 0.49789919441507197
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 33 is 0.538915619645431
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 34 is 0.4144490042284773
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 35 is 0.444938702184474

Great. Now enable UI listener again, but keep score listener too.

@raver119 now I see the training going on (sure enough it has always been working on background also before), but still the url is unreachable. Seems like the UI stuff doesn't start or something is blocking it on my system (firewall is off however).

So, there's no deadlock, and probably there never was any deadlock... Probably that's just operation that takes significant time and you just tend to see it more often there. We'll check it out.

@vincenzocaselli OK, can you try running this exact same example through maven or intellij instead of Eclipse IDE?
looking back over what you've posted this looks off
SLF4J: Found binding in [jar:file:/home/vincenzo/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-simple/1.7.25/8dacf9514f0c707cbbcdd6fd699e8940d42fb54e/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

There shouldn't be anything gradle-related on the classpath in the examples. Suggests that maybe Eclipse IDE is causing problems with logging, and we're simply not seeing some other output (like an error saying port 9000 is already used, or something of that nature).
Vertx has definitely started according to the thread dump you posted earlier.

@raver119 wow! A step forward: since I would have some problem using IntelliJ right now and don't know exactly how to use directly maven to run I followed your hint.
I therefore cleared my home .gradle/caches/ folder and now I see something different:

the console is messaging in black color (before was almost all in red), see below its content
the url is reachable!!! But it shows just "Resource not found"

o.n.l.f.Nd4jBackend - Loaded [CpuBackend] backend
o.n.n.NativeOpsHolder - Number of threads used for linear algebra: 1
o.n.l.c.n.CpuNDArrayFactory - *********************************** CPU Feature Check Warning ***********************************
o.n.l.c.n.CpuNDArrayFactory - Warning: Initializing ND4J with Generic x86 binary on a CPU with AVX/AVX2 support
o.n.l.c.n.CpuNDArrayFactory - Using ND4J with AVX/AVX2 will improve performance. See deeplearning4j.org/cpu for more details
o.n.l.c.n.CpuNDArrayFactory - Or set environment variable ND4J_IGNORE_AVX=true to suppress this warning
o.n.l.c.n.CpuNDArrayFactory - *************************************************************************************************
o.n.n.Nd4jBlas - Number of threads used for OpenMP BLAS: 1
o.n.l.a.o.e.DefaultOpExecutioner - Backend used: [CPU]; OS: [Linux]
o.n.l.a.o.e.DefaultOpExecutioner - Cores: [12]; Memory: [3.8GB];
o.n.l.a.o.e.DefaultOpExecutioner - Blas vendor: [OPENBLAS]
o.d.n.m.MultiLayerNetwork - Starting MultiLayerNetwork with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]
o.d.u.VertxUIServer - Deeplearning4j UI server started at: http://localhost:9000
o.d.u.VertxUIServer - StatsStorage instance attached to UI: FileStatsStorage(/tmp/ui-stats.dl4j)
o.d.o.l.ScoreIterationListener - Score at iteration 0 is 2.4596229357207373
o.d.o.l.ScoreIterationListener - Score at iteration 1 is 2.3847859273214307
o.d.o.l.ScoreIterationListener - Score at iteration 2 is 2.3786044546229355
o.d.o.l.ScoreIterationListener - Score at iteration 3 is 2.2900389586334073

@vincenzocaselli for maven execution, use this:
cd deeplearning4j-examples
mvn package
cd dl4j-examples
java -cp target/dl4j-examples-1.0.0-beta6-bin.jar org.deeplearning4j.examples.userInterface.UIExample

@raver119 done with maven direct. Same result ... that is: url http://localhost:9000/train is unreachable.
But I notice the message:
o.d.u.VertxUIServer - Deeplearning4j UI server started at: http://localhost:9000
so tried removing /train and ... it worked!
The final target Url is indeed http://localhost:9000/train/overview
but anyway just /train doesn't work.
Thank you @raver119 !!!
I tried again from Eclipse and it is working also there.
Now I am trying to apply that to another project where I am using DL4J and where the problem of hanging is still there. I'll try to compare the two project for the differences and let you know if I can see the DL4J UI also there.
Thank you!

OK, great. I'll close this. :+1:
Only thing I think we can do here is provide a simple redirect from /train to /train/overview. I thought we already had that, but apparently not.
I'll open a separate issue for that.

Hi @AlexDBlack, @raver119
I am adding a comment here since the problem at the root of this post is back again.
Let me know if I should better open another issue and I'll do; for now I'll explain here below.
The problem was solved for the DL4L UIExample, but it is still present for a brand new project.
In fact I started (with Eclipse IDE, but don't think this can make a difference) a brand new Maven project.
Then added the following dependencies
 		<dependency>
			<groupId>org.nd4j</groupId>
			<artifactId>nd4j-native-platform</artifactId>
			<version>1.0.0-beta6</version>
		</dependency>
		<dependency>
			<groupId>org.deeplearning4j</groupId>
			<artifactId>deeplearning4j-core</artifactId>
			<version>1.0.0-beta6</version>
		</dependency>
  		<dependency>
			<groupId>org.deeplearning4j</groupId>
			<artifactId>deeplearning4j-ui</artifactId>
			<version>1.0.0-beta6</version>
		</dependency>

Then I created a very simple class for traning and testing against MNIST (here is the complete source code http://www.rcp-vision.com/build-your-first-neural-network-with-eclipse-deeplearning4j).
The code worked successfully.
Then I introduced UI stuff, but the URLs http://localhost:9000 or http://localhost:9000/train/overview
are unreachable and the console shows no errors.
I made many tests after comparing this brand new project and the one in deeplearning4j-examples (UIExample, it's pom.xml and it's parent pom.xml).
In one of these attempt I added the following dependency
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>1.1.7</version>
        </dependency>

With this simple step the console showed much more messages.
I am attaching these messages below in the hope that the error showed may help you to understand why the UI stuff is not working.
Thank you very much
17:05:32.420 [main] DEBUG io.netty.util.internal.PlatformDependent0 - -Dio.netty.noUnsafe: false
17:05:32.421 [main] DEBUG io.netty.util.internal.PlatformDependent0 - Java version: 13
17:05:32.421 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
17:05:32.421 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
17:05:32.422 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
17:05:32.424 [main] DEBUG io.netty.util.internal.PlatformDependent0 - direct buffer constructor: unavailable
java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled
	at io.netty.util.internal.ReflectionUtil.trySetAccessible(ReflectionUtil.java:31)
	at io.netty.util.internal.PlatformDependent0$4.run(PlatformDependent0.java:224)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
	at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:218)
	at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:272)
	at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:92)
	at io.netty.channel.nio.NioEventLoop.newTaskQueue0(NioEventLoop.java:285)
	at io.netty.channel.nio.NioEventLoop.newTaskQueue(NioEventLoop.java:156)
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:138)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:138)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:78)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:73)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:60)
	at io.vertx.core.net.impl.transport.Transport.eventLoopGroup(Transport.java:148)
	at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:143)
	at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:27)
	at io.vertx.core.Vertx.vertx(Vertx.java:75)
	at org.deeplearning4j.ui.VertxUIServer.getInstance(VertxUIServer.java:87)
	at org.deeplearning4j.ui.api.UIServer.getInstance(UIServer.java:58)
	at org.deeplearning4j.ui.api.UIServer.getInstance(UIServer.java:42)
	at it.rcpvision.dl4j.workbench.Step1MnistMLP.main(Step1MnistMLP.java:82)
17:05:32.424 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
17:05:32.425 [main] DEBUG io.netty.util.internal.PlatformDependent0 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable
java.lang.IllegalAccessException: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @42a9a63e
	at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:376)
	at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:642)
	at java.base/java.lang.reflect.Method.invoke(Method.java:559)
	at io.netty.util.internal.PlatformDependent0$6.run(PlatformDependent0.java:334)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
	at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:325)
	at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:272)
	at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:92)
	at io.netty.channel.nio.NioEventLoop.newTaskQueue0(NioEventLoop.java:285)
	at io.netty.channel.nio.NioEventLoop.newTaskQueue(NioEventLoop.java:156)
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:138)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:138)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:78)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:73)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:60)
	at io.vertx.core.net.impl.transport.Transport.eventLoopGroup(Transport.java:148)
	at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:143)
	at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:27)
	at io.vertx.core.Vertx.vertx(Vertx.java:75)
	at org.deeplearning4j.ui.VertxUIServer.getInstance(VertxUIServer.java:87)
	at org.deeplearning4j.ui.api.UIServer.getInstance(UIServer.java:58)
	at org.deeplearning4j.ui.api.UIServer.getInstance(UIServer.java:42)
	at it.rcpvision.dl4j.workbench.Step1MnistMLP.main(Step1MnistMLP.java:82)
17:05:32.425 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): unavailable
17:05:32.426 [main] DEBUG io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available
17:05:32.438 [main] DEBUG io.netty.util.internal.PlatformDependent - maxDirectMemory:<PHONE_NUMBER> bytes (maybe)
17:05:32.438 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
17:05:32.438 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
17:05:32.439 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.maxDirectMemory: -1 bytes
17:05:32.439 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.uninitializedArrayAllocationThreshold: -1
17:05:32.439 [main] DEBUG io.netty.util.internal.CleanerJava9 - java.nio.ByteBuffer.cleaner(): available
17:05:32.440 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
17:05:32.446 [main] DEBUG io.netty.util.internal.PlatformDependent - org.jctools-core.MpscChunkedArrayQueue: available
17:05:32.476 [main] DEBUG io.netty.resolver.dns.DefaultDnsServerAddressStreamProvider - Default DNS servers: [/<IP_ADDRESS>:53] (sun.net.dns.ResolverConfiguration)
17:05:32.480 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv4Stack: false
17:05:32.481 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv6Addresses: false
17:05:32.482 [main] DEBUG io.netty.util.NetUtil - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
17:05:32.483 [main] DEBUG io.netty.util.NetUtil - /proc/sys/net/core/somaxconn: 128
17:05:33.096 [main] INFO org.deeplearning4j.ui.VertxUIServer - StatsStorage instance attached to UI: FileStatsStorage(/tmp/ui-stats.dl4j)

@vincenzocaselli so those stack traces you posted are likely irrelevant. You'll note they are debug level in netty, checking for certain JDK features - with properly configured logging (i.e., a logback.xml in the project resources directory like the link below) you wouldn't even see that.
https://github.com/eclipse/deeplearning4j-examples/blob/master/standalone-sample-project/src/main/resources/logback.xml
OK, three things we'll need here
(a) The exact code you're using (with your network, listener configuration, UI-related code etc)
(b) The full output when running that code - I assume there's more than you posted? - in a gist - gist.github.com
(c) Confirmation of exactly how you are building and launching your project (via eclipse IDE? via maven + command line?)
I'm a little skeptical that it's something on our end, given we use the UI pretty frequently ourselves and haven't had any other reports of issues with it. But I should be able to help debug this once I have those details

Hi @AlexDBlack ,
here they are:
(a) Java class & pom.xml
Step1MnistMLP.java.txt
pom.xml.txt
(b)
https://gist.github.com/vincenzocaselli/7fe545ed44551101228142822b7d95a8
(c)
o.s. = Linux Arch 64bits (Manjaro Deepin 18.0.2)
Java JDK 13 (tried both Oracle and OpenJDK with same results)
Eclipse 2019-12 (Note: I use this system for daily work with Java+Eclipse for about a year now without any problem)
Then in short (here you can find more details and images http://www.rcp-vision.com/build-your-first-neural-network-with-eclipse-deeplearning4j):

File, New, Project, Maven Project
click to skip archetype selection, give a name for a Group Id and an Artifact Id
open pom.xml and add block dependencies as in attachment
create Java class as in attachment
select Java class, right click, Run as Java Application

Please also note that without UI stuff the execution time is about  80 seconds, while here is 787
Let me know if you need other details.
Thank you

@vincenzocaselli
I've tried to reproduce your problem, but it works on all machines I've tried.
Can you step through VertxUIServer.start and see if something hinders it getting to the final log line?

Hi @treo !
wow, just by adding freemarker dependency make it work instantly!!!
Thank you, but what was happening?

The problem is that the UI wants a newer version of freemarker, but something makes it so that an older version is included instead.
You're basically overwriting the freemarker version of the dependency that is already added.
It is weird that the error message for it gets silently discarded.

Hi guys, I'm having the same issue as @vincenzocaselli
I downloaded the full repo from https://github.com/eclipse/deeplearning4j-examples.git then ran it without any IDE for troubleshooting. Using maven on the command line as @AlexDBlack said, and the result is a little bit weird.
Despite the net running and even end training after some time (no errors or anything) the webhost at localhost:9000 never opens, i tried all the links you guys suggested, with no success.
Following what @treo said, I noticed that freemarker now came by default, anyway I tried 2.3.30 version as well with no luck.
I tried in windows 10 x64 on java 8 and 14.
And on Linux x64 on java 11.
When running mvn package some warnings show on the terminal:
[WARNING]   - META-INF.versions.9.module-info [WARNING] maven-shade-plugin has detected that some class files are  present in two or more JARs. When this happens, only one  single version of the class is copied to the uber jar.  Usually this is not harmful and you can skip these warnings, otherwise try to manually exclude artifacts based on mvn dependency:tree -Ddetail=true and the above output.  See http://maven.apache.org/plugins/maven-shade-plugin/
Running it with java -cp dl4j-examples-1.0.0-beta7-shaded.jar org.deeplearning4j.examples.quickstart.features.userinterface.BasicUIExample I get:
o.n.l.f.Nd4jBackend - Loaded [CpuBackend] backend o.n.n.NativeOpsHolder - Number of threads used for linear algebra: 6 o.n.l.c.n.CpuNDArrayFactory - *********************************** CPU Feature Check Warning *********************************** o.n.l.c.n.CpuNDArrayFactory - Warning: Initializing ND4J with Generic x86 binary on a CPU with AVX/AVX2 support o.n.l.c.n.CpuNDArrayFactory - Using ND4J with AVX/AVX2 will improve performance. See deeplearning4j.org/cpu for more details o.n.l.c.n.CpuNDArrayFactory - Or set environment variable ND4J_IGNORE_AVX=true to suppress this warning o.n.l.c.n.CpuNDArrayFactory - ************************************************************************************************* o.n.n.Nd4jBlas - Number of threads used for OpenMP BLAS: 6 o.n.l.a.o.e.DefaultOpExecutioner - Backend used: [CPU]; OS: [Windows 10] o.n.l.a.o.e.DefaultOpExecutioner - Cores: [6]; Memory: [2,0GB]; o.n.l.a.o.e.DefaultOpExecutioner - Blas vendor: [OPENBLAS] o.d.n.m.MultiLayerNetwork - Starting MultiLayerNetwork with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE] o.d.u.VertxUIServer - Deeplearning4j UI server started at: http://localhost:9000 o.d.u.VertxUIServer - StatsStorage instance attached to UI: FileStatsStorage(C:\Users\Murilo\AppData\Local\Temp\ui-stats.dl4j) o.d.u.VertxUIServer - StatsStorage instance attached to UI: InMemoryStatsStorage(uid=a304c5ff)
(I dont know why the line endings are not working so I put it on pastebin as well: link
and then after some time it prints this:
o.d.u.VertxUIServer - Deeplearning4j UI server is auto-stopping after thread (name: main) died. o.d.u.VertxUIServer - Deeplearning4j UI server stopped.
This warning and output is consistent over the OS's I tested.
I really don't know what is happening, and I want to help to sort this out. Any additional information will be sent as soon as possible.

I've fixed the line endings for you. You've got to use tripple backticks to get it to work like that instead of the single backticks you've used.
What you are seeing is intended behavior in the current version. When the training is finished, it also stops the UI.
If you want to keep it running, you just have to keep the main thread alive. For example you could read from stdin and wait for a specific key to be entered or something like that, the easiest way would be to just let the thread sleep if you just want to look around for a few minutes.

Thanks for the line endings, I didn't know that.
And also by quick answer, appreciate it.
My problem is not that it stops running, I'm saying that I can't access the UI in the browser as expected. I just posted the full output to document it all.
Actually I have tried to add a Thread.sleep(999999) in the end, and it won't let the thread die, but I still cant access the web UI.
Sorry for not specifying that before.
Is there any other thing to try?

Well, that led me on a wild goose chase... Apparently vertx swallows exceptions when it is configured to support HTTP2 upgrades....
That weird behavior (Port is open, but it just hangs when connecting) is yet again a dependency version clash.
You'll need to add the following to the dependencyManagement section:
            <dependency>
                <groupId>io.netty</groupId>
                <artifactId>netty-buffer</artifactId>
                <version>4.1.48.Final</version>
            </dependency>

I can't even describe how awesome you are. That's a Christmas gift. I fought so hard for so long with this but I didn't have the knowledge to solve it. Thank you.

Hi,
i have the same problem with version beta7 but when i look in the pom.xml i see @treo suggestions to add depency but it did not work.
i see now this issue closed, but i got the problem. I am lost...
thanks by advance

Hi,
i have the same problem with version beta7 but when i look in the pom.xml i see @treo suggestions to add depency but it did not work.
i see now this issue closed, but i got the problem. I am lost...
thanks by advance

@krisdesalpes  mind opening a new issue with  a bit more context? Thanks!

@krisdesalpes  mind opening a new issue with  a bit more context? Thanks!

Just wanted to mention that in order to fix this "/train url is unreachable" issue in the latest codebase you need to have both netty-common and netty-buffer dependencies set to 4.1.48.Final (As per @treo 's comment).  So your dependencyManagement tag will look in the following way:
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.freemarker</groupId>
                <artifactId>freemarker</artifactId>
                <version>2.3.29</version>
            </dependency>
	    <dependency>
                <groupId>io.netty</groupId>
                <artifactId>netty-common</artifactId>
                <version>4.1.48.Final</version>
            </dependency>
	     <dependency>
                <groupId>io.netty</groupId>
                <artifactId>netty-buffer</artifactId>
                <version>4.1.48.Final</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

Env: Eclipse Version: 2019-12 (4.14.0) / OS: Linux, v.5.4.0-58-generic, x86_64 / mvn: 3.6.3 / java version: 15.0.1
In the current version of dl4j-examples/pom.xml "netty-buffer" is missing. This is where it comes from:
datavec-local->(datavec-arrow->arrow-vector->netty-buffer:4.1.22 | datavec-arrow->arrow-memory->netty-buffer:4.1.22)


[question] How to set options to decoder?

Hi, sorry for the question, but I couldn't find the answer in the documentation.
I'm looking to set a libavcodec option, this one in particular but couldn't see how. I notice there's a codec_context.options dictionary, but my quick testing didn't seem to do it (also not sure if it's the same thing, as it seems to go to the last arg in avcodec_open2). Can this be currently done?
Unfortunately, without this option, some videos are horribly broken.
Also, is there an interface to catch decoder warnings? It seems like these just get dumped to stderr - this can be redirected, but was wondering if there was a nicer way of doing it built-in.
My plan of action being that if specific warnings are generated (and the video has the right properties), retry decoding it using the flag above.
Thanks for reading and thanks for PyAV!

I've spent some time looking at this now, and this is what I've found:
The x264_build option is only available in libavcodec58, but PyAV doesn't seem to compile against it (any interest in supporting it?). Forcing the build through (by essentially deleting lines giving compile errors), I'm able to set the option with existing functionality in PyAV, something like the following seems to work for me:
container = av.open('video.mkv', options={'x264_build': '150'})
video = container.streams[0]

# specifying x264_build above doesn't seem to be necessary, but does suppress an error; the following is what seems to be needed to get it to work
video.close()
video.codec_context.options = {'x264_build':'150'}
video.open()

So I guess the issues I'm now interested in:

would you be able to add support for compiling against libav*58?
is there any way to capture warnings without redirecting stderr?

The options are there, but currently quite awkward to work with. Seems like you forced though it.
There is an FFmpeg 4.0 now?! I'd like to support that. I think our tests currently only pass in 3.3, so we're behind a bit.
About the warnings: FFmpeg's logs are passed through the Python logging system, so you could add a custom log handler to catch them. The feature/dump_format branch has some machinery to directly capture FFmpeg logs. That could be merged in without a lot of trouble.

That sounds great! Thanks for the info.

I've made a pair of tickets to continue the tangential discussions, and am closing this. Re-open if we haven't addressed what we need to. Thanks!


How to run build-opencv ?

I have install npm install opencv-build, but how to launch opencv-build-npm ? build-opencv command not found
R:\gitproj\openCV\node_modules\@u4\opencv-build\lib\BuildEnv.js:92
                throw Error(`No build found in ${this.rootDir} you should launch opencv-build-npm once`);
                ^

Error: No build found in R:\gitproj\openCV\node_modules\@u4\opencv-build you should launch opencv-build-npm once
    at new OpenCVBuildEnv (R:\gitproj\openCV\node_modules\@u4\opencv-build\lib\BuildEnv.js:92:23)
    at new OpenCVBuilder (R:\gitproj\openCV\node_modules\@u4\opencv-build\lib\OpenCVBuilder.js:49:24)
    at getOpenCV (R:\gitproj\openCV\node_modules\@u4\opencv4nodejs\lib\cvloader.js:16:21)
    at loadOpenCV (R:\gitproj\openCV\node_modules\@u4\opencv4nodejs\lib\opencv4nodejs.js:9:43)
    at Object.<anonymous> (R:\gitproj\openCV\node_modules\@u4\opencv4nodejs\lib\opencv4nodejs.js:24:12)
    at Module._compile (node:internal/modules/cjs/loader:1099:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:999:19)

Node.js v17.8.0

OpenCV version (example 3.4.1): 4.5.5
OS: Windows 10?

opencv-build-npm command not found, that was the problem. I using justadude version, good enough for now

small nodeJS lesson:
NodeJS scripts are installed in node_modules/.bin by your package manager usually npm.
so include ./node_modules/.bin in your $PATH environment variable, or prefix opencv-build-npm by ./node_modules/.bin/
All command present in the "script" part of your package.json are called including ./node_modules/.bin in they $PATH.
I should include that in the error message.


Crash In FeatureMatcher

Hi, Jonathan
I follow the tutorial in the VillageDataset and all things go well until executing TestLocalizer. The program crash down on line 88 of featurematcher.cpp. Could you please send all the xml files generated by above steps to me? My e-mail address is<EMAIL_ADDRESS>Warmest regards.

The above crash caused by null pointer of myfeature->descriptor.
After fixing the bug. All the executable work, but RenderTracker dosen't show any tracking effect even when I load an obj file. I have checked locaized.xml and tracked.xml and find that the rotation and translation values of most nodes are 0.0000. Strange....

The bug has been fixed in commit b502e7bc6e9711844bce4ee8ccb64ffe264e6032 .


fix typo

Add missing 's'

What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Release note:

/ok-to-test
/release-note-none
/override dco
/lgtm
/approve
/cherrypick master


fix: ESLint issues in botbuilder-dialogs

Addresses # 4684
#minor
Description
This PR fixes the new errors and warnings incoming in the latest version of ESLint packages in botbuilder-dialogs.
Specific Changes

Removed eslint.config.cjs file.
Fixed ESLint issues by running --fix flag.
Added missing documentation and removed unnecessary spaces.
Added some skips to rules that don't follow the current logic.
Removed duplicated unit test and combined logic of these two.

Testing
The following image shows the lint command running without problems in botbuilder-dialogs.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>1
Details

18 of 18 (100.0%)  changed or added relevant lines in 14 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.3%) to 84.355%

Totals

Change from base Build<PHONE_NUMBER>2:
-0.3%

Covered Lines:
20432

Relevant Lines:
23063

💛  - Coveralls

Promoted /4832


Create Akamaidummy to test pr validation

Dummy PR to test the PR Validations for Data Connector
Fixes #
Proposed Changes

Removing as this PR is to test Data Connector PR Validations

Removing as this PR is to test Data Connector PR Validations


Screenshot extension

I would like to implement some sort of hook or extension to take a screenshot everytime there is a url change. If the application goes from a.html to b.html I would like to take a photo of b.html with a filename corresponding to the page url. Is this possible? If so where would I start?

Hi Alex,
I don't see a way how it can be done with the current testcafe version, this requires some additional implementation in our API.
Could you please tell us why do you need it? How are you going to use it in your project? With this information, we do our best to find a solution for you.

I'm trying to map the different views in the application to another set of logs. I'd like to sync screenshots to the logs I have which are segmented by url. I could augment my scripts to to do this I was just curious if I could implement a hack locally.

Thanks for clarification. For now TestCafe doesn't provide functionality to add such hook.
I'm not 100% sure that this is what you need, but maybe the following approach will be useful for you:
fixture `Get screenshots`;

const urls = ['http://google.com', 'http://gmail.com'];

for (let i = 0; i < urls.length; i++) {
    const url = urls[i];

    test('Get screenshot', async t => {
        await t.navigateTo(url);

        const pageUrl = await t.eval(() => window.location.href);
        const fileName = pageUrl.replace(/:|\/|\./g, '_');

        await t.takeScreenshot(`${fileName}.png`);
    });
}

Or maybe the following:
import { Selector } from 'testcafe';

fixture `Get screenshots`
    .page `google.com`;

let lastPageUrl = '';

async function checkUrlAndTakeScreenshot (t) {
    const pageUrl = await t.eval(() => window.location.href);

    if(pageUrl !== lastPageUrl) {
        lastPageUrl = pageUrl;

        const fileName = pageUrl.replace(/:|\/|\./g, '_');

        await t.takeScreenshot(`${fileName}.png`);
    }
}

test('Get screenshot', async t => {
    // add the takeScreenshot function after each action
   await t.click('body');
   await checkUrlAndTakeScreenshot(t);
   await t.click(Selector('a').withText('Images'));
   await checkUrlAndTakeScreenshot(t);
   // ...
});

The second solution looks not pretty but maybe it'll ok for your task


Support default dtype in sentiment example's recursive minibatch version

Rel #6168.
This PR fixes the sentiment example's recursive minibatch version, train_recursive_minibatch.py, to support the default dtype feature so that it works in FP16 mode.

Jenkins, test this please.

Jenkins CI test (for commit 926480d8ffb7e44dc38e57b158b8c6f016fbaf3d, target branch master) failed with status FAILURE.

Retest. Jenkins, test this please.

Jenkins CI test (for commit 926480d8ffb7e44dc38e57b158b8c6f016fbaf3d, target branch master) failed with status FAILURE.

Jenkins, test this please.

Jenkins CI test (for commit 926480d8ffb7e44dc38e57b158b8c6f016fbaf3d, target branch master) failed with status FAILURE.

Jenkins, test this please.

Jenkins CI test (for commit 926480d8ffb7e44dc38e57b158b8c6f016fbaf3d, target branch master) failed with status FAILURE.

Jenkins failure looks unrelated.


value_iteration bugs

The value_iteration solution algorithm has a couple of issues:

Extra dr parameter is documented but not used in code: https://github.com/EconForge/dolo.py/blob/master/dolo/algos/value_iteration.py#L31
The solution algorithm requires the existence of controls_lb and controls_ub functions ever if there is a separately defined CartesianDomain for the model controls: https://github.com/EconForge/dolo.py/blob/master/dolo/algos/value_iteration.py#L44-L45

First item is indeed a regression. There should be an argument dr0 (resp drv0), to be used as initial guess for the decision rule (resp the value function).
I'm not sure about the second one. Right now, what is defined as a CartesianDomain is the set of boundaries for the endogenous spaces. But the domain for the domain we optimize on is always represented by two functions controls_ub and controls_lb. These can be specified in two ways: either by adding a controls_lb (resp ub) set of equations or by adding complementarity conditions to the arbitrage equations.

Oh, I apologize. I misunderstood the role of domain.
I see now why the upper and lower bounds on controls need to be dynamic.


fix: PR borders now match color scheme

What's the issue:
PR borders for comments, events, and etc were white. They are now styled.
What's fixed:
PR borders are now mostly fixed and commit forms.

Thanks for the PR. Could you provide some pictures of the before and after?


!show et ; fails in distribute mode / Byzer-notebook fails to start

When we run !show et in distribute mode (e.g. based on yarn/k8s), this macro function may fails. This will cause
the byzer-notebook fails to start because the ETService relies on this macro function to get all ets.

Verified in hadoop 2.7.1 (HDP 2.4.0).

Close as it's pr is merged


Automate integration tests after bundle build

Description
This PR updates the existing integ test jenkins workflow, and adds a step in the distribution build workflow to trigger the tests after successfully building. More specifically:

moves and updates existing test workflow (jenkins/test/testsuite/Jenkinsfile)  to jenkins/opensearch/integ-test.jenkinsfile. Params were modified to try to prevent link-building logic which has caused broken builds in the past.
adds a groovy script runIntegTests.groovy to call the integ test workflow using passed params + env variables
modifies jenkins/opensearch/distribution-build.jenkinsfile to run x64 and arm64 integ tests in parallel

NOTE: this PR is a WIP for now. Things to wrap up before it is ready:

[x] add script info and other updates in related READMEs
[x] clean up old integ test workflow which is deprecated
[x] clean up related test workflow logic that uses the old integ test workflow (see jenkins/test/orchestrator)
[ ] adding publishNotifications() back
[ ] change job in runIntegTests.groovy and jenkins/test/orchestrator/Jenkinsfile to not reference a playground job

Screenshot - distribution build + integ tests

Screenshot - integ test job

Issues Resolved
Closes #1114
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1429 (8627233) into main (a7a7e96) will increase coverage by 0.07%.
The diff coverage is 60.00%.

@@             Coverage Diff              @@
##               main    #1429      +/-   ##
============================================
+ Coverage     94.13%   94.20%   +0.07%
- Complexity       11       12       +1
============================================
  Files           139      141       +2
  Lines          3086     3074      -12
  Branches          8       10       +2
============================================
- Hits           2905     2896       -9
+ Misses          173      169       -4
- Partials          8        9       +1

Impacted Files
Coverage Δ

src/jenkins/InputManifest.groovy
62.50% <33.33%> (+0.43%)
:arrow_up:

tests/jenkins/jobs/RunIntegTests_arm64_Jenkinsfile
100.00% <100.00%> (ø)

tests/jenkins/jobs/RunIntegTests_x64_Jenkinsfile
100.00% <100.00%> (ø)

src/system/temporary_directory.py
83.87% <0.00%> (-12.91%)
:arrow_down:

src/system/os.py
92.30% <0.00%> (-7.70%)
:arrow_down:

src/jenkins/Messages.groovy
75.00% <0.00%> (+6.57%)
:arrow_up:

src/jenkins/BuildManifest.groovy
100.00% <0.00%> (+26.66%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7a7e96...8627233. Read the comment docs.

FYI. I have a task to trigger OpenSearch Dashboards integ test post bundle build https://github.com/opensearch-project/opensearch-build/issues/1372  We can sync up to see if we can leverage each other's work.

Let's have test cases for all the jenkinsfile that are added.

distribution-build.jenkinsfile
integ-test.jenkinsfile

Tests have been added for integ-test.jenkinsfile via runIntegTests.groovy, and is tested via TestRunIntegTests.groovy.
distribution-build.jenkinsfile already exists, and is essentially a compilation of a bunch of these groovy scripts that already have associated tests. Since there is no related groovy script for this one, I don't know how else to test.
I'm not aware of any other ways to test these in an automated way, do you have any suggestion?

Let's have test cases for all the jenkinsfile that are added.

distribution-build.jenkinsfile
integ-test.jenkinsfile

Tests have been added for integ-test.jenkinsfile via runIntegTests.groovy, and is tested via TestRunIntegTests.groovy. distribution-build.jenkinsfile already exists, and is essentially a compilation of a bunch of these groovy scripts that already have associated tests. Since there is no related groovy script for this one, I don't know how else to test.
I'm not aware of any other ways to test these in an automated way, do you have any suggestion?

TestRunIntegTests.groovy should only test runIntegTests.groovy like a unit test. integ-test.jenkinsfile is a jenkins job that may or may not use more functionalities and libraries in the future apart from runIntegTests.groovy and therefore should be tested separately.
We can add tests for the intire job ina very similar way to what we do for our libraries. You can refer to this as reference

Let's have test cases for all the jenkinsfile that are added.

distribution-build.jenkinsfile
integ-test.jenkinsfile

Tests have been added for integ-test.jenkinsfile via runIntegTests.groovy, and is tested via TestRunIntegTests.groovy. distribution-build.jenkinsfile already exists, and is essentially a compilation of a bunch of these groovy scripts that already have associated tests. Since there is no related groovy script for this one, I don't know how else to test.
I'm not aware of any other ways to test these in an automated way, do you have any suggestion?

TestRunIntegTests.groovy should only test runIntegTests.groovy like a unit test. integ-test.jenkinsfile is a jenkins job that may or may not use more functionalities and libraries in the future apart from runIntegTests.groovy and therefore should be tested separately.
We can add tests for the intire job ina very similar way to what we do for our libraries. You can refer to this as reference. I hope this helps

Link seems to be broken. I'm still unsure what you mean

Let's have test cases for all the jenkinsfile that are added.

distribution-build.jenkinsfile
integ-test.jenkinsfile

Tests have been added for integ-test.jenkinsfile via runIntegTests.groovy, and is tested via TestRunIntegTests.groovy. distribution-build.jenkinsfile already exists, and is essentially a compilation of a bunch of these groovy scripts that already have associated tests. Since there is no related groovy script for this one, I don't know how else to test.
I'm not aware of any other ways to test these in an automated way, do you have any suggestion?

TestRunIntegTests.groovy should only test runIntegTests.groovy like a unit test. integ-test.jenkinsfile is a jenkins job that may or may not use more functionalities and libraries in the future apart from runIntegTests.groovy and therefore should be tested separately.
We can add tests for the intire job ina very similar way to what we do for our libraries. You can refer to this as reference. I hope this helps

Link seems to be broken. I'm still unsure what you mean

https://github.com/opensearch-project/opensearch-build/blob/2ca4c8d849563352003c7bf37dacaebec868f1a9/tests/jenkins/TestSignArtifacts.groovy#L43-L60

Are we using/will we ever use jenkins/test/orchestrator/Jenkinsfile?
I'm good with this putting back notifications and removing any Tyler-references ;)

I think the framework is worth leaving for now, to help with future work at getting the entire test orchestrator running. I imagine we will want the distribution build and/or other test pipelines to be able to kick off all integ/perf/bwc tests
cc @peternied

my vote is to nuke anything unused, and move it to a link to a specific commit for anyone who wants to resurrect it when it's ready to be used

my vote is to nuke anything unused (and doesn't work)

+1; however, I am comfortable with making this a new PR focused on removing these components and fixing the documentation

my vote is to nuke anything unused (and doesn't work)

+1; however, I am comfortable with making this a new PR focused on removing these components and fixing the documentation

Got it, makes sense. I'll update #1447 with some details and resolve in a separate PR

We did a deep dive today, and that resulted in some things coming out. I'd recommend doing some smaller PRs that would make this more manageable.

To break this up easier, I'm thinking of splitting this up into 3 parts / PRs:

Revive old integ test workflow and get a working version via jenkins/opensearch/integ-test.jenkinsfile (this PR)
Hook new integ test workflow into distribution build workflow (jenkins/opensearch/distribution-build.jenkinsfile) (future PR)
Remove legacy integ-test jenkins code in jenkins/test/ and update related READMEs (future PR)


Refactor backend side

Mainly refactoring index.ts shorter. By refactoring I learn easier and faster the whole code base :) Also I have formatted some files with prettier so everything is consistent 🎉
I guess you are previously Java coder right @dolanmiu ? ;)

Yes previously Java, then C#, and now Angular/Node... wait I am a "full-stack" dev at my current job actually, so I do Java lol
How did you know?

I am just guessing from your code style :P Lots of classes. That usually means Java background. I more like a functional programmer. Have you done any of that?

Can we use 4 spaces? I prefer this over 2

I have set prettier to 4 spaces now. I am more like 2 spaces guys since it is more compact 😎

Ah yeah, I like splitting up code as much as possible, less to focus on, and easier to test. Perhaps it came from my Java background.
I never understand why developers create 700 line files.
And yeah I've done functional programming, my newer stuff is quite functional. I tend to do a hybrid OOP + functional these days.
I have wrote some horrendus long functional snippets, when I got too excited about functional 😂

Everyone at my work likes 2 spaces, and the internet seems to like 2 spaces too, but I find it harder to read cos everything is squashed LOL! I think I am alone on this one

Yeah people are different :P


Options Arg Refactor and more (see below)...

Refactored to support an options object instead of individual params to the factory (Breaking change!).
Refactored test/coverage running methods to be consistent in how they are invoked from gulp tasks.
Added helper methods for All, Behavior and Integration tests.
Added common gulpfile module for the typical 80% use-case(s).
Added a project self-referencing gulpfile.
Added jshint and jscs tasks.

@arobson not going to lie, it's some breaking changes and you might hate it. That's OK if so. If you need anything changed, lemme know.

Looking great!


Start-WBHyperVRecovery does not want an array for -BackupSet

This does not seem to work:
https://github.com/MicrosoftDocs/windows-powershell-docs/blob/7a7defe93f88781f0b0b3c10b2c09bbe35bdd8be/docset/winserver2012r2-ps/windowsserverbackup/Start-WBHyperVRecovery.md?plain=1#L32-L33
PS C:\Users\Administrator> $Backup.GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Object[]                                 System.Array

PS C:\Users\Administrator> $Backup[0].GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     False    WBBackupSet                              System.Object

Get-WBBackupSet seems to return an Array of WBBackupSets whereas Start-WBHyperVRecovery -BackupSet does not seem to wotk with such an array:
Start-WBHyperVRecovery : "System.Object[]" kann nicht in den Typ "Microsoft.Windows.ServerBackup.Commands.WBBackupSet"
konvertiert werden, der für den Parameter "BackupSet" erforderlich ist. Die angegebene Methode wird nicht unterstützt.
In Zeile:1 Zeichen:35
+ Start-WBHyperVRecovery -BackupSet $Backup -VMInBackup $Backup.Application[0].Com ...
+                                   ~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Start-WBHyperVRecovery], ParameterBindingException
    + FullyQualifiedErrorId : CannotConvertArgument,Microsoft.Windows.ServerBackup.Commands.StartWBHyperVRecovery

(I'm sorry, but Windows unfortunately has localization for errors and I did not yet figure out how to do something like LC_ALL=C to prevent this...; but you'll get the gist)
It seems to work though if I pass $Backup[0].

To make it easier for you to submit feedback on articles on learn.microsoft.com, we're transitioning our feedback system from GitHub Issues to a new experience.
As part of the transition, this GitHub Issue will be moved to a private repository. We're moving Issues to another repository so we can continue working on Issues that were open at the time of the transition. When this Issue is moved, you'll no longer be able to access it.
If you want to provide additional information before this Issue is moved, please update this Issue before December 15th, 2023.
With the new experience, you no longer need to sign in to GitHub to enter and submit your feedback. Instead, you can choose directly on each article's page whether the article was helpful. Then you can then choose one or more reasons for your feedback and optionally provide additional context before you select Submit.
Here's what the new experience looks like.
Note: The new experience is being rolled out across learn.microsoft.com in phases. If you don't see the new experience on an article, please check back later.
First, select whether the article was helpful:

Then, choose at least one reason for your feedback and optionally provide additional details about your feedback:

Article was helpful
Article was unhelpful

Finally, select Submit and you're done!


Pin django-modeldict dependency in Python 2

django-modeldict-yplan has removed support for Python 2 in its 2.0.0
release [0]. Being this Python version not supported anymore, but still
supported in this project, we need to pin its version to never use 2.x
in Python 2.
Current error when using a Python 2 virtualenv:
$ pip install gargoyle-yplan
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Looking in indexes: http://localhost:3141/root/pypi/+simple/
Collecting gargoyle-yplan
  Downloading http://localhost:3141/root/pypi/+f/93e/2e3d2767aceb1/gargoyle_yplan-1.5.0-py2.py3-none-any.whl (49kB)
    100% |████████████████████████████████| 51kB 253kB/s
Collecting django-modeldict-yplan>=1.5.0 (from gargoyle-yplan)
  Downloading http://localhost:3141/root/pypi/+f/f45/ba978634464b9/django_modeldict_yplan-2.0.0-py2.py3-none-any.whl
django-modeldict-yplan requires Python '>=3.4' but the running Python is 2.7.15

[0] https://github.com/adamchainz/django-modeldict/blob/master/HISTORY.rst#200-2019-01-29

This is now only needed for gargoyle-yplan < 2.0.0

Hi @adamantike,
I dug into this mess a bit more. The reason pip on Python 2 was still pulling down django-modeldict-yplan 2.0.0 after I removed the universal wheel is the metadata requires_python on PyPI for that version was unmodified, left as the empty string, meaning "no known restrictions so just try it". Therefore pip would down the wheel, then try install it and hit the version specifier locally as incompatible.
I contacted PyPI devs on IRC and there is no way of modifying the metadata.
Therefore, as a workaround, I've released django-modeldict-yplan 2.0.1 as a new version that has the corrected requires_python since universal wheels are disabled. I'll pull version 2.0.0 after 2 weeks, as a way of allowing users a fair window to "upgrade" to the new version. Since you're the only one reporting this issue, and already pin your requirements, I think this is a fair compromise.
This will mean the pin on gargoyle won't be required, as Pip on Python 2 will resolve the dependency to the 1.X series of django-modeldict-yplan. Therefore, this PR won't be required after the version 2.0.0.
Thanks for helping on this, it tracked down a subtle error that I won't be making as I drop Python 2 support from my other maintained open source packages.
Thanks,
Adam

I've now pulled version 2.0.0.

Thanks @adamchainz ! I can confirm this is fixed. Trying to install Gargoyle on Python 2 retrieves 1.5.4 instead of 2.x.

Thanks for confirming.


Publish package with typechain v7

It would be great to get a package with typechain v7

Done in:
https://github.com/dethcrypto/eth-sdk/releases/tag/%40dethcrypto/eth-sdk%400.3.0


Change keycloack admin role to showreel-admin

We now have two roles in keycloak: showreel-staff and showreel-admin which are mapped to the corresponding roles of the showreel app.


Blob tx and loading

Tracking issue for Blob transactions and related opcodes.
As discussed with the client team on 2024-06-20, to allow contracts larger than the contract size limit, and to allow upgradability, we'll be adding a new tx type that's just a bunch of bytes, and opcodes to load subsections of that. The whole thing reminds contracts and LDC a bit, but blob won't have state or balances, and the corresponding load instructions will be slightly different.
Implemented in:

https://github.com/FuelLabs/fuel-specs/pull/592
https://github.com/FuelLabs/fuel-vm/pull/780
https://github.com/FuelLabs/fuel-core/pull/1988

All sub-tasks are done=)


HOCS-5732 - Amend SLA for UKVI-PSU & vice-versa

HOCS-5732 - Currently SLA by default is set to
20 working days. This PR would help to update SLA
to 60 working days when case is accepted by PSU and
when its moved to PSU outcome stage. When the user
reject this case and send it back to UKVI - then
SLA would be updated to 20 working days. Changes
are covered using BPMN changes similar to IEDET
to PSU workflow stages.

Closing PR - as this is address in another open PR - https://github.com/UKHomeOffice/hocs-workflow/pull/901
Syncing with master branch.


InputConfigurator.validatePermissions checks for Table.READ but not Namespace.READ

When reading a table via the Accumulo Shell or via Scanner/BatchScanner, it is sufficient for the user to have either Table.READ or Namespace.READ permission for the table or its namespace. However, when reading a table via AccumuloInputFormat (in Hadoop or Spark), prior to launching the job, InputConfiguration.validatePermissions checks only whether the user has Table.READ permission, ignoring any Namespace.READ permission that the user may have. This means that even if a user can read a table via a non-Hadoop client, reading the same table in Hadoop/Spark fails with the error "Unable to access table".

This looks to be the result of some overly-aggressive fast-failure checks. We could remove the check entirely and let the tasks fail if the user doesn't have permission, or we can patch the check to allow Namespace.READ also. I'm inclined to do the latter, since the fast-failure is probably still useful.

@reggert Are you interested in working on this?

Sure, I could take a stab at it. It seems like it should be a pretty straightforward fix.
Is there an existing function somewhere to extract the namespace from a table name? It wouldn't be hard to write one, but there's no point reinventing the wheel if it already exists.

Submitted pull request targeting the 1.9 branch.

Is there an existing function somewhere to extract the namespace from a table name?

Probably somewhere deep in the code, used internally. Nothing that is likely intended for reuse.


Sprint 2

[ ] HTML and CSS

[x] HTML & CSS is Hard: No 3
[ ] HTML & CSS is Hard: No 4

[ ] JavaScript

[ ] Reading

[ ] Eloquent JavaScript: Chapter 2 (Program Structure)
[ ] Eloquent JavaScript: Chapter 3 (Functions)

[ ] Videos

[ ] Crockford on JavaScript - Chapter 2: And Then There Was JavaScript

[ ] Exercises

[ ] Nodeschool: javascripting
[ ] Nodeschool: learnyouhtml

[ ] Personal project

[ ] Decide on what project you want to pursue

Got it- thank you!

Is there a set standard to when it's best to use single or double quotation marks or is it just a preference thing?

Did you mean to close the issue?
Generally I've seen and I do single quotes always in JavaScript. HTML uses double-quotes by convention, so it's nice to differentiate between the two languages. I recommend just using single quotes for JavaScript and double quotes for HTML

No I didn't mean to close it. Thank you - that's helpful!

When I clicked on the link it looked like the same javascripting exercises as I did last week - am I missing the new ones or do you suggest doing the other ones again?

Finished everything else from this sprint. I'm not quite finished with the video but will finish that up tomorrow morning before I start the next sprint.

When I clicked on the link it looked like the same javascripting exercises as I did last week - am I missing the new ones or do you suggest doing the other ones again?

Yes, my mistake

Finished the video. Now I get why our billing was written without decimals - I had wondered about that!


Need some way to register a Receive method for non-parsable payloads.

It would be nice if there was a way to hook into the transports to catch un-parsable payloads so we can have a chance to do something else with them, or even just report this. At the moment we just report an error in response if the transport supports responses.

Addressed by using directly the messages and checking for the encoding


head, header and footer can be specified as a function

head, header and footer can be specified as a function that receives the page’s meta data (path, title and unrestricted front matter), and returns an HTML fragment (string).
I removed the strict normalization of the front matter; now only the "known" properties are normalized, and custom properties are left as given in the page.
closes #56
Partially addresses #1036, but I'm not sure how we can (or if we should) expose the meta data more generally:

to the page renderer, so that we could pass the values to be rendered SSR (#931); probably in post-processing?
to the page runtime (via a define("meta", …))?

Partially addresses #1161.

Why is it necessary to remove the strict normalization of the front matter? I don’t think we want to encourage users to put anything they want in the front matter; that will make it harder for us to introduce new options in the future without breaking projects. What is the use case for adding arbitrary things into the front matter?

The idea is coming from metadata that I would like to both display in the page, and consume outside.
For example, author (to create a byline), source (which I use in pangea to refer to the original notebook, when applicable), tags, date_published, og_image, thumbnail, etc.
Some of these might go in the head (to inform opengraph tags), some in the indexation / listings (typically, the tags), yet others in the footer (e.g. tags with links to the tags pages.
Maybe it's the wrong approach and we should reserve front matter for “official business”, but then we'd need another mechanism that would be more extensible in user land. Maybe it's just a "meta" key in front matter that is not normalized, and made available both for SSR and for client-side JS?
Anyway I can dumb down this PR so that it only does the "head as a function" part and finally closes #56, and we continue the discussion on the extensibility separately.

This also overlaps with the page loader concept #1216 which allows the entire page (not just the head, header, footer) to be dynamically generated.

rebased

showing "undefined" or "null" was not very pretty (but … informative!) :)

I've tried defining a dynamic head function by following the footer example, to no avail. When testing it for header and footer, rather than evaluating the function it returns the function as string in the rendered HTML (when in dev preview, at least). Maybe I am doing something wrong?

@dleeftink What version of framework are you using?

@Fil an older one it seems (1.5.1), updated now tot 1.9.0 and no issues with the main observablehq.config.js. Just wondering how to override on a per file basis? Adding an arrow function to the YAML frontmatter of a page like doesn't seem right:
footer: ()=>`test`

@Fil an older one it seems (1.5.1), updated now tot 1.9.0 and no issues with the main observablehq.config.js. Just wondering how to override on a per file basis? Adding an arrow function to the YAML frontmatter of a page like so doesn't seem right:
footer: ()=>`test`

Also, how about making the per-page overrides configurable to replace (append=false) or append (append=true) to the globally defined functions in config.js? This way pages don't interfere with the global settings if only minor additions are required. Something like:
header: ({append = true}) => `<i>some subtitle</i>`

Maybe overrides can even be configured to replace/append to specific metadata fields, for instance to append a date to a globally defined title or a metadata page description. Not sure what that interface would like tho as it involves creating per-field flags for potentially nested frontmatter items.

The current logic works the other way around: the function you define in observablehq.config.ts is downstream from the page, not upstream — it receives as input the page's meta data (slug, title and front-matter). One aspect we could make better is to allow "non normalized" front matter fields, so you could e.g. define categories: x, y, z in the front matter and use those in the footer. If you want to open a new issue, please try to provide details of the use case you have in mind.


 But unfortunately at the moment when a filter is executed the stack is already unwinded. How does it affect the exception logging? Well, since C# 6.0 we have exception filters for try/catch block and exception filters can be used to log an exception at the moment when the stack is not unwinded. And that gives us an easy way to implicitly add contextual information to log messages by putting them into logical call context. In my examples I'll use Serilog library since it implements such functionality (see the documentation about LogContext here). Consider the following example:
public void Run(ILogger logger)
{
	try
	{
		logger.Information("Message without additional properties");

		using (LogContext.PushProperty("packageId", 434))
		{
			logger.Information("Message with packageId");

			using (LogContext.PushProperty("itemId", 22))
			{
				logger.Information("Message with packageId and itemId");

				throw new Exception("Boom!");
			}
		}
	}
	catch (Exception e) when (LogException(e))
	{
		// no op, the catch block won't be called
	}
}

private bool LogException(Exception e)
{
	Log.Logger.Exception(e, "An error occured. Will log also packageId and itemId");
	return false;
}

Running this code will produce raw log messages similar to:
{ level: "Info", message: "Message without additional properties"}
{ level: "Info", message: "Message with packageId", fields.packageId: 434}
{ level: "Info", message: "Message with packageId and itemId", fields.packageId: 434, fields.itemId: 22}
{ level: "Error", message: "An error occured. Will log also packageId and itemId", fields.packageId: 434, fields.itemId: 22, exception: "..."}

But when we tried to implement global exception handler using exception or action filters we found that the framework doesn't allow us to inject the code inside of the try/catch block in Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker which executes the action. Since C# 6.0 has been released and people start looking into using exception filters trick for logging, it'll be good if the framework provides some additional injection points.

Hi @zihotki , are you asking for a feature to allow running arbitrary code when any exception is thrown and/or caught in the system? That would probably need a feature request to https://github.com/dotnet/corefx. There's nothing in ASP.NET Core itself that could do this. It can only have its own catch to do logging of the original exception stack, which it already does.

@Eilon Yes, at the moment when the exception is thrown.  And no, .Net Core as well as .Net Framework already have a way for doing that by using exception filters - http://www.thomaslevesque.com/2015/06/21/exception-filters-in-c-6/ but only before the exception is handled, as soon as it's handled (the catch block runs) the stack is being unwinded. So like I mentioned in the ticket, the only blocking part in Asp.Net Core is the try/catch blocks that handle exceptions in Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker. But looking at the code it's quite hard to find a good way to make it supported without dramatically changing the logic, unfortunately :(. Also the same can be achieved by changing the way how the data is cleaned from call context.

Exception filters must be hand-written by the code that is catching the exception, it's not something that ASP.NET Core can do automatically for a developer. The ControllerActionInvoker only looks at unhandled exceptions. If you have a controller that wants to handle exceptions, either handle it within the controller action, or write an IExceptionFilter and then the ControllerActionInvoker will ask your filter to try to handle the exception.

@Eilon Sorry but you completely missed my point. My point was that if we have action which has a using block and an exception occures in the using block then at the moment when an ExceptionFilter or ActionFilter is executed the stack is already unwinded and Dispose method is called due to additional try/catches in the ControllerActionInvoker. Exception Filters from C# 6 allow to run some action before disposing the object (before stack unwinding). So the initial suggestion was to run IExceptionFilter in the context of Exception Filter from C# 6 instead of running it in the context of Catch like it's currently implemented.
So if we have something like this:
public class DisposableObject : IDisposable
{
	public DisposableObject()
	{
		Debug.WriteLine("+++ Created");
	}

	public void Dispose()
	{
		Debug.WriteLine("--- Disposed");
	}
}

public class ExceptionLogger : IExceptionFilter
{
	public void OnException(ExceptionContext context)
	{
                // all using blocks in action already disposed
                // and all Finally blocks are already run
		Debug.WriteLine("!!! Filter");
	}
}

[Route("api/[controller]")]
public class ValuesController : Controller
{
    [HttpGet]
    public IEnumerable<string> Get()
    {
		using (var obj = new DisposableObject())
		{
			throw new Exception();
		}

        return new string[] { "value1", "value2" };
    }
}

The output will be:
+++ Created
--- Disposed
!!! Filter

My idea was to run Asp.Net IExceptionFilter in the Exception Filter block from C# 6 so that the output would be:
+++ Created
!!! Filter
--- Disposed

@zihotki thanks, I think I see what you're saying. I think the issue is that C#'s new "exception filter" feature is something only the person who is writing the code can use. It's essentially a C# language feature, not an actual .NET feature. A framework such as MVC can't intercept these to run additional custom code. All MVC can do it catch the exception once it leaves the app's code and it can then run its own MVC IExceptionFilter.

It's essentially a C# language feature, not an actual .NET feature.

It's an old CLR feature, which was not used so much before C# 6.0. This blog post describes it really well.
Most probably you have sth like this now somewhere in the request flow in ASP.NET / Kestrel impl:
try
{
    UserCode();
}
catch(Exception ex)
{
    exceptionFilter?.Handle(ex);
}

What @zihotki ask you is to extend it by sth like:
try
{
    UserCode();
}
catch (Exception ex) when (BeforeStackUnwind(exceptionFilter, ex))
{
  // will never get here
}
catch (Exception ex)
{
    exceptionFilter?.Handle(ex);
}

private bool BeforeStackUnwind(IExceptionFilter filter, Exception ex)
{
    filter?.BeforeStackUnwind(ex); // call the custom code if present
    return false; // return false to proceed to the next exception handler
}

This is interesting. @Eilon we should consider this but maybe it needs to be a new method or another interface on an IExceptionFilter. It might make the state machine in the ControllerActionInvoker crazier though but @rynowak can handle it 😄

Yes, this is an extension of what I've written in https://github.com/aspnet/Logging/issues/677. I've been talking about the general problem that C# 6 exception filters seem to not work in asynchronous scenarios. If MVC (or any other component that runs before your exception filter) catches the exception, it obviously also breaks the usage.
I think it is a pretty big deal that logging scopes can not be added to error logs and I think it makes them almost useless. It's just not reasonable to write custom try...catch statements within every scope block.


will you reformulate this project based on Detectron2?

Hi Tengfei,
Currently, I do not have the plan, because MMDetection is popular enough.
But it depends on the situation in the future(e.g.  if more people have this demand, maybe I will reimplement it on Detectron2).


Change order of roll notes for action macros

Move the extra roll notes for action macros after the roll notes for the used statistic. These extra notes are usually related to the degree of success and makes most sense at the bottom of the list.

Found an alternative way to deal with this.


CLOSED and CLOSING states of WebSocket are not covered

So messages sent during that period may never reach the server, what with JSON Patch queuing enabled may block any further communication..

@miyconst after doing current work, could you investigate this and propose changes?

Looks like yes, this situation may happen.
Those are the possible values for Socket.readyState:

A value of 0 indicates that the connection has not yet been established.
A value of 1 indicates that the connection is established and communication is possible.
A value of 2 indicates that the connection is going through the closing handshake.
A value of 3 indicates that the connection has been closed or could not be opened.

Currently PuppetNetworkChannel establishes connection and sends a message in onopen if the state is equal to 0 or sends a message immediately in other case.
I suggest that we:

[x] 0: establish connection and send message when callback is established.
[ ] 1: send message.
[ ] 2: add message to onopen queue.
[ ] *: send message.
[ ] *: send message via http.

@tomalec - WDYT?
The way I understand it, we have a choice to create yet another queue to store messages pending for WS (for me that is very undesired) or send patches over HTTP if WS is in other state than 1.

@warpech there is already a pseudo queue for outgoing messages over WebSocket, so using it won't create a new one, but I would also like to avoid it.

Where is that in code?

@warpech  here: https://github.com/PuppetJs/PuppetJs/blob/gh-pages/src/puppet.js#L168
@miyconst How your proposition differ from current implementation? 3 and 4 are just not implemented, right? So we would need just 2 more elsseifs at https://github.com/PuppetJs/PuppetJs/blob/gh-pages/src/puppet.js#L177, wouldn't we?

@tomalec, you are right, I thought we could just implement case 3 and 4.

Why not simplify it.
As I proposed on today's meeting, WS (re)connection should be maintained automatically in bootstrap (#59, #37) and therefore can be removed from PuppetNetworkChannel.prototype.send.
PuppetNetworkChannel.prototype.send gets simplified to:
PuppetNetworkChannel.prototype.send = function(msg){
    var that = this;
    if (this.useWebSocket && this._ws && this._ws.readyState === 1) {
        this._ws.send(msg);
        that.onSend(msg, that._ws.url, "WS");
    }
    else {
      var url = this.referer || this.remoteUrl;
      //"referer" should be used as the url when sending JSON Patches (see https://github.com/PuppetJs/PuppetJs/wiki/Server-communication)
      this.xhr(url, 'application/json-patch+json', msg, function (res, method) {
          that.onReceive(res.responseText, url, method);
        });
    }
    return this;
  };

LGTM

I also agree.

@warpech should I do this one?

Updated puppetJs, but it still does not work if ws connection drops, because of remote version mismatch issue described in here: https://github.com/PuppetJs/PuppetJs/issues/37#issuecomment-111513786

@warpech can this issue be closed? For ws connection we have another issue.

Looks good, thanks!


chore(ci): fix lint

@fkleuver

@bigopon Can you just take out the circleci please, I'm planning to do this with orbs


Terminology change, esp in URLs in API examples

https://*<project-name>.hasura-app.io -> https://*<cluster-name>.hasura-app.io
Change project-name to cluster-name everywhere

This is done in the docs.
@shahidhk Are you referring to something else?


Teach the check-DNS script to ignore temporary files

A plain grep was occasionally attempting to grep temporary files that
were being created and deleted concurrently by other sanity check
scripts, leading to TOCTOU problems. This now only searches indexed
files (thanks for the suggestion, @ctiller!).
We still need to exclude third_party since upb is not a submodule.
git grep will ignore all submodules by default, so the other
third_party subfolders were already excluded.

Apologies for the late review -- I thought this was c-ares related and didn't want to page that stuff in today :)

Thanks! And no worries, I'd have pinged you before long. I wanted to avoid "DO NOT SUBMIT" ending up in a commit message 😬.


feat(GridForm): resetOnSubmit

Overview

PR Checklist

[ ] Related to designs:
[ ] Related to JIRA ticket: ABC-123
[ ] I have run this code to verify it works
[ ] This PR includes unit tests for the code change

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.56-alpha.b28747.0@codecademy/gamut-labs@19.23.19-alpha.b28747.0@codecademy/gamut@36.9.2-alpha.b28747.0@codecademy/monaco@0.3.10-alpha.b28747.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://6132764a1bfb018b3bdc9cbd--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.61-alpha.92200c.0@codecademy/gamut-labs@19.24.5-alpha.92200c.0@codecademy/gamut@37.1.1-alpha.92200c.0@codecademy/monaco@0.3.13-alpha.92200c.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://61327876bac5dd86f2af33c0--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.61-alpha.09215d.0@codecademy/gamut-labs@19.24.5-alpha.09215d.0@codecademy/gamut@37.1.1-alpha.09215d.0@codecademy/monaco@0.3.13-alpha.09215d.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://613293749d12cda7fe4c999b--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.64-alpha.aeef48.0@codecademy/gamut-labs@19.24.8-alpha.aeef48.0@codecademy/gamut@37.1.1-alpha.aeef48.0@codecademy/monaco@0.3.13-alpha.aeef48.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://61376b5c6ad27a9a16671b8c--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.64-alpha.b35829.0@codecademy/gamut-labs@19.24.8-alpha.b35829.0@codecademy/gamut@37.1.1-alpha.b35829.0@codecademy/monaco@0.3.13-alpha.b35829.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://61378ea7f9bfb40a65588f3c--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.64-alpha.e09ce7.0@codecademy/gamut-labs@19.24.8-alpha.e09ce7.0@codecademy/gamut@37.1.1-alpha.e09ce7.0@codecademy/monaco@0.3.13-alpha.e09ce7.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://6138fb62c1d70b0c11bd41eb--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.67-alpha.a7a91d.0@codecademy/gamut-labs@19.24.11-alpha.a7a91d.0@codecademy/gamut@37.1.3-alpha.a7a91d.0@codecademy/monaco@0.3.15-alpha.a7a91d.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://613a31864cc96628c8ae916e--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.67-alpha.fbb360.0@codecademy/gamut-labs@19.24.11-alpha.fbb360.0@codecademy/gamut@37.1.3-alpha.fbb360.0@codecademy/monaco@0.3.15-alpha.fbb360.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://613a33a1ef9e2b34b6630309--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/eslint-config@5.4.0-alpha.4f0ba0.0@codecademy/gamut-icons@5.10.0-alpha.4f0ba0.0@codecademy/gamut-illustrations@0.18.0-alpha.4f0ba0.0@codecademy/gamut-kit@0.5.71-alpha.4f0ba0.0@codecademy/gamut-labs@19.26.0-alpha.4f0ba0.0@codecademy/gamut-patterns@0.6.0-alpha.4f0ba0.0@codecademy/gamut-styles@11.10.0-alpha.4f0ba0.0@codecademy/gamut-system@0.8.0-alpha.4f0ba0.0@codecademy/gamut-tests@2.6.0-alpha.4f0ba0.0@codecademy/gamut@37.3.0-alpha.4f0ba0.0@codecademy/konamimojisplosion@0.3.0-alpha.4f0ba0.0@codecademy/macros@2.2.0-alpha.4f0ba0.0@codecademy/markdown-overrides@0.6.0-alpha.4f0ba0.0@codecademy/monaco@0.4.0-alpha.4f0ba0.0@codecademy/styleguide@44.3.0-alpha.4f0ba0.0@codecademy/tracking@0.14.0-alpha.4f0ba0.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://613b68b1082f9b0efdab08d4--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.73-alpha.005a5a.0@codecademy/gamut-labs@19.26.2-alpha.005a5a.0@codecademy/gamut@37.3.2-alpha.005a5a.0@codecademy/monaco@0.4.2-alpha.005a5a.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://613b7bea0fce7100c2ba8589--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.73-alpha.4466a6.0@codecademy/gamut-labs@19.26.2-alpha.4466a6.0@codecademy/gamut@37.3.2-alpha.4466a6.0@codecademy/monaco@0.4.2-alpha.4466a6.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://613b80c0c187a600e2a626cf--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.73-alpha.c1013b.0@codecademy/gamut-labs@19.26.2-alpha.c1013b.0@codecademy/gamut@37.3.2-alpha.c1013b.0@codecademy/monaco@0.4.2-alpha.c1013b.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://613b9fe0c187a62417a626e2--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.75-alpha.26f03f.0@codecademy/gamut-labs@19.26.4-alpha.26f03f.0@codecademy/gamut@37.3.3-alpha.26f03f.0@codecademy/monaco@0.4.3-alpha.26f03f.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://613f8735be6ef90d32736c59--gamut-preview.netlify.appDeploy Logs

🚀 Styleguide deploy preview ready!https://614099db06be0b395e704da1--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.76-alpha.37d143.0@codecademy/gamut-labs@19.26.5-alpha.37d143.0@codecademy/gamut@37.4.1-alpha.37d143.0@codecademy/monaco@0.4.4-alpha.37d143.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://6140c713a656fc608acccf75--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.78-alpha.8747c7.0@codecademy/gamut-labs@19.26.7-alpha.8747c7.0@codecademy/gamut@37.4.2-alpha.8747c7.0@codecademy/monaco@0.4.5-alpha.8747c7.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://6141f298b486c343c75fbe23--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-kit@0.5.78-alpha.e7d9e6.0@codecademy/gamut-labs@19.26.7-alpha.e7d9e6.0@codecademy/gamut@37.4.2-alpha.e7d9e6.0@codecademy/monaco@0.4.5-alpha.e7d9e6.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://6141fb4b50231a5382932ad4--gamut-preview.netlify.appDeploy Logs


[BACKLOG-6573] Regression with Metadata injecting the Excel input step

@mdamour1976 Please Sir!

@mdamour1976 Also... please advice on the comment I left in the Jira case and see the attached screenshoot.


Typescript Definition for run doesn't allow array of streams

The run command's definition looks something like this (there are a few more):
run(image: string, cmd: string[], outputStream: NodeJS.WritableStream, createOptions: {}, startOptions: {}, callback: Callback<any>): events.EventEmitter;
To allow for parsing and splitting stdout and stderr, an array of streams should be passed into the run command.
I'm happy to put in a pull request for the definitely typed definition file, assuming a definition like the following would be ok:
 run(image: string, cmd: string[], outputStream: NodeJS.WritableStream | NodeJS.WritableStream[], createOptions: {}, startOptions: {}, callback: Callback<any>): events.EventEmitter;
I have tested the change locally and it works fine, not sure if this would introduce any breaking changes since it probably relies on a newer version of typescript?

Fixed!