JOSHUA-252 Make it possible to use Maven to build Joshua

Hi Folks,
This PR is a beast. I think a Google Hangout would be best to talk through what is going on.
Basically it,

Mavenizes the entire codebase e.g. sets out the src/main/java/org/apache/joshua/... structure.
goes a long way to implementing the correct structure and notation for Unit testing e.g. src/test/java/org/apache/joshua/...
moves everything from $JOSHUA_HOM/tst to the correct src/test/java/... directories then removes tst
moves all test .java files from $JOSHUA_HOME/test to src/test/java/...
removes the previous Eclipse project definitions which were committed to the repos
removes the thrax submodule. We should just pull this in as a dependency from now on. I've therefore put up a PR for this over on https://github.com/joshua-decoder/thrax/issues/11
...

 If you pull this PR in to a branch and execute mvn clean install you will see that this is the case with around 6 or 7 test files. We need to collectively decide what to do with the missing imports.
Apart from that, work still t be done here is as follows

ensure that all testing resources are moved into $JOSHUA_HOME/src/test/resources. This folder is not currently created.
Remove build.xml
plan on how we are going to compile and package GIZA++, KenLM, etc. This is something @KellenSunderland @buggtd and I discussed at the ApacheCon meetup however there is still work to be done here.

The first thing i think, is to get the test classes compiling. Once we have done that we can move on to getting the tests passing and stabilizing the Java build code.
After that we can move on to the C++ build challenges.
All in all, a VERY successful week for Joshua. Looking forward to stabilizing the Maven build as it will make things so much easier or us all moving forward.

@lewismc  Please merge this ASAP.
This is a great PR. Maven for the win!

My ant build is failing..

[ivy:resolve] 	  http://ivyroundup.googlecode.com/svn/trunk/repo/modules/org.apache.commons/commons-cli/1.2/packager.xml
[ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
[ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] 		:: net.sourceforge.ant-doxygen#ant-doxygen;1.6.1: not found
[ivy:resolve] 		:: org.apache.commons#commons-cli;1.2: not found
[ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve]
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

Merging this PR helps me to move forward for resolving other issues.

The PR is not finished yet. There are a number of issues as highlighted above which need attention.

Hey Lewis.  I'm flying today but can take a look at getting the tests to
compile next week.  Thanks for all the work.  I'd be game for a google
hangout to discuss any issues we have.
On Sun, May 15, 2016 at 7:19 PM, Lewis John McGibbney <
<EMAIL_ADDRESS>wrote:

The PR is not finished yet. There are a number of issues as highlighted
above which need attention.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-joshua/pull/12#issuecomment-219335321

ACK
Safe flight Kellen.
On Sun, May 15, 2016 at 7:25 PM, Kellen Sunderland<EMAIL_ADDRESS>
wrote:

Hey Lewis. I'm flying today but can take a look at getting the tests to
compile next week. Thanks for all the work. I'd be game for a google
hangout to discuss any issues we have.
On Sun, May 15, 2016 at 7:19 PM, Lewis John McGibbney <
<EMAIL_ADDRESS>wrote:

The PR is not finished yet. There are a number of issues as highlighted
above which need attention.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
<
https://github.com/apache/incubator-joshua/pull/12#issuecomment-219335321>

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-joshua/pull/12#issuecomment-219335946

--
Lewis

this is indeed a big change and I think it's better to split into multiple small PRs.
Maybe we can have a 'maven' branch in parallel for a short transition time,
resolve the remaining issues on that branch with smaller PRs and merge it with master?

I second @thammegowda's idea of merging this into a "maven" branch first which will also continue to pull the latest from master. This is a big change and I'm not going to have time to look at it too closely for a while yet (probably later next week).

I agree as well
It's pretty huge as I said so we need lots of eyes on it.
Current compilation issues as as follows
[INFO] 100 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.155 s
[INFO] Finished at: 2016-05-15T20:09:45-07:00
[INFO] Final Memory: 29M/252M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:testCompile (default-testCompile) on project joshua: Compilation failure: Compilation failure:
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/ArityPhrasePenaltyFFTest.java:[21,38] error: cannot find symbol
[ERROR] package org.apache.joshua.decoder.ff.tm
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/ArityPhrasePenaltyFFTest.java:[22,38] error: cannot find symbol
[ERROR] package org.apache.joshua.decoder.ff.tm
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[192,22] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/packed/PrintRules.java:[29,42] error: package org.apache.joshua.util.quantization does not exist
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/packed/PrintRules.java:[30,42] error: package org.apache.joshua.util.quantization does not exist
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/packed/PrintRules.java:[42,10] error: cannot find symbol
[ERROR] class PrintRules
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/ArityPhrasePenaltyFFTest.java:[38,24] error: cannot find symbol
[ERROR] class ArityPhrasePenaltyFFTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/ArityPhrasePenaltyFFTest.java:[46,16] error: cannot find symbol
[ERROR] class ArityPhrasePenaltyFFTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/ArityPhrasePenaltyFFTest.java:[50,4] error: cannot find symbol
[ERROR] class ArityPhrasePenaltyFFTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/ArityPhrasePenaltyFFTest.java:[50,47] error: cannot find symbol
[ERROR] class ArityPhrasePenaltyFFTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/ArityPhrasePenaltyFFTest.java:[58,25] error: cannot find symbol
[ERROR] class ArityPhrasePenaltyFFTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/ArityPhrasePenaltyFFTest.java:[60,69] error: cannot find symbol
[ERROR] class ArityPhrasePenaltyFFTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[106,4] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[106,28] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[118,4] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[118,28] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[124,9] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[156,4] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[156,28] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[163,4] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[163,28] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[165,4] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[165,20] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[172,4] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[172,28] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[174,4] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[174,20] error: cannot find symbol
[ERROR] class ArpaFileTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[183,27] error: constructor LMGrammarBerkeley in class LMGrammarBerkeley cannot be applied to given types;
[ERROR] actual and formal argument lists differ in length
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[194,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[195,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[196,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[197,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[198,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[199,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[200,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[201,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[204,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[204,92] error: cannot find symbol
[ERROR] class JoshuaConfiguration
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[207,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[208,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[209,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[210,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[213,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[216,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[217,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[218,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[219,52] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java:[222,23] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/zmert/BLEUTest.java:[64,28] error: maxGramLength has protected access in BLEU
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/zmert/BLEUTest.java:[67,28] error: effLengthMethod has protected access in BLEU
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/zmert/BLEUTest.java:[67,50] error: EffectiveLengthMethod is not public in BLEU; cannot be accessed from outside package
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/lattice/LatticeTest.java:[108,37] error: cannot find symbol
[ERROR] class Lattice
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/packed/CountRules.java:[49,26] error: incompatible types: String cannot be converted to File
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/packed/PrintRules.java:[63,24] error: incompatible types: String cannot be converted to File
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/packed/PrintRules.java:[66,23] error: cannot find symbol
[ERROR] class PrintRules
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/packed/PrintRules.java:[160,6] error: cannot find symbol
[ERROR] class PrintRules
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/packed/VocabTest.java:[32,41] error: incompatible types: String cannot be converted to File
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/DecoderThreadTest.java:[78,38] error: cannot find symbol
[ERROR] class Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/DecoderThreadTest.java:[82,28] error: cannot find symbol
[ERROR] class DecoderThreadTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/DecoderThreadTest.java:[94,38] error: cannot find symbol
[ERROR] class Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/DecoderThreadTest.java:[100,28] error: cannot find symbol
[ERROR] class DecoderThreadTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/DecoderThreadTest.java:[112,6] error: cannot find symbol
[ERROR] class DecoderThreadTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/DecoderThreadTest.java:[112,33] error: cannot find symbol
[ERROR] class DecoderThreadTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/DecoderThreadTest.java:[141,6] error: cannot find symbol
[ERROR] class DecoderThreadTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/DecoderThreadTest.java:[141,35] error: cannot find symbol
[ERROR] class DecoderThreadTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/DecoderThreadTest.java:[148,6] error: cannot find symbol
[ERROR] class DecoderThreadTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/decoder/DecoderThreadTest.java:[148,38] error: cannot find symbol
[ERROR] class DecoderThreadTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/system/MultithreadedTranslationTests.java:[110,16] error: cannot find symbol
[ERROR] variable joshuaConfig of type JoshuaConfiguration
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/system/MultithreadedTranslationTests.java:[118,4] error: cannot find symbol
[ERROR] class MultithreadedTranslationTests
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/system/MultithreadedTranslationTests.java:[118,33] error: cannot find symbol
[ERROR] class MultithreadedTranslationTests
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[48,6] error: cannot find symbol
[ERROR] class CorpusArrayTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[49,6] error: cannot find symbol
[ERROR] class CorpusArrayTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[49,27] error: cannot find symbol
[ERROR] class CorpusArrayTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[54,6] error: cannot find symbol
[ERROR] class CorpusArrayTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[54,45] error: cannot find symbol
[ERROR] class CorpusArrayTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[91,4] error: cannot find symbol
[ERROR] class CorpusArrayTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[107,32] error: cannot find symbol
[ERROR] class Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[110,22] error: cannot find symbol
[ERROR] class CorpusArrayTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[142,16] error: cannot find symbol
[ERROR] class Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[143,6] error: cannot find symbol
[ERROR] class CorpusArrayTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[143,27] error: cannot find symbol
[ERROR] class CorpusArrayTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[147,6] error: cannot find symbol
[ERROR] class CorpusArrayTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/CorpusArrayTest.java:[147,45] error: cannot find symbol
[ERROR] class CorpusArrayTest
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/util/io/BinaryTest.java:[48,23] error: constructor Vocabulary in class Vocabulary cannot be applied to given types;
[ERROR] actual and formal argument lists differ in length
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/util/io/BinaryTest.java:[55,11] error: cannot find symbol
[ERROR] variable vocab of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/util/io/BinaryTest.java:[57,36] error: type argument Vocabulary is not within bounds of type-variable E
[ERROR]
[ERROR] E extends Externalizable declared in class BinaryIn
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/system/StructuredTranslationTest.java:[118,16] error: cannot find symbol
[ERROR] variable joshuaConfig of type JoshuaConfiguration
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/system/StructuredTranslationTest.java:[131,16] error: cannot find symbol
[ERROR] variable joshuaConfig of type JoshuaConfiguration
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/system/StructuredTranslationTest.java:[146,16] error: cannot find symbol
[ERROR] variable joshuaConfig of type JoshuaConfiguration
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/system/StructuredTranslationTest.java:[168,16] error: cannot find symbol
[ERROR] variable joshuaConfig of type JoshuaConfiguration
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/system/StructuredTranslationTest.java:[187,16] error: cannot find symbol
[ERROR] variable joshuaConfig of type JoshuaConfiguration
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/system/StructuredTranslationTest.java:[207,16] error: cannot find symbol
[ERROR] variable joshuaConfig of type JoshuaConfiguration
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/vocab/VocabularyTest.java:[48,24] error: constructor Vocabulary in class Vocabulary cannot be applied to given types;
[ERROR] actual and formal argument lists differ in length
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/vocab/VocabularyTest.java:[52,29] error: cannot find symbol
[ERROR] variable vocab1 of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/vocab/VocabularyTest.java:[54,29] error: no suitable method found for getWords(no arguments)
[ERROR] is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method Vocabulary.getWords(Iterable<Integer>) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/vocab/VocabularyTest.java:[55,28] error: cannot find symbol
[ERROR] variable vocab1 of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/vocab/VocabularyTest.java:[55,51] error: cannot find symbol
[ERROR] class Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/vocab/VocabularyTest.java:[56,30] error: no suitable method found for getWords(no arguments)
[ERROR] is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method Vocabulary.getWords(Iterable<Integer>) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/vocab/VocabularyTest.java:[56,49] error: cannot find symbol
[ERROR] variable vocab1 of type Vocabulary
[ERROR] /usr/local/incubator-joshua/src/test/java/org/apache/joshua/corpus/vocab/VocabularyTest.java:[59,49] error: UNKNOWN_WORD is not public in Vocabulary; cannot be accessed from outside package
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

@mjpost do you know where I can find the class ArpaFile.java ? It is referenced within [src/test/java/org/apache/joshua/decoder/ff/lm/ArpaFileTest.java|https://github.com/apache/incubator-joshua/blob/0990ebc07f44073156c1c7923fee7ebaaf7677f5/test/joshua/decoder/ff/lm/ArpaFileTest.java#L107] but not available within master source.

I don't. However, the "Arpa" denotes the standard SRILM format for language model files (read by both BerkeleyLM and KenLM), so the file would have been very similar to this one in BerkeleyLM.
As a warning, I have not touched many of those unit test files in years, and many of them may be way, way out of date.

ACK @mjpost I knew this. I am tempted to merely skip the ones for which classes no longer exist. I'm going to try and stabilize the build tonight.

No way, I actually found the file online
http://www.programcreek.com/java-api-examples/index.php?source_dir=relax-decode-master/third-party/joshua-1.3/src/joshua/decoder/ff/lm/ArpaFile.java

Wow, nice. We might want to put some of that back in there for completeness / posterity.

+1
@lewismc  I will then fix logs, system.exits() and other trivial issues in code.

Hi Folks, this is now a feature branch and can be found at https://github.com/apache/incubator-joshua/tree/JOSHUA-252
I have set up a build for the branch as well to make interpretation and understanding a little clearer. The build can be seen at https://builds.apache.org/view/H-L/view/Joshua/job/joshua_maven/
I would kindly ask that all code committed to master branch also be committed to the JOSHUA-252 branch otherwise the codebases will quickly diverge.

The next step here is for me to stabilize the test suite. This will involve putting the test resources in to src/test/resources and then removing the top level test directory. I'll work on this over the next few days.

I didn't see your JOSHUA-252 branch. I just created a "maven" branch that pulls PRs 4 and 5 into this one. However, I don't see these on github / incubator. It seems that Apache is not pushing branches down to github.
Everything seems to compile for me when I type the commands @thammegowda provided. However, I don't really know how to use Maven, and eclipse is totally broken. Can you add a BUILD file that describes how to compile and how to get things set up in eclipse?

Yep I'll add all of this to the wiki space. In the meantime you can do
mvn eclipse:eclipse
Easy as that
On Monday, May 16, 2016, Matt Post<EMAIL_ADDRESS>wrote:

I didn't see your JOSHUA-252 branch. I just created a "maven" branch that
pulls PRs 4 and 5 into this one. However, I don't see these on github /
incubator. It seems that Apache is not pushing branches down to github.
Everything seems to compile for me when I type the commands @thammegowda
provided
https://github.com/apache/incubator-joshua/pull/13#issuecomment-219570096.
However, I don't really know how to use Maven, and eclipse is totally
broken. Can you add a BUILD file that describes how to compile and how to
get things set up in eclipse?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-joshua/pull/12#issuecomment-219586976

--
Lewis

@lewismc I tried to create a training model from maven build. Found that pipeline.pl  script uses berkeleyaligner.jar.

berkeleyaligner.jar is not in maven repo.  I see a project in @mjpost 's github which has ant build  so I couldnt do local maven install to try it out. Should we need to port that to maven build or do you know a quick way to install ant jar to maven local?

The original licence is GPL v2, I am not sure if it has been relicensed to Apache. Can we add this as dependency to create a fat jar of all dependencies?

You can build it locally by simply using the following scope
    <dependency>
      <groupId>edu.berkeley.nlp</groupId>
      <artifactId>jberkeleyaligner</artifactId>
      <version>X</version>
      <scope>system</scope>
      <systemPath>/path/to/berkeleyaligner.jar</systemPath>
    </dependency>

I think you can remove the version element for the time being and try that out. We may need to create a Maven build for berkeley aligner and push it to Maven Central

great work on this @lewismc so awesome

@lewismc I was trying to fix logs, and found this problematic package names in strings: https://github.com/apache/incubator-joshua/blob/JOSHUA-252/src/main/java/org/apache/joshua/decoder/Decoder.java#L967

What's problematic about that? FYI, that line is part of the code that lets feature functions be loaded by reflection, e.g., so you can write a new feature function and call it directly from the command line without having to add stub code to check for and load it.

@mjpost Sorry for my previous incomplete comment.
The problem I see with that line is, now we have new package org.apache.joshua.decoder.ff instead of old joshua.decoder.ff.
Since lewis is currently working on this, I didnt dare to raise a new PR to fix a single line :-)

I'll have the Javadoc for this PR finished tonight. The build will then be 'stable'.
At that stage we can rebase @thammegowda 's most recent patch for System calls, etc into the JOSHUA-252 branch.
We can then think about how the hell to get all of the other pipeline running :)

dusted

@thammegowda @mjpost  OK the Maven build is now 'stable'.
We need to list which changes have been made to master branch since this work began. These then need to be merged into the JOSHUA-252 branch. Additionally, the following needs to be addressed

merge all work which has been done to master branch into JOSHUA-252 branch
plan on how we are going to compile and package GIZA++, KenLM, berkeleylm, thrax, etc. Right now the ext directory has been completely removed as that code should not live but instead be acquired as part of the build system.

great work @lewismc .
All my activity is branched off from this branch

#13  should be able to merge with this branch (have conflicts now, but i can resolve them)
#14 can be merged with this, no conflicts
#15 is targetted to branch named maven . Probably I need to raise a new PR targeted to this JOSHUA-252 branch ?

hi @thammegowda thanks for reply

#13 should be able to merge with this branch (have conflicts now, but i can resolve them)
Please rebase the above against this branch

#14 can be merged with this, no conflicts
I will merge this

#15 is targetted to branch named maven . Probably I need to raise a new PR targeted to this JOSHUA-252 branch ?
Yes please do. All branches should be named after the associated Jira ticket. That way we have full blown provenance tracking for individual issues.
Thanks, I'll merge #14 now

#14 is merged.
Work still to be done

merge #13 into JOSHUA-252 branch
merge #15 into JOSHUA-252
plan on how we are going to compile and package GIZA++, KenLM, berkeleylm, thrax, etc. Right now the ext directory has been completely removed as that code should not live but instead be acquired as part of the build system.

@mjpost have you made any changes to master which we need to merge into JOSHUA-252 branch?
Thanks

@lewismc thanks for the reply.
rebased #13 , it is ready for merge!

Cool @thammegowda
#14 is merged.
Work still to be done
#13 merged
work still to be done

merge #15 into JOSHUA-252
plan on how we are going to compile and package GIZA++, KenLM, berkeleylm, thrax, etc. Right now the ext directory has been completely removed as that code should not live but instead be acquired as part of the build system.

@mjpost have you made any changes to master which we need to merge into JOSHUA-252 branch?
Thanks

@lewismc Please merge #17 ( it is a duplicate of #15 )

Cool @thammegowda
#15 merged into JOSHUA-252
Work still to be done

plan on how we are going to compile and package GIZA++, KenLM, berkeleylm, thrax, etc. Right now the ext directory has been completely removed as that code should not live but instead be acquired as part of the build system.

@mjpost have you made any changes to master which we need to merge into JOSHUA-252 branch?
Thanks

There have been changes on master (JOSHUA-271 and JOSHUA-272), will try to get to this soon, probably not till tomorrow.
Do you want me to be the one to merge master into JOSHUA-252? Then we can test and merge back into master. I think this might be our 6.1 release?
For packages:

Is it possible to have Maven do git checkouts (on specific versions?) That's one route
Otherwise, we should just rely on the user to set them up. The only real problem is for BerkeleyLM and KenLM, since those are runtime. I don't have much experience in how to do this. But we could have a maven target that does a git checkout of a specific version, kind of like a submodule.
Thrax and GIZA (and fast_align) we can tell the user to set up.

matt

Hi @mjpost thanks for response
Do you want me to be the one to merge master into JOSHUA-252? Then we can test and merge back into master. I think this might be our 6.1 release?

Yes please if you could take care of merging your work on master into JOSHUA-252 then that would be grand. I am not overly familiar with the work done so I don't want to botch it. Thanks Matt.
Is it possible to have Maven do git checkouts (on specific versions?) That's one route

I was thinking the exact same thing and I am actively investigating here.
Otherwise, we should just rely on the user to set them up. The only real problem is for BerkeleyLM and KenLM, since those are runtime. I don't have much experience in how to do this. But we could have a maven target that does a git checkout of a specific version, kind of like a submodule.

Hopefully I can provide insight and input into this. Give me a wee bit of time and I can get around to answering in due course.
Thrax and GIZA (and fast_align) we can tell the user to set up.

Agreed. I sent a PR for Thrax (which is incomplete due to changes in the underlying API) which Mavenies the entire build https://github.com/joshua-decoder/thrax/pull/12 we could then pull Thrax as a dependency within Joshua.
For GIZA we will get the user to set up.
Thanks

Okay, cool, I got this merged and compiling. First, THANK YOU for all of this amazing work! This feels like a real milestone and it was fun to work through the merge.
Before I push, I need to run the tests, though, and am not sure how to invoke the decoder. Can you provide some assistance here? I don't know maven very well, so if there are any maven commands to type first, that would also help to know. Basically, if you can get me to the point where I can type
$ echo test | joshua

and it returns
0 ||| test ||| tm_glue_0=1.000 ||| 0.000

then I can run my regression tests and push up.

Nevermind, found Thamme's old instructions.

There will be a number of scripts which we need to update with the paths to the generated jar files. This is all part of the new build.

Yep, working through those. Making good progress on the build. Once I can verify that the test cases pass under the new build (so I can know I've done the merge correctly), I'll push it up.

Excellent

Can someone tell me how to turn off logging in slf4j? I have tried passing
-Dorg.slf4j.Logger.defaultLogLevel=info

but this does not seem to work. Extensive googling around and walking through the manual have not turned up anything. I've probably missed something.

Does setting it to error instead of info help?
On May 27, 2016 12:04 AM, "Matt Post"<EMAIL_ADDRESS>wrote:

Can someone tell me how to turn off logging in slf4j? I have tried passing
-Dorg.slf4j.Logger.defaultLogLevel=info
but this does not seem to work. Extensive googling around and walking
through the manual have not turned up anything. I've probably missed
something.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-joshua/pull/12#issuecomment-222008424

Okay, the answer is that our slfj4 is using log4j12 as the backend. I had to add
-Dlog4j.configuration=$JOSHUA/logging.properties

to the invocation, and then put
log4j.rootLogger=OFF

(or info, etc).

@mjpost as you pointed out, you need to edit log4j settings.
The other way is to edit src/main/resources/log4j.properties
If you dont want to rebuild the jar after editing the file, you can create an updated log4j.properties and  prefix  its path to the class path (so that it gets priority over the one inside jar)

Okay, cool. It doesn't seem to look for that file automatically, though; if I remove the -Dlog4j.configuration, it doesn't find the file you pointed to. So I am just having it hard-point to that path.

Okay, I have merged. The handful of test cases I checked passed, but there is bad news: the decoder is significantly slower, like by an order of magnitude. Tests that returned instantly before or took just a second or two now take 10–20.

Also, multithreading is broken. I randomly (not always) get the following error when decoding with more than 1 thread:
Exception in thread "Thread-4" java.lang.RuntimeException: Input 1: FATAL UNCAUGHT EXCEPTION: null
	at org.apache.joshua.decoder.Decoder$DecoderThreadRunner.run(Decoder.java:433)
Caused by: java.util.ConcurrentModificationException
	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
	at java.util.ArrayList$Itr.next(ArrayList.java:851)
	at org.apache.joshua.decoder.chart_parser.DotChart.addDotItem(DotChart.java:362)
	at org.apache.joshua.decoder.chart_parser.DotChart.expandDotCell(DotChart.java:223)
	at org.apache.joshua.decoder.chart_parser.Chart.expand(Chart.java:585)
	at org.apache.joshua.decoder.DecoderThread.translate(DecoderThread.java:141)
	at org.apache.joshua.decoder.Decoder$DecoderThreadRunner.run(Decoder.java:424)

Having thought about this, but not looked, my guess is that the arguments to logging functions are getting computed even if they're not printed, and that's costing lots of time. I'm not sure why that would affect multithreading though.
@lewismc and @thammegowda, if you have any thoughts on any other changes you made that might have caused this, please let me know.
(I think the lesson is that I should have let JOSHUA-252 get fully merged before pulling in the logging code. That would have let us separate these issues)

My guess is that ConcurrentModificationException occurs when we try to loop (eg: for) on a collection and inside the loop the collection is being updated (by adding or removing items).
However, based on the exception, I checked https://github.com/apache/incubator-joshua/blob/JOSHUA-252/src/main/java/org/apache/joshua/decoder/chart_parser/DotChart.java#L362
there isnt any such code. I suspect Rule.toString().
That line executes only if DEBUG level is enabled! try disabling DEBUG (by setting Log4j level to INFO or higher until we find a fix

Okay, I just pushed up the changes after merging in master and making some other changes to that the tests will run (with manual calls). The code on master runs fine, so it must have been something in the 252 or 262 branches. I've done some initial work wrapping LOG.debug() calls in checks for LOG.isDebugEnabled(), but that didn't seem to help. All my tests were done with
log4j.rootLogger=OFF, stdout

I may have some more time to look at this tomorrow.

What kind of 'tests' are we talking about here? The only tests which I know of are invoked by running mvn clean test. These execute very quickly by the looks of it however I assume that they cover little or none of the multithreaded functionality you are referring to.
Do you have something we could code, profile and see what is taking so long @mjpost  ? Thanks
One last thing. No I changed absolutely no code. I did however introduce some classes such as the ArpaFile, etc which were required to get the current tests running.

On a hunch, I went through and removed all the calls LOG.*() in Decoder, DecoderThread, JoshuaDecoder, Translation, Chart, DotChart, and ComputeNodeResult. This fixed the problem; the decoder now runs as fast as before.

I went into joshua/decoder/chart_parser/ComputeNodeResult; adding and removing the following lines makes a huge difference in speed. So it looks like logging is neither (a) time-safe nor (b) thread-safe.
Or that something is wrong. I can't get the decoder to output any statements using the logging properties. It's sort of a pain to have to rebuild a tarball every time you want to test something...

On even further investigation, with some System.err.println()s, I discovered that the properties file isn't getting loaded correctly. Consequently, isDebugEnabled() is always returning true, but the logging statements are not printed anywhere visible. So the work is getting triggered, and is just not seen, and that is the problem.
I've pushed up the merge. Can you folks take care of making sure that the logger file is running correctly? You can test this by running
cd $JOSHUA/src/test/resources/bn-en/hiero/
time ./test.sh

Depending on your hardware, of course, that should run in just a few seconds tops.
Incidentally, there are at least two other problems I am hoping you can help address?

I see how to build a big tarball and run Maven that way, but that's a pain for development, since you have to incur the time of rebuilding the tarball. Is there a way to just run against the files in target/classes directly? I've tried this, but the problem is that the jar dependencies don't seem to get downloaded anywhere.

Your response might be, "actually, you just have to regenerated the tarball, that's the proper way to do things." In which case, can you describe how to make eclipse do this? Right now, when I'm developing, eclipse is always compiling files, so you can just switch over and test right away. It's a small but significant paint to have to type "(cd $JOSHUA; mvn compile assembly:single install"

I really like being able to use the -v (verbosity) flag to Joshua, which sets the debugging level. This is much simpler than changing a properties file and linking to that from the command line. Is it possible to set the log4j debug level programmatically based on the value of -v? We could do
-v 0 = OFF
-v 1 = INFO
-v 2 = DEBUG

Or I would be okay to switch to those values directly (e.g., "-v OFF"). But the former seems more Unix-like.

Thanks again for all your work on this!
matt

@thammegowda please revert logging patch.

Just to be clear, not sure it needs to be reverted. We just need to find a way to make sure the config file gets read and that logging gets set to INFO by default. And logging should all go to STDERR, I think, not STDOUT.
In general, too, having INFO prepended to all informative lines is a bit ugly. Is this a useful convention for reasons I can't understand? It's often useful to see the decoder working with just these high-level labels, and I don't mind debug lines being prepended with DEBUG.

Make an entry in logging file?
Parameterized logging as implemented by @thammegowda is magnitudes more efficient than traditional string + <?>Excepton logging
@thammegowda maybe we need INFO degreded (logging sense) to DEBUG
Thank you @thammegowda for trying to understand the logging. When I was doing the build thats something I didn't approach :)

Yes, I hope I was clear that this is all great work. We just have to figure out how to get log4j to read the config file, which it's currently not doing. Then, we can make sure the test cases pass, and merge back into master?

@lewismc @mjpost  Just looked into this. I will debug this issue and come back with my findings soon.
+1 for rethinking about log levels.  we can lower some frequent INFO messages to DEBUG level.

Found the issue with logger config file. Seems like -Dlog4j.configuration to override is no longer supported.
Please review  #18 and merge.

@mjpost I am unable to run test.sh script, because -

Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: no ken in java.library.path
	at org.apache.joshua.decoder.ff.lm.KenLM.(KenLM.java:52)
	... 10 more
Caused by: java.lang.UnsatisfiedLinkError: no ken in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at org.apache.joshua.decoder.ff.lm.KenLM.(KenLM.java:43)
	... 10 more
Exception in thread "main" java.lang.RuntimeException: * FATAL: could not find a feature 'LanguageModel'

How do I pass this ?

These are the logs I am getting when the level is OFF, WARN, INFO and DEBUG.
I ran echo "Subject is not object" | bin/joshua > file.txt
off.txt
warn.txt
info.txt
debug.txt

OK @mjpost  thanks
@thammegowda for starters, more or less every file that you write at Apache, and that you wish to build community around should probably have a licensse header
https://github.com/apache/incubator-joshua/blob/JOSHUA-252/src/main/resources/log4j.properties
does not.
For starters, can you please update it.
Can you also please scope the Slf4j over Log4j I implemented over on Nutch for an example of how to help @mjpost with the logging issues.
It is important in this build that we retain the efficiency of everyones commits over the build. The build is an enabler not a master.

Its more than overdue as well for me to say @thammegowda thank you. The Logging effort is absolutely paradise. I looked at a lot of code during reformat and your logging patches are very welcome. Thank you Sir.

Okay, great! That worked. I assume the edits to $JOSHUA/bin/joshua mean that eclipse compiled files will override the jar? So I can do fast development in Eclipse?
We still have to solve the KenLM problem. My tests ran because I linked $JOSHUA/ext and $JOSHUA/lib to directories on master. Do we have ideas for how to pull in KenLM and compile it? And BerkeleyLM? I think that's the big thing remaining.
After that, I'll have to go over the log messages in a bit more detail.

I assume the edits to $JOSHUA/bin/joshua mean that eclipse compiled files will override the jar? So I can do fast development in Eclipse?

Yes all you need to execute in your terminal is mvn eclipse:eclipse

We still have to solve the KenLM problem.

Yes we do :(

Do we have ideas for how to pull in KenLM and compile it? And BerkeleyLM?

https://bitbucket.org/atlassian/bash-maven-plugin
Honestly thats the most I can find right now without hacking a Maven Github download build plugin. I am fed up of XML @mjpost  :0 :) :) :)

Hi @lewismc Thanks so much.

For starters, can you please update it.

Oh, I missed the licence header. Got it. I do not yet have write permissions to directly fix it, so  guess I have to raise another PR to fix it.

Can you also please scope the Slf4j over Log4j I implemented over on Nutch for an example of how to help @mjpost with the logging issues.

Sorry, I am not clear what do you mean?

The Logging effort is absolutely paradise.

Thanks :-)

@mjpost

Okay, great! That worked. I assume the edits to $JOSHUA/bin/joshua mean that eclipse compiled files will override the jar? So I can do fast development in Eclipse?

Seems like you have hard-time with eclipse maven integration.
last time when I used eclipse there was a eclipse plugin that recognized maven projects. That plugin took care of all the complexities under the hood. If you have proper setup, you can just go to "JoshuaDecoder" (or whichever class you want to run) and "Run main" (no need to package, because that takes time)
P.S. I use Intellij Idea opensource edition and the maven integration is a breeze.

yes @thammegowda the process is welcome and appreciated Sir. on a side note hove you tried th Docker containersTZ?

@lewissmc Thank you very much, Sir.
Joshua in Docker - not yet tried so far, I will definitely try it.
I am much interested in your PR to Tika to integrate this translator :+1:

ack

Another issue: BerkeleyLM is currently pulled from the Maven repository, but it is very old (1.1.2, but 1.1.6 is available). I'm pretty sure its author has abandoned it; how can we get it updated? Joshua has a few bug fixes on it, even. One option: we could just roll the codebase into Joshua (it's Apache 2.0).

So here is what is outstanding:

We need to either (a) get a new BerkeleyLM in Maven (anyone know how to do this?) or (b) pull the codebase into Joshua (is this permissible, including the renaming?)
I think we can safely leave Thrax as an external dependency that the user has to install, and provide instructions on the website for how to do so.
We can do the same for KenLM, and make the BerkeleyLM build and runtime tools the default.

That way we don't have to mess with any shell scripts from Maven, which is not advised (because of portability concerns).
matt

Unless there are any objections, I'd like to merge this branch back into master. There are a few small issues, but none that can't be addressed with documentation. I'd love to do this either tonight or tomorrow, so please voice any objections as soon as possible.

+1 great work! watch out there are conflicts to resolve before merging @mjpost

Ho @mjpost can you point me at the latest BerkeleyLM codebase? I can Mavenize it and get it in to Maven Central so we can use it as a dependency in pom.xml

https://github.com/joshua-decoder/berkeleylm/

Actually @chrismattmann it's just a fast-forward at this point — all is merged on JOSHUA-252, it's just not showing up on this PR.
https://github.com/apache/incubator-joshua/tree/JOSHUA-252


Code commentary.

hakunator.pm6 content: I am not too sure about a few of the words in your lists; 'shinny' is not an adverb, it is a verb. Probably should be shiny. 'hushy' seems to be a slang term. 'hola' is not a noun.
hakunator.pm6 code: Not too bad. A little verbose perhaps. Certainly not taking advantage of many Perl 6 idioms. Take a look at https://gist.github.com/thundergnat/fa1dcae2393c89120c4f for how I would have written it. (Feel free to use or ignore any/all of the changes I made)
In 010-use.t:  (\N** 4 ) is looking for any 4 characters that aren't newlines. Probably want (\xdigit ** 4 ) instead. ( 4 hexadecimal digits).
([<[a..z]><[a..z]>+]) may be better written as (<[a..z]> ** 2 .. *) or just <[a..z]> ** 2 .. * if you aren't using the matches for anything.
There isn't any \n added in the haikunate routine so it is pointless to check for optional newlines. '\n?'
Other than that, not too bad. Thanks for contributing!  »ö«   :-)

Wow, your code looks way cleaner. I will definitely update my code with your version.
In terms of Regex you are right. I just put this together yesterday and I had to get the hang out of it.
Thank you!


Allow inclusion of custom redirect_uri when configuring provider

In certain scenarios (for instance when coming from a slack team style subdomain) we want the ability to set a custom redirect_uri, since Slack doesn't allow a wildcard for the subdomain.
Example: Your application allows creation of arbitrary team subdomains (exactly how Slack works). The user clicks the Add to Slack from subdomain1.example.com. However the allowable redirect_uri in the Slack App settings is www.example.com/slack/*. This will break because the gem will currently set the redirect_uri from the current location.

one year ago... will be fine to get it merged @kmrshntr


feat: expose setup function

Superseded by https://github.com/fredrikaverpil/neotest-golang/pull/100


bypress validate content type

bypress validate content type

  -http-loader-accept string
        HTTP Loader set request Accept header and validate response Content-Type header (default "*/*")

Try bypass content type by setting wildcard accept?
HTTP_LOADER_ACCEPT=*


Design suggestions

Thrilled that someone built a working compass!!   I started to build a similar compass card a few weeks back, but didn't have time to dig into why it wasn't displaying properly.
A few suggestions:

I feel like the wind speed is the most important data—and actionable by the user—as opposed to the wind direction. I'd suggest allowing user specification of the larger data point in the ring, and have the direction smaller and somewhere else (after all the direction is already on the arrow on the ring).

Visually, I'd suggest adding a small N to the top of the ring to indicate North. Small detail, but ties this all together.

Here is a similar compass from Ambient Weather.  I like that this can cleanly display optional data like maximum gust wind speed for the day.

I have to thank you, because I got the idea for the card while having a look at your windcompass repo.
They are all good suggestions, I already thought of the possibility to select a different type of indicator

arrow inwards (wind)
arrow outwards (e.g. lightning location see screenshots in #10)
circle (e.g.for sun/moon)

And you're right, the direction is already made visible by the indicator, so it could be a lot less in your face and make space for other usefull info.
Consider this added to the wish/todo list
/offtopic, as the devcontainer in the repo is not working for me anymore, I cannot easily develop at this moment. Somehow the container is running, open ports for a develop home assistant instance, but the browser on my host throws a ERR_CONNECTION_REFUSED
When I get this fixed I can work on the wishlist again.

Not all imporvements done yet...

Thanks for the card!
A suggestion is that the arrow changes colors according to user-set wind speeds, like in bar-chart, and big number card.
Thanks again.

Thanks for the card!
A suggestion is that the arrow changes colors according to user-set wind speeds, like in bar-chart, and big number card.
Thanks again.

Issue #12 covers this. As this is a bit more work, it will take a bit longer to implement.

More design suggestions here.


Use new Authentication Provider API in vscode

https://github.com/microsoft/vscode/issues/88309
https://code.visualstudio.com/updates/v1_46#_accounts-management
vscode now has a built-in authentication provider so no need to have our own authentication UI

Interesting.  Can you expand on the use case for this or provide more context for it's implementation?

This seems to be a better more baked in way of managing and syncing credentials for logging into the bigips.  Also seems to be in active development.  will keep an eye on this as the final details shake out and see how this can applied to this extension

this seems to be an experimental feature for vscode at this time.  moving to backlog for later review.


Please add difince and annajung from VMware

annajung is part of KF 1.4 release team
in addition to the commits:
https://github.com/kubeflow/pipelines/commit/0d84877c98726be6417474816b38baac92b45ee7
https://github.com/kubeflow/pipelines/commit/ebf5310764db36940c5025f3941058f29d5d80bb
difince has the following commits:
https://github.com/kubeflow/pipelines/commit/a8ffc50285bbf36a080652e24f447fa9297b26d8
https://github.com/kubeflow/pipelines/commit/fd72b3c9d59803726c41f287cd1ee98b8f95cc5a
Signed-off-by: Malini Bhandaru<EMAIL_ADDRESS>

Hi release team members @kimwnasptd @DavidSpek @RFMVasconcelos, would anyone be able to sponsor me in addition to Malini?

@annajung I'm happy too!
/lgtm
/cc @Bobgy

/lgtm
/approve
Thank you for your contributions!
Welcome to Kubeflow!


Custom port configuration setting name in Application Settings

Under 'Diagnose and solve problems' -> 'Availability and Performance' -> 'Application Logs ' section in App Services, it clearly states that the application settings for port change should be mentioned under setting named - 'WEBSITES_PORT' and not 'PORT'.
But under the note section in the following link - https://docs.microsoft.com/en-us/azure/java/spring-framework/deploy-spring-boot-java-app-on-linux#create-a-web-app-on-linux-on-azure-app-service-using-your-container-image , setting name for custom port is mentioned as 'PORT' instead of 'WEBSITES_PORT' which needs to be corrected.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6f50e3e9-904e-0ffd-1d18-2b7be7d15dd9
Version Independent ID: 191f1eb6-dcab-3ec8-bbf3-1a8109116429
Content: Deploy a Spring Boot Web App on Azure App Service for Container
Content Source: java/spring-framework/deploy-spring-boot-java-app-on-linux.md
Service: app-service
GitHub Login: @bmitchell287
Microsoft Alias: brendm

PR# 75 - 'PORT should be changed as WEBSITES_PORT in Application settings' closes this issue. Please merge.

#sign-off

@MikeDodaro  Hi Mike, I m not able to link the PR# 75 to this issue. If you can link it you can merge the PR I guess.

#sign-off

@gandhirajan @MikeDodaro - I merged PR #75, thanks for the submission.

Merged

@gandhirajan @MikeDodaro - I merged PR #75, thanks for the submission.
Thanks a lot @MikeDodaro @bmitchell287


NIFI-9726 Remove duplicate nifi-utils dependencies from graph modules

Description of PR
NIFI-9726 Removes duplicate dependency declarations for nifi-utils from nifi-neo4j-3-cypher-services and nifi-other-graph-services to resolve Maven build warnings.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[X] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[X] Has your PR been rebased against the latest commit within the target branch (typically main)?

[X] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

+1 LGTM, sorry I missed those being re-introduced while reviewing the recent graph PR. Good catch and thanks for the fix! Merging to main


Start the executor and implement the first Node type.

This is the beginning of the actual Executor implementation. The overall change is very large, but it's broken up into several parts to hopefully be easier to follow:

Defining the basic interface of an executor.
Filling in the very skeleton of that interface.
Implementing a stored-parameter replacer.
Creating the first Node implementation, the UDF node.

This is basically the bare-minimum task graph implementation. It's not complete, since it doesn't handle SQL, arrays, or inputs, but (as shown by the tests) can build viable graphs. Other node types will follow in subsequent changes, but I wanted to get this out before it got even larger.
[sc-14031]

Please feel free to leave post-submission review comments on this—I’m submitting it so that I can build on top of it, but I am happy to answer questions and address concerns in follow-ups. With changes this big, there will no doubt be a bit of cleanup that can be done afterwards.


Documentation maintenance updates

This PR deals adds missing applications and modules to the documentation.
In detail:

minor text updates to the main readme and documentation text (mostly removing duplications).
adding of missing applications and modules
fix of a couple of bugs in docstrings leading to sphinx errors or warnings
simplified theme (note theme change!), allow to remove a couple of items from conf.py. This theme looks a bit nicer, is pydata-thema based (as those of many other cta tools)

Building the documentation in the container is not working for me:
Running Sphinx v7.3.7

Traceback (most recent call last):
  File "/usr/lib64/python3.11/importlib/metadata/__init__.py", line 371, in __getitem__
    return next(iter(self.select(name=name)))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workdir/env/lib64/python3.11/site-packages/sphinx/registry.py", line 155, in preload_builder
    entry_point = builder_entry_points[name]
                  ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib64/python3.11/importlib/metadata/__init__.py", line 373, in __getitem__
    raise KeyError(name)
KeyError: 'build'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workdir/env/lib64/python3.11/site-packages/sphinx/cmd/build.py", line 332, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workdir/env/lib64/python3.11/site-packages/sphinx/application.py", line 232, in __init__
    self.preload_builder(buildername)
  File "/workdir/env/lib64/python3.11/site-packages/sphinx/application.py", line 328, in preload_builder
    self.registry.preload_builder(self, name)
  File "/workdir/env/lib64/python3.11/site-packages/sphinx/registry.py", line 157, in preload_builder
    raise SphinxError(__('Builder name %s not registered or available'
sphinx.errors.SphinxError: Builder name build not registered or available through entry point

Sphinx error:
Builder name build not registered or available through entry point
make: *** [Makefile:20: build] Error 2

If it doesn't look for you, maybe you picked up a container without sphinx_book_theme installed (remember that this has been added with this pull request).
Install temporary in your container this package with pip install. Latest after merge-to-main this will be included in the latest container.

@VictorBarbosaMartins - thanks for the review.
The test badges show the status of the tests from the last scheduled runs. This was last night and before integration of the housekeeping changes from this morning. So 'failed' was correct (should look better tomorrow, otherwise we have to fix it)


Lineage not working correctly with versioned models

Expected behavior
The DBT model should be valid and the lineage displayed
Actual behavior
When using versioned models via https://docs.getdbt.com/docs/collaborate/govern/model-versions DBT interprets filenames from e.g. schema.yml/model.yml/<any_yml_with_models_block> file via the versions, not from the actual model names (i.e. joins them). This breaks DBT Power Users lineage graph by giving the warning A valid dbt file (model, seed etc.) needs to be open and active in the editor area above to view lineage and an empty graph even though that model is valid DBT model.
Steps To Reproduce
This should be reproducible by

Create two models a_v1.sql and b_v1.sql with some simple working SQL
Create models.yml with

---
version: 2
models:
  - name: a
    latest_version: 1
    versions:
      - v: 1
   - name: b_v1

Focus VSCode into a_v1.sql and then b_v1.sql

The a will fail to render as lineage node, but b is alright
Log output/Screenshots
A valid dbt file (model, seed etc.) needs to be open and active in the editor area above to view lineage
Operating System
MacOS 14.6.1
dbt version
1.8.6
dbt Adapter
Redshift
dbt Power User version
v0.45.0
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Duplicate of https://github.com/AltimateAI/vscode-dbt-power-user/issues/709

hey @ellipsis-dev can you implement this?


UnicodeDecodeError when downloading GLUE-MNLI

When I run
dataset = nlp.load_dataset('glue', 'mnli')

I get an encoding error (could it be because I'm using Windows?) :
# Lots of error log lines later...
~\Miniconda3\envs\nlp\lib\site-packages\tqdm\std.py in __iter__(self)
   1128         try:
-> 1129             for obj in iterable:
   1130                 yield obj

~\Miniconda3\envs\nlp\lib\site-packages\nlp\datasets\glue\5256cc2368cf84497abef1f1a5f66648522d5854b225162148cb8fc78a5a91cc\glue.py in _generate_examples(self, data_file, split, mrpc_files)
    529
--> 530                 for n, row in enumerate(reader):
    531                     if is_cola_non_test:

~\Miniconda3\envs\nlp\lib\csv.py in __next__(self)
    110             self.fieldnames
--> 111         row = next(self.reader)
    112         self.line_num = self.reader.line_num

~\Miniconda3\envs\nlp\lib\encodings\cp1252.py in decode(self, input, final)
     22     def decode(self, input, final=False):
---> 23         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
     24

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 6744: character maps to <undefined>

Anyway this can be solved by specifying to decode in UTF when reading the csv file. I am proposing a PR if that's okay.

It should be good now, thanks for noticing and fixing it ! I would say that it was because you are on windows but not 100% sure

On Windows Python supports Unicode almost everywhere, but one of the notable exceptions is open() where it uses the locale encoding schema. So platform independent python scripts would always set the encoding='utf-8' in calls to open explicitly.
In the meantime: since Python 3.7 Windows users can set the default encoding for everything including open() to Unicode by setting this environment variable: set PYTHONUTF8=1 (details can be found in PEP 540)
For me this fixed the problem described by the OP.


EHN: Added new converter type for video data, in progress

Just created a converter for video stream, will work on the igtliodevice soon.

Thanks so much for effort on video streaming. Since Junichi is going to be joining us at our satellite project week, would you be willing to bring him up to speed on how to configure this setup (if he's not already comfortable)? I'd love to set this up for our streaming ultrasound.

Hi Adam,
Sure, i will update Junichi on the the VideoStreaming stuff.
I also made some modification on the plusBuild to integrate the video Streaming, though still in my forked branch.
I will try to merge it back to PlusBuild soon.
Hope it will be helpful If you are testing ultrasound streaming  using Plus library.

That's awesome, I will see if I can make time to check this out during our project week.
(pasting link for my convenience)
https://github.com/leochan2009/PlusLib

Would you be able to investigate the OpenIGTLinkIO build? It is currently failing while looking for igtlVideoMessage.h

Hi Adam,
did you link to the correct version of OpenIgtlink?
The cmake variable OpenIGTLink_DIR should set to build directory this branch:
https://github.com/openigtlink/OpenIGTLink/tree/VideoStreamMerge
Let me know if you have further question.
Best Regards,
Longquan

Hmm, we do not want to point to a branch of OpenIGTLink. Could you merge that work into master or revert the changes to OpenIGTLinkIO?

I see, i need to discuss with Junichi about the merge.
As this work is still primitive...

I will remove the OpenIGTLinkIO link to video message for now

Hi Adam,
I just tagged the previous openigtlink master branch as release-3.0.0, and create a pull request to merge the video streaming branch into the master.
Let's wait for Junichi's response.
Best Regards,
Longquan

@adamrankin ,
I just merge the video streaming branch into the master branch.
Are you meeting with Junichi in the project week?
Any feedbacks are welcomed.
Best Regards,
Longquan


Technical analysis is cringe and a scam

Firstly, this is a really cool library and some of it (SMA and EMA) is useful for things that aren't technical analysis.
Secondly, you're clearly a smart guy and I can see you're interested in other quant trading and quant finance stuff. I would encourage you to focus on those things instead, because are actually useful and won't be embarrassing like TA is. And also because the existence of a high-performance TA library like this serves to legitimize TA.

why is TA not useful? I think people do not need to work on the things you would like them to.

why is TA not useful? I think people do not need to work on the things you would like them to.

Because it's financial astrology.  The people peddling it are predominantly liars and schmucks.

Yeah you can do whatever you want with TA since it's nonsense to begin with.
I can't parse the other things you're saying

ok good i guess, since that means we dont have to argue anymore


Server port autoselect doesn't work reliably with SO_REUSEPORT on.

Context
When creating a grpc server it is possible let grpc pick an unused port automatically by selecting ":0" (wildcard port).
Problem
When grpc.so_reuseport is enabled (currently the default) and there's a lot of grpc servers being spun up concurrently (e.g. in tests), there is a race and clients occasionally connect to a wrong server (which ends up listening on the same port).
Crosstalk between different client & server pairs (that are supposed to be hermetic) has been shown to cause flakiness of some of our tests (e.g. in C#).  In real-life scenarios this is not likely to be a problem, unless there is a lot of servers and client being created & shutdown concurrently.
Workaround
Setting grpc.so_reuseport  channel argument to 0 for the servers will mitigate the problem (that's not usable in general because it SO_REUSEPORT is important for performance, but it's good enough for tests).

Hi,
@kpayson64
Is SO_REUSEPORT disabled by default in grpc? I am seeing an issue when trying to bringup multiple grpc servers.
[root@tuns-deployment-2602672641-089j6 tuns]# python /var/tmp/test_grpc.py &
[1] 3484
[root@tuns-deployment-2602672641-089j6 tuns]#
[root@tuns-deployment-2602672641-089j6 tuns]#
[root@tuns-deployment-2602672641-089j6 tuns]# netstat -an |grep 3333
tcp6       0      0 :::3333                 :::*                    LISTEN
[root@tuns-deployment-2602672641-089j6 tuns]# python /var/tmp/test_grpc.py &
[2] 3491
[root@tuns-deployment-2602672641-089j6 tuns]# E0607 21:59:24.669550425    3491 server_chttp2.cc:40]        {"created":"@1528408764.669478381","description":"No address added out of total 1 resolved","file":"src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":307,"referenced_errors":[{"created":"@1528408764.669472752","description":"Failed to add any wildcard listeners","file":"src/core/lib/iomgr/tcp_server_posix.cc","file_line":345,"referenced_errors":[{"created":"@1528408764.669452377","description":"Unable to configure socket","fd":6,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":202,"referenced_errors":[{"created":"@1528408764.669444826","description":"OS Error","errno":98,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":175,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1528408764.669471667","description":"Unable to configure socket","fd":6,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":202,"referenced_errors":[{"created":"@1528408764.669467475","description":"OS Error","errno":98,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":175,"os_error":"Address already in use","syscall":"bind"}]}]}]}

@hexinw
In the manylinux binary distribution, yes it is disabled.  SO_REUSEPORT isn't part of the manylinux standard, so we can't distribute binaries with it enabled.
If you want to use SO_REUSEPORT with Python, you will need to install from source.

Great. Works now after re-install from source. Thanks @kpayson64.
[root@tuns-deployment-2602672641-s06qd grpc]# python /var/tmp/test_grpc.py&
[1] 2772
[root@tuns-deployment-2602672641-s06qd grpc]# netstat -an |grep 3333
tcp6       0      0 :::3333                 :::*                    LISTEN
[root@tuns-deployment-2602672641-s06qd grpc]# python /var/tmp/test_grpc.py&
[2] 2779
[root@tuns-deployment-2602672641-s06qd grpc]# ps axww
PID TTY      STAT   TIME COMMAND
241 ?        Ss     0:00 bash
2772 ?        Rl     0:09 python /var/tmp/test_grpc.py
2779 ?        Rl     0:03 python /var/tmp/test_grpc.py
2786 ?        R+     0:00 ps axww
[root@tuns-deployment-2602672641-s06qd grpc]# cat /var/tmp/test_grpc.py
from concurrent import futures
import grpc
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
port = server.add_insecure_port("[::]:%s" % 3333)
server.start()
while True:
pass


[Bug] <can't upgrade to 1.8>

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
can't upgrade to 1.8
Expected Behavior
should be able to?
Steps To Reproduce
/
Relevant log output
No response
Environment
- OS:
- Python:
- dbt:

Which database adapter are you using with dbt?
No response
Additional Context
No response

Thanks for reaching out @mrmarvisfrancesco.
We haven't released v1.8 yet, so that explains why!
We do have a beta available on PyPI, and if you are using pip, you can install it by adding the --pre flag to your installation command. See here for examples.

Hi Doug, thanks. I got confused with the beta version. Thank you
Francesco AngileriData & Analytics Manager
Stores: Amsterdam
https://www.mrmarvis.com/pchooftstraat-amsterdam?utm_source=signature&utm_medium=email&utm_campaign=signature
| The Hague
https://www.mrmarvis.com/hoogstraat-thehague?utm_source=signature&utm_medium=email&utm_campaign=signature
| Laren
https://www.mrmarvis.com/sintjanstraat-laren?utm_source=signature&utm_medium=email&utm_campaign=signature
| Maastricht
https://www.mrmarvis.com/kleinestaat-maastricht?utm_source=signature&utm_medium=email&utm_campaign=signature
| Antwerp
https://www.mrmarvis.com/schuttershofstraat-antwerp?utm_source=signature&utm_medium=email&utm_campaign=signature
| Munich
https://www.mrmarvis.com/odeonsplatz-munich?utm_source=signature&utm_medium=email&utm_campaign=signature
Office: Huddestraat 3, 1018 HB Amsterdam
https://goo.gl/maps/6yFC3PRPerknNGMm6
Instagram https://www.instagram.com/mrmarvis/ | Facebook
https://www.facebook.com/MRMARVIS.official | LinkedIn
https://www.linkedin.com/company/mrmarvis
https://www.mrmarvis.com/?utm_source=signature&utm_medium=email&utm_campaign=signature
https://www.mrmarvis.com/blog/we-are-a-certified-b-corp?utm_source=signature&utm_medium=email&utm_campaign=signature
The information contained in this message is of a confidential nature and
legally protected under applicable law. The message is intended solely for
the addressee(s). If you are not the intended addressee, you are hereby
notified that any use, forwarding, dissemination, or reproduction of this
message is strictly prohibited and may be unlawful. If you are not the
intended addressee, please contact the sender by return e-mail and destroy
all copies of the original message.
On Wed, 3 Apr 2024 at 14:54, Doug Beatty @.***> wrote:

Thanks for reaching out @mrmarvisfrancesco
https://github.com/mrmarvisfrancesco.
We haven't released v1.8 yet, so that explains why!
We do have a beta available on PyPI
https://pypi.org/project/dbt-core/1.8.0b1/, and if you are using pip,
you can install it by adding the --pre flag to your installation command.
See here https://github.com/dbt-labs/docs.getdbt.com/issues/2144 for
examples.
—
Reply to this email directly, view it on GitHub
https://github.com/dbt-labs/dbt-core/issues/9847#issuecomment-2034530260,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZHBBBAQPAJBU5AWQKYABQTY3P32HAVCNFSM6AAAAABFVAYZLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZUGUZTAMRWGA
.
You are receiving this because you were mentioned.Message ID:
@.***>


Junos structured output parsing problem

Hi,
It seems large output causes structured output parsing problem:
[CRITICAL] 20211022 06:25:58 | hyperglass.parsing.juniper:94 | parse_juniper → no element found: line 999, column 98
[CRITICAL] 20211022 06:25:58 | hyperglass.exceptions:40 | __init__ → [DANGER] Error parsing response data

When default command was modified to
show route protocol bgp table inet.0 {target} best detail | display xml | no-more
the problem seems to be solved.
Basically I think you should add | no-more at the end of default commands.

Due to several strange issues similar to this, and after significant performance improvements in the Netmiko library, I've decided to remove Scrapli from hyperglass v2.0 entirely (and have already done so in the v2.0.0 branch). I think it will simplify things a great deal.
@brevius - did setting the driver to netmiko solve this for you? If so, I'm going to close this out since it's technically fixed in 2.0, which I'm hoping to make available soon.

Yes - with netmiko everything works fine.


Try partial_decompress=True

Especially for NWPs, where we often only want a single value per chunk.
Also try in combination with uncompressed chunks
See https://github.com/zarr-developers/zarr-python/pull/667

Tried it.  Doesn't seem to speed things up.
Using consolidated metadata seems to significantly (double?) the read speed though.
Code:
# Pretty sure this sets partial_decompress to True:
import zarr
array = zarr.open_array(
    FILENAME, path='stacked_eumetsat_data', partial_decompress=False, mode='r'
    storage_options=dict(consolidated=True))

%%time
data = array[:12, :128, :128, 0]  # takes about 35 ms with or without partial_decompress.

This code might set partial_decompress=True, but I'm not certain!
ds = xr.open_dataset(
    FILENAME,
    engine='zarr',
    consolidated=True,
    storage_options=dict(partial_decompress=True)
)


Using ng12 partial lib and source map causing build memory exception

🐞 Bug report
Command (mark with an x)

[ ] new
[x] build
[x] serve
[ ] test
[ ] e2e
[ ] generate
[ ] add
[ ] update
[ ] lint
[ ] extract-i18n
[ ] run
[ ] config
[ ] help
[ ] version
[ ] doc

Is this a regression?
Worked with full ivy libs. Stopped working with partial ivy libs
Description
I have an app with the following source map settings:
            "sourceMap": {
              "scripts": true,
              "styles": true,
              "vendor": true
            },

I updated to angular 12 and changed my lib (that my app is using) to build using partial ivy. Now I get memory exception when building the app. It works with my lib in full ivy.
Changing the vendor property above to false will make it build. But I need the vendor script debugging, i.e. for it to be true.
🔬 Minimal Reproduction
See above
🔥 Exception or Error

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF713284C9F napi_wrap+111007
 2: 00007FF713228226 v8::base::CPU::has_sse+59910
 3: 00007FF713229126 node::OnFatalError+294
 4: 00007FF713B02ABE v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF713AE788D v8::SharedArrayBuffer::Externalize+781
 6: 00007FF7139909DC v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1516
 7: 00007FF71399BDFA v8::internal::Heap::ProtectUnprotectedMemoryChunks+1258

🌍 Your Environment

Angular CLI: 12.2.5
Node: 14.17.6
Package Manager: yarn 1.22.5
OS: win32 x64

Angular: 12.2.5
... animations, cli, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1202.5
@angular-devkit/build-angular   12.2.5
@angular-devkit/core            12.2.5
@angular-devkit/schematics      12.2.5
@schematics/angular             12.2.5
ng-packagr                      12.2.1
rxjs                            6.6.7
typescript                      4.3.5

Anything else relevant?
No

This seems like a bug but we'll need to look at a reproduction to find and fix the problem. Can you setup a minimal repro please?
You can read here why this is needed. A good way to make a minimal repro is to create a new app via ng new repro-app and adding the minimum possible code to show the problem. Then you can push this repository to github and link it here.
This might be related to your directory structure so its really important to get an accurate repro to diagnose this.

I'm having the same problem. I do not have a repo (the repo is my whole app indeed) which has about 60 angular libraries modules and many other common/shared modules.
It's hard to setup a small repo here, I think the problem is in fact coming precisely because the repo is big (@rbirkgit might confirm ?)
Setting souceMaps to false makes everything compiling well again.

Yes, our library has many modules too (33). I am not sure how how to replicate this. For now we are back using the full ivy mode on our libraries as for us the partial mode is not building.
We have a demo app in the same repo as the library, and that one compiles, but not the main app in another repo. My guess is that the demo app is not including as many 3rd party vendor scripts? and maybe that's why the main app cannot handle it as it just reaching over the memory limit.

Unfortunately, without a reproduction, even shared privately we'll be unable to investigate this issue.

I'm trying to see with my company if it's possible to host the code of our app on a private online repo. This might be complicated because of internal confidentiality policy.

@alan-agius4  I might get this friday a positive answer to share the whole app code on a private repo. I'll keep you in touch. Is that a possible solution ?

I got the approval from my boss to host the app on a private shared repo. I will do that asap.

Thanks djflex68! I think it's a good chance we have the same issue.

I will provide this to you tomorrow. Is it possible to send you repo URL per e-mail instead of posting it here ?
Thanks

Yeah, or since it’s hosted on GitHub you can invite us to the repo via usernames.

I've sent you an e-mail about this ;-)

Thanks for the update and very exciting!
If it would help and easy to apply, I could also test on my project where we have the same issue.

@JoostK  many thanks for the investigation. I will have a look tomorrow and tell you if that fixed the issue ;-)


Item Modal: User Can Move Item to Current List

If "Move item to" is not selected, the item will be copied to the current list.
Instead:

display an error if that modal is not selected.
Grey out the "Update" and "Next" buttons until the form is in a valid condition.

The ability to move items from the item modal was removed.


HOCS-5564: Implement business events

@SWhitlam/ @LiamMacP, we've resolved the comments here, please could you have another look when you get the chance


Added missing parameter in example

There is an extra parameter for onDragOver and onDrop handlers, the resource itself, so I've added it to example if someone else has the need for it.

@MilosPaunovic Was this supposed to go to the next branch?

@hawkeye64 We were using Q1 at the time, so the 3.x.x version of calendar was used. I haven't got the chance to play with v4.x.x yet.

It went to the right place. I double-checked.


[Multithread] Make plan executor asynchronous

This is a separate PR that makes the necessary interface changes. There's no real multithreading in this PR.
The key is to have Query::Execute have the following interface:
void Query::Execute(
    std::unique_ptr<executor::ExecutorContext> executor_context,
    QueryResultConsumer &consumer,
    std::function<void(executor::ExecuteResult)> on_complete,
    RuntimeStats *stats = nullptr);

The parameters are now baked into executor_context;
The query results are stored in the consumer;
When execution completes, the user-specified callback on_complete gets invoked.

@pmenon This PR changes the interface of the codegen Compile and Execute, so it would be good for you to make a review, too. This also includes some of the cleanups that were discussed before.

Coverage decreased (-0.01%) to 75.84% when pulling 44fbc0d07314f6f0691b12d41a4cc2e939e43a4d on phisiart:callback-exec into 12133fada1e430fc2507afadac52ab3d7cdcbad2 on cmu-db:master.

@pmenon Review comments are addressed.
@seojungmin I have ruthlessly trimmed down threadpool/*. More specifically:

No more class Task, just std::function.
No more class Worker, just std::thread.
No more class TaskQueue, just peloton::LockFreeQueue<std::function<void()>>.

Coverage decreased (-0.03%) to 75.802% when pulling a45866a63476ce68886ff8fe7e3d49fbdaaeacdc on phisiart:callback-exec into 4ac67de3363e3614199abee8fff4ba899abf25d0 on cmu-db:master.

Coverage decreased (-0.04%) to 75.791% when pulling a45866a63476ce68886ff8fe7e3d49fbdaaeacdc on phisiart:callback-exec into 4ac67de3363e3614199abee8fff4ba899abf25d0 on cmu-db:master.

I will merge this next when the build passes.

Coverage decreased (-0.01%) to 75.809% when pulling f5428ed52cb93d94b33ffe4b04f3e1085c97f549 on phisiart:callback-exec into 638eee7facc7146f9670224e7c45fd77fd0d9931 on cmu-db:master.


[resolves #359] Use camel-parent BOM defined artefact versions

Refer to this link for build results (access rights to CI server needed):
https://fabric8-ci.fusesource.com/job/wildfly-camel-pullreq/247/
Test PASSed.


Add eslint-plugin-eslint-comments with recommended settings

Adds the eslint-comments linter as a new plugin for gestalt. This only raised one error in our repo which was fixed by adding an enable comment in the file.

Deploy preview for gestalt ready!
Built with commit 10b3853fd8cb6872542291abca5cc5325c38ce68
https://deploy-preview-200--gestalt.netlify.com

Deploy preview for gestalt ready!
Built with commit f39671cb8c897de618e302843882fd9bf34872b4
https://deploy-preview-200--gestalt.netlify.com


Replace nodeunit by mocha

sweet!
On Thu, Feb 23, 2012 at 9:55 PM, Andr Rodrigues <
<EMAIL_ADDRESS>
wrote:

Reply to this email directly or view it on GitHub:
https://github.com/WindowsAzure/azure-sdk-for-node/issues/98

done.


Rename hyperkube_image variable to kubernetes_image_uri for consistency

with kubernetes_binaries_uri, the default kubernetes deployment method.
(KRAK-407).

please retest this

please retest this

discussed yesterday, I'm opposed to the term "uri" since google doesn't show it to be idiomatic when referring to docker images
for discoverability we could call it hyperkube_container_image to match the container deployment mode?  otherwise let's close this out

After rereading RFC 3986 I am convinced that uri is appropriate, however it is not idiomatic. Since we are refactoring this code now, I propose we close this and name the binary and image identifierskubernetes_* in the new branch.


Allow listeners execution for subrequests

Should we really limit those listeners to master requests ?
Here is my use case:
We have a BulkRequest system that allows to execute multiple requests at once, through a payload describing the requests to execute.
This system directly use the kernel and sub requests to handle the described requests and gather their responses to be finally sent in a single one to the client.
This system won't work anymore with the latest version of the api bundle.

@ogizanagi the problem I see with you PR is that you are enabling SUB_Request for a use-case which will led to many more calls for the vast majority of use cases.

@dupuchba : Yes, but actually, the question is: Does it has more benefits than pros, as I don't see why sub requests won't be able to return a proper response for an "internal api call".
The impact is very light, as listeners do return very early if they are applicable or not. So not much more code is executed.

I agree with @ogizanagi. It was a reflex to add that test but there are several use cases where it is useful to have a subrequest returning an API result, for instance a Twig template embedding a JSON-LD API result in a script tag for SEO.

OK then if everyone sees a benefit, it should be added :-)