[haxe] Reformatter shouldn't reformat nested string interpolation expression blocks

'${'${1}'}'

turns into
'${'$ { 1 } '}';

No problems when they're not nested.

This was fixed by #1012. :)
@Neverbirth Is this currently unit-testable?

Wait, nevermind, I can actually still reproduce this... However, only in the cloned dev repo, not my real FD installation (despite both being on the same commit). Different settings perhaps?

I'd say that there is no change in that PR that could have affected this in any way.
At first glance the related part of code seems to be fully testable through ASComplete OnChar (because I guess this issue comes from ReformatLine).

Looks like the DEFAULT_SPACEDCHARS  value change by @elsassph makes the difference (see https://github.com/fdorg/flashdevelop/commit/cb55111d0e6496a3c2f84ff41400cbaa29f585e2). Doesn't fix the underlying issue, but at least it doesn't happen by default with new settings...

Same for #368.

OK, so this is for the reformatter, I thought that it was completely disabled for Haxe. I don't remember about that part of code, but I guess it should be unit testable.


Remove pre-cleaning behavior

I want the generated class to be written to my existing src folder, instead of the build folder.
For example, if I have a structure like below:
app
  /src
      /main
         /java
            /models
                SomePojo.java
        /avro
            AnotherPojoSchema.asvc

And I have this build.gradle
task generateAvro(type: com.commercehub.gradle.plugin.avro.GenerateAvroJavaTask) {
    source("src/main/avro/AnotherPojoSchema.avsc")
    outputDir = file("src/main/java/models")
}
compileJava.source(generateAvro.outputs)

And this AnotherPojoSchema.asvc
{
  "namespace": "models",
  "type": "record",
  "name": "AnotherPojo",
  "fields": [
    {"name": "FirstName", "type": "String"},
    {"name": "LastName", "type": "String"}
  ]
}

I would like the class AnotherPojo.java to be created under src/main/java/models/.
However, instead what happen is it

wipes (rm -rf) out directory src/main/java/models.   This can be quite dangerous as you can lose your work.
writes generated file under src/main/java/models/models/AnotherPojo.java

On the contrary, if you use the avro utility by itself, it does not behave this way.  It will create the AnotherPojo.java file, while preserving any existing source code in the package path.
java -jar ./avro-tools-1.8.2.jar compile schema src/main/avro/AnotherPojoSchema.avsc src/main/java/.
leaves you with
app
  /src
      /main
         /java
            /models
                AnotherPojo.java
                SomePojo.java

Can this behaviour be changed?
Thanks
Tim

The upstream compiler has a behavior that it silently refuses to re-generate class files that already exist if the class file is newer than the schema file.  This was intended as a performance optimization, and is not something that is currently possible to disable.
In the case where one of the other schema files that was included by reference in the schema file was changed, this behavior results in the generated classes being out-of-date when schema files that are referenced in other schema files are modified.  The solution was to work around the upstream behavior by adding a cleaning behavior.
For your use-case, I recommend something like this:
task generateAvro(type: com.commercehub.gradle.plugin.avro.GenerateAvroJavaTask) {
    source("src/avro")
    outputDir = file("dest/avro")
}
task copyAvro(type: Copy) {
    dependsOn generateAvro
    from "dest/avro"
    into "src/main/java"
}
compileJava.dependsOn copyAvro

This still has a separate, dedicated folder for Avro generation, but results in the generated files being present in your existing source folder.

Thank you.  I think this will work just fine.   I would recommend changing the behavior so that it is not by default - since you could very easily lose work.  Perhaps be explicit about what you are deleting (i.e. only delete files you are replacing), or make the switch so that deleting is not the default behavior, i.e. you need to opt into it via an option.
Other than that, this is a very handy plugin!

Thank you for your feedback.
The upstream Avro compiler may generate many Java files based on a single Avro schema file, and there's no way to know what files it would generate without either running it or parsing the schema files independently.  Thus, I don't think that being explicit about what is being deleted is a viable option.
Adding a configuration setting to disable the pre-cleaning wouldn't make sense, as there isn't a case where it is a recommended approach to disable the pre-cleaning.  Disabling the pre-cleaning results in exposure to risk of inconsistent, incorrect generated files, while the intended usage (a dedicated directory under the buildDir) has little to no downside.
I could take away the ability to configure the output directory of GenerateAvroJavaTask, hoping to eliminate potential for human error, but whatever path was chosen, there's still risk of collision, and it could result in incompatibility with particular environments.
The options that make most sense to me are:

Propose an enhancement to the upstream compiler to remove or allow disabling the "only compile newer schema files" performance enhancement.  If accepted, this would eventually allow removing the pre-cleaning from this plugin.
Update the README for this plugin to warn about the pre-cleaning behavior.
Update the plugin to refuse to refuse to run when the outputDir is not within the project's buildDir, indicating a potentially dangerous configuration.  Possible variant would be to log a warning, but by the time you would see the warning, you may have already lost data.

I'll pursue all of these three options.

Before writing to the avro-dev mailing list to request a feature change, I wanted to have a clear example of what scenario we were talking about improving.  While I'm certain I at one point hit a problem which pre-cleaning solved, I can't recall it clearly enough to write a test to reproduce it.  Upon deeper thought, the only cases I'm coming up with where the pre-cleaning would result in a behavior difference would be:

A schema file is removed (or a section of a multi-type schema file) resulting in an orphaned class file in the outputDir.  This is a situation where the user should manually clean.
You build, and then perform a version control operation which switches your schema files for older versions (which are then out-of-sync with the build output).  This is a situation where the user should manually clean.

Having multiple schema files defining the same type may result in inconsistent behavior, but that is still inconsistent regardless of whether pre-cleaning is performed.
Thus, I'll be removing pre-cleaning entirely until/unless a clearer use-case for it emerges.

Thanks, that makes a lot of sense.   The onus is on the developer to ensure correct schema usage.  It's better to have it fail, cause there may be other implications as you well stated.


Remove generic parameters from this ffxml file

Per discussion on openforcefield issue#75 we want to remove generic parameters from our ffxml files.

I ran across a wildcard error tonight: SMIRKSParsingError: (SMIRKSParsingError(...), 'Error SMIRKS ([*:1]~[*:2]~[*:3]~[*:4]) was not parseable                        into a ChemicalEnvironment'). I have not reduced the problem to something simple, so I don't have an easy, concise example -- but is it possible that removing the generic parameters left some vestiges somewhere else in the code?
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/openforcefield/typing/chemistry/environment.py in __init__(self, smirks, label, replacements)
    466             try:
--> 467                 self._parse_smirks(smirks)
    468             except:

~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/openforcefield/typing/chemistry/environment.py in _parse_smirks(self, input_smirks)
    592                     bondANDtypes=bAND, newORtypes=aOR, newANDtypes=aAND,
--> 593                     newAtomIndex=index, newAtomRing=ring, beyondBeta=True)
    594

~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/openforcefield/typing/chemistry/environment.py in addAtom(self, bondToAtom, bondORtypes, bondANDtypes, newORtypes, newANDtypes, newAtomIndex, newAtomRing, beyondBeta)
    917         # create new bond
--> 918         newBond = self.Bond(bondORtypes, bondANDtypes)
    919

~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/openforcefield/typing/chemistry/environment.py in __init__(self, ORtypes, ANDtypes)
    346             """
--> 347             super(ChemicalEnvironment.Bond,self).__init__(ORtypes, ANDtypes, None, None)
    348             self._atom = False

TypeError: super(type, obj): obj must be an instance or subtype of type

During handling of the above exception, another exception occurred:

SMIRKSParsingError                        Traceback (most recent call last)
<ipython-input-33-4af53ed20f7e> in <module>()
     24     convert(destination, name,
     25             reference_prmtop, reference_inpcrd,
---> 26             host_resname, guest_resname)

~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/smirnovert/convert.py in convert(destination, prefix, reference_prmtop, reference_inpcrd, host_resname, guest_resname, debug)
    117     molecules = [host, guest]
    118
--> 119     ff = ForceField('forcefield/smirnoff99Frosst.ffxml')
    120     system = ff.createSystem(
    121         hg_topology.topology,

~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/openforcefield/typing/engines/smirnoff/forcefield.py in __init__(self, *files)
    532         """
    533         self._forces = []
--> 534         self.loadFile(files)
    535
    536     def loadFile(self, files):

~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/openforcefield/typing/engines/smirnoff/forcefield.py in loadFile(self, files)
    579
    580         # Parse XML, get force definitions
--> 581         self.parseXMLTrees()
    582
    583     def parseXMLTrees(self):

~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/openforcefield/typing/engines/smirnoff/forcefield.py in parseXMLTrees(self)
    626             for child in root:
    627                 if child.tag in parsers:
--> 628                     parsers[child.tag](child, self)
    629
    630

~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/openforcefield/typing/engines/smirnoff/forcefield.py in parseElement(element, ff)
   1781         # Register all SMIRNOFF torsion definitions.
   1782         for torsion in element.findall('Proper'):
-> 1783             generator.registerProperTorsion(torsion, element)
   1784         for torsion in element.findall('Improper'):
   1785             generator.registerImproperTorsion(torsion, element)

~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/openforcefield/typing/engines/smirnoff/forcefield.py in registerProperTorsion(self, node, parent)
   1761     def registerProperTorsion(self, node, parent):
   1762         """Register a SMIRNOFF torsiontype definition for a proper."""
-> 1763         torsion = PeriodicTorsionGenerator.ProperTorsionType(node, parent)
   1764         self._propertorsiontypes.append(torsion)
   1765

~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/openforcefield/typing/engines/smirnoff/forcefield.py in __init__(self, node, parent)
   1680             # Check that the SMIRKS pattern matches the type it's supposed to
   1681             try:
-> 1682                 chemenv = ChemicalEnvironment(self.smirks)
   1683                 thistype = chemenv.getType()
   1684                 if thistype != 'Torsion':

~/data/applications/anaconda3/envs/smirnoff-paprika/lib/python3.6/site-packages/openforcefield/typing/chemistry/environment.py in __init__(self, smirks, label, replacements)
    468             except:
    469                 raise SMIRKSParsingError("Error SMIRKS (%s) was not parseable\
--> 470                         into a ChemicalEnvironment" % smirks)
    471
    472         # Check that the created Environment is valid

SMIRKSParsingError: (SMIRKSParsingError(...), 'Error SMIRKS ([*:1]~[*:2]~[*:3]~[*:4]) was not parseable                        into a ChemicalEnvironment')

Weirdly, I was tying my conda environment to a particular commit and hence my usage of forcefield/smirnoff99Frosst.ffxml. But if I update to commit 32ad91a0fc20e167738810392a7d9452d4dda3bc and use forcefield/smirnoff99Frosst.offxml, things seem to work. I guess there's an import dependency tree that gets traversed that I don't quite understand. I've found it useful to test my notebooks in separate conda environments, created by having conda call pip to install a specific commit, like so:
name: smirnoff-paprika
channels:
  - anaconda
  - bioconda
  - conda-forge
  - omnia
  - openeye
dependencies:
  - python=3.6
  - anaconda::numpy=1.13.3
  - anaconda::scipy=1.0.0
  - anaconda::matplotlib=2.0.2
  - anaconda::ipykernel=4.8.0
  - anaconda::networkx=1.11
  - anaconda::lxml=3.8.0
  - anaconda::pylint
  - anaconda::pip
  - bioconda::nglview=1.0
  - omnia::parmed==2.7.3
  - omnia::openmm=7.1.1
  - omnia::openmoltools=0.8.1
  - openeye::openeye-toolkits
  - yapf
  - pip
  - pip:
    - git+https://github.com/open-forcefield-group/openforcefield@32ad91a0fc20e167738810392a7d9452d4dda3bc
    - git+https://github.com/slochower/smirnoff-host-guest@ea17852ca64127340baec2731ab7427ced8d9407
    - git+https://github.com/slochower/paprika/@62869c7bb86379f9dda44feb904fb8f2c8379432

...but I guess this shows that approach can have hidden issues.

I take back what I said about everything working. I'm sorry for not providing a full pipeline, but now I'm using the latest version of this repository and the offxml file.
SMIRKSParsingError: (SMIRKSParsingError(...), 'Error SMIRKS ([*:1]-[#6X4:2]-[#6X4:3]-[*:4]) was not parseable                        into a ChemicalEnvironment')

This seems unexpected to me. I'm reading in SYBYL format (yes!) mol2 files that were not giving an error previously. I've uploaded them here for reference -- but that such a common-seeming parameter is missing makes me think it's an issue with the force field file -- right? I can help diagnose further tomorrow.
MGO-sybyl.mol2.txt
MOL-sybyl.mol2.txt

@slochower - this sounds like it's a problem with parsing of the OFFXML file rather than a problem with your molecules.  Have you been making changes to the code? It would be easiest for me to understand this message if I thought you had broken the ChemicalEnvironments class in your openforcefield installation.
Can you provide the exact OFFXML file you are using?

I actually have not been messing with ChemicalEnvironments, although the error may still be on my end. I apologize for making work without finding the most minimal example where the issue occurs. It may be a problem with having openforcefield installed in multiple conda environments (I found it helpful to pin a certain version of openforcefield while working on the conversion functions to eliminate one potentially changing variable). I was under the impression they would be effectively "firewalled" from each other, but I'm starting to doubt that.
Here is the offxml file -- it looks like the one online (but I didn't sha1sum or antyhing).
smirnoff99Frosst.offxml.txt

Hmm, sounds odd. So I think the next step would be for me to try applying the FF to your molecules in your system, but I'm missing one file for that: Can you provide whatever file it is you're getting your Topology from? (It looks like you are saying you're processing two molecules at once, so...)
FWIW, parsing of this DOES get tested automatically on Travis-CI, and it works, so I suspect some kind of issue on your end.
You're right that the conda environments should in general be sandboxed from one another.

I am increasingly sure this is not your problem at all and something else local going wacky. I have the latest version of openforcefield in my current conda environment, but I'm loading other modules I wrote that simply call import openforcefield and I think import statement was pulling in an older version of openforcfield installed in the default (I think called root) conda environment. I surmise this is an issue with jupyter notebook kernel because I was not getting the problems running the scripts directly, which is why I perceived this to be a new issue and given that the first error popped up with the wildcard SMIRKS pattern, I suspected it might have had to do with this PR.
I have attached the topology file, but the caveat is that I split the topology (to eliminate waters and ions) before loading into createSystem, so the file isn't ready to go as-is. However, I don't believe it is worth your time to continue troubleshooting because I'm not sure I have all my ducks in a row over here.
smirnoff.pruned.pdb.txt

OK. Maybe check if your ducks are in a row. It's not obvious to me what old version you might have which would have this issue; on the other hand, I'm having trouble wrapping my head around how we could have something so wrong that it would lead to this problem, while also still passing tests on Travis.
(This is part of what Travis-CI is for -- it's basically automatically verifying, everytime we change anything, that everything that should run, DOES run and give correct results in a clean install on a virtual machine. So I can tell you that not only this works for me on my machine, but it works in a clean install on a new virtual machine, which strongly suggests it's a problem on your end. Also it's an argument for -- if you're a developer -- CI testing since you don't have to worry as much that you'll break something and only find out about it when your users complain.)

Yes, I agree 100% -- we have Travis running for some of our projects too, for precisely those reasons, but I don't usually use it with exploratory notebooks when it is not obvious what to test yet. I also agree that the error was perplexing! I am afraid that I've seemed to fix the issue but not know the cause. I basically removed the relevant conda environments and reinstalled them, shutdown all the notebook kernels, and so on. Maybe it was a "namespace" issue of some sort, where both old and new modules were in memory.

@slochower
It sounds like this was an environment problem, can we close this issue, or do you still need help working it out?

Close!
FWIW, in case anyone else stumbles upon this, I believe I know the issue. I was updating my conda environment and then restarting the kernel in my jupyter notebook to re-run stuff. I could see that new versions were being parsed by checking __version__ strings but something was lurking in memory causing problems (this may be why the new version of openforcefield was looking for some old wildcard patterns). Completely killing the jupyter process and then re-launching got rid of  the errors.

We removed generics back in PR #74 and then I forgot to close this after the discussion above.


Players not seeing hosted games in lobby.

Which version you have issues with?
Using the prebuilt on Dockerhub
What is this issue is about?
Players unable to see hosted games in lobby
When does this occur?
In lobby
Are you running the Docker as user or root?
root
How do we replicate the issue?
We are using an OVH Debian 10 instance with docker pre-installed with 4GB of RAM.
Expected behavior (i.e. solution)
Players should be able to see hosted games
Other Comments
Ports appear to be open, test-netconnection returns TCPTestSucceeded:True for ports 6969-6971
when players enter lobby the server logs become flooded with the following message:
[Client Request]  \coop\server\getAllForLocation
Error: incorrect header check
Error: incorrect header check
at Zlib.zlibOnError [as onerror] (node:zlib:189:17)
at processChunkSync (node:zlib:457:12)
at zlibBufferSync (node:zlib:178:12)
at Object.inflateSync (node:zlib:809:14)
at IncomingMessage. (/snapshot/src/servers/http/AkiHttpListener.ts:69:60)
at IncomingMessage.emit (node:events:518:28)
at endReadableNT (node:internal/streams/readable:1696:12)
at processTicksAndRejections (node:internal/process/task_queues:82:21)
Error: incorrect header check
Error: incorrect header check
at Zlib.zlibOnError [as onerror] (node:zlib:189:17)
at processChunkSync (node:zlib:457:12)
at zlibBufferSync (node:zlib:178:12)
at Object.inflateSync (node:zlib:809:14)
at IncomingMessage. (/snapshot/src/servers/http/AkiHttpListener.ts:69:60)
at IncomingMessage.emit (node:events:518:28)
at endReadableNT (node:internal/streams/readable:1696:12)
at processTicksAndRejections (node:internal/process/task_queues:82:21)

This is interesting.
I just came here to post about a similar issue, but we have not had issues with players being unable to see each other.
When we get this header error, it makes it so damage recording is turned off. Players can shoot, and be shot. Animations will play, ammo gets expended, but no health is deducted.
The code segment you listed under 'other comments' is exactly the same error report I get.
We are using the pre-built docker image, and I am running on Ubuntu 20.04 running as a user.

can you verify your port(s) are open? (6969, 6970, 6971) check here: https://portchecker.io/ or with nmap: nmap -p6969-6971 ip.addr.ess

Server is 1.6.5, but the client is 1.10. SIT.Manager does not appear to have an upgrade showing for me to 1.11

change image: stayintarkov/stayintarkov to image: stayintarkov/stayintarkov:<IP_ADDRESS>
which is the last 1.10 build.
enable the dev build setting flag in SIT Manager if you want to use 1.11 pre-release which is stayintarkov/stayintarkov:latest

Can confirm that using the <IP_ADDRESS> tag solved this issue for me! It was a version mismatch. The latest points to the pre-release 1.11


Fix text backend for determinants + check consistency of determinants

The text backend used 10 characters for storing 64-bit integers in text format, but this was too small. I changed it to 20 characters.
When you pass a list of determinants to the write function, it now checks that the number of alpha and beta occupied orbitals in the determinant is consistent with nup and ndn, and it also checks that the MO indices are in the correct range [0:mo_num[
I had to update the Fortran test

Thanks @scemama !
Fortran tests fail in the CI: Segmentation fault. Some illegal memory access?

And one general concern: the new determinant check requires that number of up and down electrons to be present in the file. If they are missing - the new error code can be added to TREXIO and returned from trexio_write_determinant_list in this case to tell the user that they forgot to write up/down electrons. This should also be stated explicitly in the documentation and we need to inform @NastaMauger to modify the PySCF CI dumper accordingly.

@q-posev I am confused regarding your comment.
I naively used the bitfield format to register the determinants in this PR. I was thinking it might a better suit since the determinant list can be huge

@NastaMauger what you did was right. In this PR @scemama introduced an additional correctness check for the CI determinants (see this line). This correctness check requires that the number of up and down electrons are written in the TREXIO file before you start writing the determinants (otherwise this check cannot be performed). What you implemented in pyscf-forge is correct with the current version of TREXIO but will break once this PR is merged. But if in your PySCF CASCI producer you also perform  trexio_write_electron_up_num and trexio_write_electron_dn_num - then the code will remain compatible with both pre-PR and post-PR version of TREXIO.

And one general concern: the new determinant check requires that number of up and down electrons to be present in the file. If they are missing - the new error code can be added to TREXIO and returned from trexio_write_determinant_list in this case to tell the user that they forgot to write up/down electrons. This should also be stated explicitly in the documentation and we need to inform @NastaMauger to modify the PySCF CI dumper accordingly.

Now the function returns TREXIO_INVALID_ELECTRON_NUM if the number of electrons is wrong in a determinant, and TREXIO_INVALID_MO_INDEX if the MO index is out of bounds.
I added it to the documentation.

Thanks @scemama !
Fortran tests fail in the CI: Segmentation fault. Some illegal memory access?

I don't understand what is wrong. I can't reproduce the problem on my computer, and valgrind doesn't detect anything.
In the artifact file, the segfault happens at line 24 of test_f.f90, which seems to be just a print statement. I really don't see anything wrong.. :-(

@scemama it must be some non-standard Fortran stuff or the way you iterate when printing. Can you please

Install the same gfortran as in the CI (e.g. gfortran-12 which we use macos)
make a clean clone of the repo, checkout to your branch, compile and test via make check? Maybe you have smth locally which allows the Fortran test to pass

@scemama I confirm that following approach 2 (clean repo run) and gfortran-11 I can reproduce this error in Fortran test. It fails right after the call to trexio_cp function in the Fortran test, so maybe there is a memory leak somewhere on this branch when this function is executed?

@scemama I found a bug in the C code (unclosed comment) but the Fortran test bug is still there. It must be a memory leak somewhere.

OK, now this modification broke the python test because the determinant data in the test is dummy, and now we are checking that the numbers of up and dn electrons in the determinants are OK. This means that the modification works well! But we need to change the python tests :-(

./configure FCFLAGS="-O2" => ERROR!

We have always used -O2 as our compiler flag by default and the code worked fine, there were no memory leaks! I am pretty sure that there is something about changes in this branch that break the Fortran test and not the compiler.

OK, now this modification broke the python test because the determinant data in the test is dummy, and now we are checking that the numbers of up and dn electrons in the determinants are OK. This means that the modification works well! But we need to change the python tests :-(

Good news then! Can you translate the commands from io_determinant.c into the Python test?

We have always used -O2 as our compiler flag by default and the code worked fine, there were no memory leaks! I am pretty sure that there is something about changes in this branch that break the Fortran test and not the compiler.

You are right. Address-sanitizer helped me find it. I fixed it, and it now works with -O2. Sorry.

Good news then! Can you translate the commands from io_determinant.c into the Python test?

I won't have time in the two coming weeks.

Thanks a lot @scemama !
I will see if I can find time to translate the tests. I have to make release of trexio-tools ASAP following recent additions and changes.
Looks like we are converging to the complete PySCF-TREXIO integration! 😊


Improve SA1516 performance and reduce allocations

With this change the analyzer does no longer create two unnecessary immutable lists.

:+1: looks good to me


Update repository-settings.md

put a note that cross-repo policy branch patterns are case-sensitive

@vijayma
Can you review the proposed changes?
Important: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


The Model Configuration in Machine Translation task

During model configuration, after fired the command
:~/neuralmonkey$ bin/neuralmonkey-train exp-nm-mt/translation.ini
Exception an error! "Unexpected fields: runners_batch_size"
It raises the error in the following way:
2019-06-23 13:43:17: Loading INI file: 'exp-nm-mt/translation.ini'
2019-06-23 13:43:17: INI file is parsed.
2019-06-23 13:43:17: Building model based on the config.
2019-06-23 13:43:17: Traceback (most recent call last):
File "/home/rupjyoti/neuralmonkey/bin/neuralmonkey/train.py", line 66, in _main
exp.train()
File "/home/rupjyoti/neuralmonkey/bin/neuralmonkey/experiment.py", line 243, in train
self.build_model()
File "/home/rupjyoti/neuralmonkey/bin/neuralmonkey/experiment.py", line 207, in build_model
self.config.build_model(warn_unused=self.train_mode)
File "/home/rupjyoti/neuralmonkey/bin/neuralmonkey/config/configuration.py", line 89, in build_model
self._check_loaded_conf()
File "/home/rupjyoti/neuralmonkey/bin/neuralmonkey/config/configuration.py", line 116, in _check_loaded_conf
.format(", ".join(unexpected)))
Exception: Unexpected fields: runners_batch_size
Any help for correcting is much appreciated.
Thank you so much

de-duping with #823


Is it possible to add zoom max value

Thank you for the solution, work really great for me. I have only one issue is the zoom max, we can zoom infinity. Is it possible to add zoom max value that block user from zooming.

Hi @TrabelsiAchraf, I tried this and works for me, try it and let us know:
@available(iOS 17.0, *)
    private var magnificationGesture: some Gesture {
        MagnifyGesture(minimumScaleDelta: 0)
            .onChanged { value in
                let maxZoomScale: CGFloat = 6.0 // Sets the maximum allowed zoom
                let newScale = min(value.magnification * lastTransform.scaleX, maxZoomScale)
                let newTransform = CGAffineTransform.anchoredScale(
                    scale: newScale / lastTransform.scaleX, // Adjust the scale relative to the last transform
                    anchor: value.startAnchor.scaledBy(contentSize)
                )
                withAnimation(.interactiveSpring) {
                    transform = lastTransform.concatenating(newTransform)
                    if !transform.isIdentity {
                        originalControlsVisible = controlsVisible?.wrappedValue ?? true
                        controlsVisible?.wrappedValue = false
                    }
                }
            }
            .onEnded { _ in
                onEndGesture()
                if transform.isIdentity {
                    withAnimation(.interactiveSpring) {
                        controlsVisible?.wrappedValue = originalControlsVisible
                    }
                }
            }
    }


init: intensified all existing tests

Before proceeding to port all other primitive string functions, we have to make sure the existing ones are behaving as expected. Hence, let's intensify all the test cases.
This patch intensify all existing test cases in init/ directory.

merge first since production is unaffected.


[azservicebus] Receiving duplicated messages when maxMessages is greater than 1

Bug Report
ServiceBus receiver freezes for a minute (the message lock period) and then receives two duplicated messages (one of them with expired lock).
2022-02-14T10:45:46Z Receiving messages...
2022-02-14T10:46:50Z Got 2 messages...
2022-02-14T10:46:50Z Processing message: 956d72d8889e42aaa6c3e3028ddbb3ba
2022-02-14T10:46:50Z Processing message: 956d72d8889e42aaa6c3e3028ddbb3ba
2022-02-14T10:46:51Z Error completing event: *Error{Condition: com.microsoft:message-lock-lost, Description: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue. For more information please see https://aka.ms/ServiceBusExceptions . Reference:c1bd8e02-2e29-43f7-b1ca-0532bb732279, TrackingId:71d9126f001701570027cc55620702b4_G10_B11, SystemTracker:securityside-development-v2:Topic:dev-contracts|facade-services-cache-updater, Timestamp:2022-02-14T10:46:51, Info: map[]}
2022-02-14T10:46:51Z Message ID: 956d72d8889e42aaa6c3e3028ddbb3ba
2022-02-14T10:46:51Z Messaged enqueued time: 2022-02-14 10:45:50.944 +0000 UTC
2022-02-14T10:46:51Z Locked until: 2022-02-14 10:46:50.959 +0000 UTC
2022-02-14T10:46:51Z Error completing event: *Error{Condition: com.microsoft:message-lock-lost, Description: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue. For more information please see https://aka.ms/ServiceBusExceptions . Reference:c1bd8e02-2e29-43f7-b1ca-0532bb732279, TrackingId:71d9126f001701570027cc55620702b4_G10_B11, SystemTracker:securityside-development-v2:Topic:dev-contracts|facade-services-cache-updater, Timestamp:2022-02-14T10:46:51, Info: map[]}
2022-02-14T10:46:51Z Message ID: 956d72d8889e42aaa6c3e3028ddbb3ba
2022-02-14T10:46:51Z Messaged enqueued time: 2022-02-14 10:45:50.944 +0000 UTC
2022-02-14T10:46:51Z Locked until: 2022-02-14 10:47:50.885 +0000 UTC

As you can see from the above logs, the consumer froze for a minute (2022-02-14T10:45:46Z until 2022-02-14T10:46:50Z), then received 2 messages with the same MessageID, one of them with the lock already expired (2022-02-14 10:46:50.959 +0000 UTC).
func (c *ClientConsumer) receive(ctx context.Context, receiver receiver, handler consumer.EventHandler, topic string) error {
	for {
		fmt.Println("Receiving messages...")
		messages, err := receiver.ReceiveMessages(ctx, 1, nil)
		if err != nil {
			if err.Error() == "amqp: link closed" && c.closed {
				return nil
			}
			return errors.NewErrorMessage(err.Error(), false)
		}

		if len(messages) == 0 {
			continue
		}

		fmt.Printf("Got %d messages...\n", len(messages))

		wg := sync.WaitGroup{}
		wg.Add(len(messages))
		for index := range messages {
			go func(message *azservicebus.ReceivedMessage) {
				defer wg.Done()

				fmt.Printf("Processing message: %s\n", message.MessageID)
				c.handle(context.Background(), message, topic, receiver, handler)
			}(messages[index])
		}
		wg.Wait()
	}
}

// Handle handles the event.
func (c ClientConsumer) handle(ctx context.Context, e *azservicebus.ReceivedMessage, topic string, receiver receiver, handler consumer.EventHandler) {
	if e.LockedUntil != nil && e.LockedUntil.Before(time.Now()) {
		fmt.Printf("Lock already lost\nMessage ID: %s\nMessaged enqueued time: %s\nLocked until: %s\n", e.MessageID, e.EnqueuedTime, e.LockedUntil)
		return
	}

	err := handler(ctx, e)
	if err == nil {
		if err := receiver.CompleteMessage(ctx, e); err != nil {
			fmt.Printf("Error completing event: %s\nMessage ID: %s\nMessaged enqueued time: %s\nLocked until: %s\n", err.Error(), e.MessageID, e.EnqueuedTime, e.LockedUntil)
		}
		return
	}

	if err := receiver.AbandonMessage(ctx, e, nil); err != nil {
		fmt.Printf("Error abandoning event: %s\n", err.Error())
	}
}

Tested with: v0.3.3, v0.3.4 and v0.3.5

Thank you for submitting this @SpaWn2KiLl.
I think this is a result of a few things - overzealous message returning and connection recovery. I'll take a look and see if I can make that more useful - returning two of the same message in a batch makes me think it needs a bit more work.


Javascript Errors on Menu

What steps will reproduce the problem?
I must update my old Yii2 Installation with composer to 2.0.14, before i must install the fxp/composer-asset-plugin:^1.4.1 globally. After successfully composer update, my project displays a Javascript Error in the console after click on a menu item (with submenu):
jquery.js:1580 Uncaught Error: Syntax error, unrecognized expression: #     at Function.Sizzle.error (jquery.js:1580)     at Sizzle.tokenize (jquery.js:2232)     at Sizzle.select (jquery.js:2659)     at Function.Sizzle [as find] (jquery.js:884)     at jQuery.fn.init.find (jquery.js:2922)     at new jQuery.fn.init (jquery.js:3032)     at jQuery (jquery.js:98)     at getParent (bootstrap.js:790)     at HTMLAnchorElement.<anonymous> (bootstrap.js:774)     at Function.each (jquery.js:362) Sizzle.error @ jquery.js:1580 Sizzle.tokenize @ jquery.js:2232 Sizzle.select @ jquery.js:2659 Sizzle @ jquery.js:884 find @ jquery.js:2922 jQuery.fn.init @ jquery.js:3032 jQuery @ jquery.js:98 getParent @ bootstrap.js:790 (anonymous) @ bootstrap.js:774 each @ jquery.js:362 each @ jquery.js:157 clearMenus @ bootstrap.js:773 dispatch @ jquery.js:5206 elemData.handle @ jquery.js:5014
What is the expected result?
The menu works fine
What do you get instead?
Nothing happens and the Javascript Error displayed in the console.
Additional info

Q
A

Yii version
2.0.14

PHP version
7.0.30

Operating system
Mac OS 10.13.4 and also on the Live Server with Ubuntu 16.04.2

When does the issue occur?
When you click on a menu item

What do you see?
Nothing! If the Browser console are open, you see the javascript error above.

What was the expected result?
The Dropdown opens

Can you supply us with a stacktrace? (optional)
Only this, what i have posted above.

Do you have exact code to reproduce it? Maybe a PHPUnit tests that fails? (optional)
No, what i have describe: I have update with composer (also global require fxp-asset before) and not more.

I don't know, what i shall more describe. I didn't have change anything else. The Yii version before was 2.0.6 and now i have updated to 2.0.14, nothing more.

It's not enough info to reproduce the issue. We don't have your project and have no idea what menu you use and how.

Hm, ok..
The project was developed ca. on January '18. The Menu:
`<?php
NavBar::begin([
'brandLabel' => Html::img('@web/img/logo.png',['title'=>Yii::t('app','Olicci Testpages'),'alt'=>Yii::t('app','Olicci Testpages')]),
'brandUrl' => Yii::$app->homeUrl,
'options' => [
'class' => 'navbar-default navbar-fixed-top',
],
]);
$items[]=['label' => Yii::t('app','Start'), 'url' => ['/site/index']];
// Kategorien Auslesen und als Menü anzeigen
$cats = Categories::getOrdenedCategories();
$subitems = [];
if(!empty($cats)){
foreach($cats as $catid=>$cat) {
if(!empty($cat['subcats'])&&is_array($cat['subcats'])){
$sitems=[];
foreach($cat['subcats'] as $scatid => $scat) {
$sitems[]=['label'=>$scat['label'],'url' => ['/category/index','id'=>$scatid]];
}
$items[]=[
'label' => $cat['label'],
'url' => ['/category/index/','id'=>$catid],
'items' => $sitems
];
} else {
$items[]=[
'label' => $cat['label'],
'url' => ['/category/index/','id'=>$catid]
];
}
}
}
if(Yii::$app->user->isGuest) {
    $items[]=['label' => Yii::t('app','Login'), 'url' => ['/site/login']];
} else {
    //if (Yii::$app->user->can('editTemplates')||Yii::$app->user->isSuperadmin) {
    if (User::hasPermission('editTemplates')||Yii::$app->user->isSuperadmin) {
        $subitems=[
	        ['label'=>Yii::t('app','Kategorien'),'url'=>['/categories/index']],
	        ['label'=>Yii::t('app','Templates'),'url'=>['/templates/index']],
            ['label'=>Yii::t('app','Mandanten'),'url'=>['/clients/index']],
	        '

',
        ];
    }
    if (User::hasPermission('viewUsers')||Yii::$app->user->isSuperadmin) {
        $subitems[]=['label'=>Yii::t('app','Users'),'url'=>'#','items'=>\wpler\modules\UserManagement\UserManagementModule::menuItems()];
    }
    $items[]=['label' => Yii::t('app','System'),'url'=>'#','items' => $subitems];
    $items[]=('
'.Html::beginForm(['/user-management/auth/logout'],'post').Html::submitButton(Yii::t('app','Logout ({user})',['user'=>Yii::$app->user->identity->username]),['class'=>'btn btn-link logout']).Html::endForm().'
');

}

echo \app\components\widgets\ClientSwitchWidget::widget([]);

echo Nav::widget([
    'options' => ['class' => 'navbar-nav navbar-right'],
    'encodeLabels' => false,
    'items' => $items,
]);
NavBar::end();
?>`

and if i am click on a menu item with submenu, the above javascript error occured into the browser console (see screenshot).

The Steps i have do:
Before on composer.json was the Yii version 2.0.6 ( "yiisoft/yii2": "2.0.6",). I have change it now to "yiisoft/yii2": "~2.0.14",
After that i do on terminal composer update, but it was failed because some dependencies are failed. Anywhere i have read, that i must update the fxp-asset-plugin, so i have do:
composer global require "fxp/composer-asset-plugin:^1.4.1" --no-plugins
and after that i could do composer update without any problems. But after the update this javascript errors occured and the submenu not open.


Zbd breaks damus contacts, relays

client initiating action
Zbd
client receiving or observing action (or lack thereof)
Damus
timestamp
https://njump.me/nevent1qqszsdxqueud7dg8f28fwn5e2g4vfx8704l2mfm7ykvqpqlmzr923qcprpmhxue69uhhyetvv9ujumn0wd68ytnrdakjuct4qyg8wumn8ghj7mn0wd68ytnhd9hx2qg6waehxw309aex2mrp0yhx7unpdenk2urfd3kzuer9wcq3vamnwvaz7tmpw3kxzuewdehhxarj9ekxzmnyr42750
what happens

Esiste un modo per tenere sincronizzati i contatti e i relay tra #ZBD e #damus? Perché tutte le volte che entro nella sezione social di #ZBD poi in #damus succede un casino, mi spariscono parecchi contatti e alcuni relay.
Oppure #ZBD e #damus non vanno d’accordo

Is there a way to keep contacts and relays synchronized between #ZBD and #damus? Because every time I enter the social section of #ZBD then a mess happens in #damus, several contacts and some relays disappear.
Or #ZBD and #damus don't go together

suggestion
@fiatjaf pls fix your client

@zantoshi

ZBD Social profiles are really not meant to be used from the external world.

Imagine being the first social gaming company to adopt an open standard identity system, and make sure it works. I think this will bring in dividends for y'all.
Irregardless of what I think will you tell your customer, or do you want me to?

why not build something proprietary if you don't want leakage to a wider open protocol?

I want leakage and interaction of ZBD users with the rest of Nostr, it's just that the custodial backend, for various reasons, is not made for users that control their keys from the outside, they're meant to be custodial users forever (or they can export their key, but then they stop existing internally).
Unfortunately there was an "import key" feature in the past that caused the situation here described. That feature has been disabled now so it shouldn't happen again.
My recommendation is that the user uses Damus only for browsing and posting, but do contact list changes only inside ZBD -- or go on ZBD and click to export key so they exit fully and their keys get deleted from ZBD.
Or use https://follows.nostr.com/ to fix their contact list every once in a while.


Add Cloudify plugin

Add Cloudify plugin
Backstage plugin that connects with the Cloudify Manager instance and pulls all available blueprints.
Blueprints are listed in a table and accesible via a link to the Cloudify Manager.

Thanks for this fine contribution @jakubcierlik 🙌 Please let me know if you’d like some help to contribute further.


Home app: 'not responding', OK in Hue app

Issue
The Home app is reporting (an Innr) lamp connected to Hue bridge as 'not responding'. However, it is not shown as 'unreachable' in the Hue app, so I am able to control it.
Log Messages

 I can't see anything obvious in homebridge logs.

Debug Files

homebridge-hue.json.gz

homebridge-hue doesn't set nor clear the Not responding state.  The Home app sets this when it doesn't receive a response from the HomeKit accessory (i.c. homebridge) when querying the state.  Afaik the only way to clear this state is to switch to another room (or switch from the Home tab to the Rooms tab) and back, forcing Home to refresh the state.
Since only one light shows Not responding, I think it's just a glitch.  If all your homebridge accessories show Not responding, there's usually an issue with your network not handling Bonjour well.

Thanks Erik. I think I’ve narrowed the problem to single lights (although different ones) in Home app-defined grouped accessories. I intend to plug the Homebridge server directly to my router to obviate network throughout issues (though I appreciate that’s not necessarily going to help flaky Bonjour).
Given that I’ve no need to control the grouped lights individually, and that I have never seen a case where none of a group is unresponsive, I think I might be better doing the grouping at the Hue side.

I think I might be better doing the grouping at the Hue side.

You can expose groups on the Hue bridge as HomeKit lights by setting "groups": true in config.json.  Alternatively, for finer-grained control, whitelist the group(s) you want to expose, and blacklist the member lights, see https://github.com/ebaauw/homebridge-hue#configuration.
Generally, it's a good idea to minimise the traffic between HomeKit and the Hue birdge (or deCONZ gateway).  I control my lights using Hue bridge (deCONZ gateway) rules recalling scenes and/or issuing group commands.  I use CLIPGenericFlag sensors (exposed to HomeKit as Switch) to trigger the rules from HomeKit, both manually and through HomeKit automations.

I'm getting there - slowly! May I just check something please: with the blacklist/whitelist setup, I notice that you say that homebridge-hue ignores the classid, so do I just put all my blacklisted lights into a single blacklist definition and all my groups into a single whitelist?
I initially stumbled because I think there's a new requirement to include "recycle": true|false in definition of resourcelinks.
Finally - to your last point about using the CLIPGenericFlag (exposed to HomeKit as Switch) - do you mean that you do not generally expose the group at all, but rely on a Hue rule acting on HomeKit turning the switch on|off?

Thank you Erik. I appreciate your help and patience!


When setting the load_tensors parameter to True, it is expected that the self.data_root=PosixPath('/root/autodl-tmp/video-dataset-disney/videos') contains two folders named video_latents and prompt_embeds. However, these folders were not found. Please make sure to have prepared your data correctly using prepare_data.py.

System Info / 系統信息
Cuda 12.1, Python 3.12
Information / 问题信息

[X] The official example scripts / 官方的示例脚本
[ ] My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程
I use official disney dataset and run train_text_to_video_lora.sh, but get the error like " When setting the load_tensors parameter to True, it is expected that the self.data_root=PosixPath('/root/autodl-tmp/video-dataset-disney/videos') contains two folders named video_latents and prompt_embeds. However, these folders were not found. Please make sure to have prepared your data correctly using prepare_data.py. "
How can I solve this error? Thanks for help!
Expected behavior / 期待表现
If can help me solve my error, really thanks.

You need to run "prepare_dataset.sh" before training, it will pre-process the prompts and videos into "video_latents" and "prompt_embeds".

Thanks for your help! But when I finish running prepare_dataset.sh before training, I find the videos change in the output directory. The original video content is Disney black and white but after doing the pre-process, the video content is like the following file.
.
I am not sure if this is right and don't know why the videos changed. Even though they do the embedding work, I think the contents of the videos should be same.
Could you help me to solve this? really thanks !

The content of the serialized videos should not not affected qualitatively. Could this is be some video codec problem? I am able to open the serialized videos on my VSCode.

Closing due to inactivity.


cadviser: add storageDriverPasswordFile option

This gives users the option of storing the storageDriverPassword outside the world-readable Nix store.
This is part of https://github.com/NixOS/nixpkgs/issues/24288.
@offlinehacker I tested if the module evaluates without problems.  I haven't tested the actual service yet. Would you be able to do that?
One concern I have is if the secret in -storage_driver_password secret will show up in top/htop/ps/ etc. since it's passed as a command line argument. Will cadvisor do The Right Thing and remove the secret from its command line?
Ideally we request the cadviser developers to add a -storage_driver_password_file option.

It would be great if this can be merged so that we get some progress on  #24288.
I also requested a -storage_driver_password_file option in: https://github.com/google/cadvisor/issues/1633 so that we don't have to pass secrets via the insecure command line.


RPi-#37. Enable communication between controller and CLI

What is this PR for?

What type of PR is it?
[Feature]
Todos
None
What is the Github issue?
https://github.com/NTUEELightDance/LightDance-RPi/issues/37
How should this be tested?
python3 /cli/cli.py
./test_controller/bin/controller testName
Screenshots (if appropriate)
None
Questions:

Do the license files need updating? No
Are there breaking changes for older versions? Yes
Does this need new documentation? No

@Kenchu123 Can you help me review my code?


Add a function for the overlay to remember the position and size after closing.

Is there a way to make the overlay remember the position and the size I resized/set it to? currently on 0.0.10 whenever I close the program completely and reopen it, it just defaults back to the default size and position on the screen, making the user resize and reposition it every time.

This is definitely a bug and not intended. The position/size should be saved and restored when you relaunch the application. I'll investigate to see if it's a quick fix for the next release.


Add node-token-validation cache manager

This PR adds caching to node-token-validation, with unit tests.

Codecov Report

Merging #4906 (010f3ed) into dev (8449715) will decrease coverage by 0.00%.
The diff coverage is 83.87%.

Flag
Coverage Δ

*Carryforward flag

msal-angular
96.42% <ø> (ø)

Carriedforward from 8449715

msal-browser
86.87% <ø> (ø)

Carriedforward from 8449715

msal-common
84.41% <ø> (ø)

Carriedforward from 8449715

msal-core
82.67% <ø> (ø)

Carriedforward from 8449715

msal-node
81.94% <ø> (ø)

Carriedforward from 8449715

msal-node-extensions
76.03% <ø> (ø)

Carriedforward from 8449715

msal-react
94.50% <ø> (ø)

Carriedforward from 8449715

node-token-validation
86.11% <83.87%> (-0.81%)
:arrow_down:

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files
Coverage Δ

...ode-token-validation/src/cache/NodeCacheManager.ts
83.87% <83.87%> (ø)


Rip collects wheels unrelated to the platform it is running on

This is possibly a misunderstanding on my side but on Linux  on Python 3.8 when I run:
$ python3.8 -m venv .venv
$ source .venv/bin/activate
$ cargo r -- PyWavelets

I get a lot of output of it collecting wheels related to MacOS and Windows and Python versions that I was not on:
    Finished dev [unoptimized + debuginfo] target(s) in 0.07s
     Running `target/debug/rip PyWavelets`
2023-10-02T23:41:07.956549Z  INFO rip: cache directory: /home/damian/.cache/rattler/pypi
2023-10-02T23:41:07.996443Z  INFO rip: extracted the following environment markers from the system python interpreter:
Pep508EnvMakers {
    os_name: "posix",
    sys_platform: "linux",
    platform_machine: "x86_64",
    platform_python_implementation: "CPython",
    platform_release: "<IP_ADDRESS>-microsoft-standard-WSL2",
    platform_system: "Linux",
    platform_version: "#1 SMP Fri Jan 27 02:56:13 UTC 2023",
    python_version: "3.8",
    python_full_version: "3.8.18",
    implementation_name: "cpython",
    implementation_version: "3.8.18",
}
2023-10-02T23:41:07.998434Z  INFO rattler_installs_packages::resolve: collecting pywavelets
2023-10-02T23:41:07.998542Z  INFO rattler_installs_packages::http: executing request url=https://pypi.org/simple/pywavelets/ cache_mode=Default
2023-10-02T23:41:08.132606Z  WARN rattler_installs_packages::resolve: Not considering pywavelets 0.2.2, 0.2.0, 0.1.6, 0.1.4, 0.1.2 because there are no wheel artifacts available
2023-10-02T23:41:08.132798Z  INFO rattler_installs_packages::resolve: obtaining dependency information from pywavelets=1.4.1
2023-10-02T23:41:08.133125Z  INFO rattler_installs_packages::resolve: collecting numpy
2023-10-02T23:41:08.133187Z  INFO rattler_installs_packages::http: executing request url=https://pypi.org/simple/numpy/ cache_mode=Default
2023-10-02T23:41:08.310962Z  WARN rattler_installs_packages::resolve: Not considering numpy 1.26.0rc1, 1.26.0b1, 1.25.0rc1, 1.24.0rc2, 1.24.0rc1, 1.23.0rc3, 1.23.0rc2, 1.23.0rc1, 1.10.0.post2, 1.5.0, 1.4.1, 1.3.0 because there are no wheel artifacts available
2023-10-02T23:41:08.311069Z  WARN rattler_installs_packages::resolve: Not considering numpy 1.26.0, 1.25.2, 1.25.1, 1.25.0 because none of the artifacts are compatible with Python 3.8.18
2023-10-02T23:41:08.313444Z  INFO rattler_installs_packages::resolve: obtaining dependency information from numpy=1.24.4
2023-10-02T23:41:08.313789Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/6b/80/6cdfb3e275d95155a34659163b83c09e3a3ff9f1456880bec6cc63d71083/numpy-1.24.4-cp310-cp310-macosx_10_9_x86_64.whl#sha256=c0bfb52d2169d58c1cdb8cc1f16989101639b34c7d3ce60ed70b19c63eba0b64 cache_mode=OnlyIfCached
2023-10-02T23:41:08.314588Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/64/5f/3f01d753e2175cfade1013eea08db99ba1ee4bdb147ebcf3623b75d12aa7/numpy-1.24.4-cp310-cp310-macosx_11_0_arm64.whl#sha256=ed094d4f0c177b1b8e7aa9cba7d6ceed51c0e569a5318ac0ca9a090680a6a1b1 cache_mode=OnlyIfCached
2023-10-02T23:41:08.314751Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/5a/b3/2f9c21d799fa07053ffa151faccdceeb69beec5a010576b8991f614021f7/numpy-1.24.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl#sha256=79fc682a374c4a8ed08b331bef9c5f582585d1048fa6d80bc6c35bc384eee9b4 cache_mode=OnlyIfCached
2023-10-02T23:41:08.315047Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/10/be/ae5bf4737cb79ba437879915791f6f26d92583c738d7d960ad94e5c36adf/numpy-1.24.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=7ffe43c74893dbf38c2b0a1f5428760a1a9c98285553c89e12d70a96a7f3a4d6 cache_mode=OnlyIfCached
2023-10-02T23:41:08.315178Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/c0/64/908c1087be6285f40e4b3e79454552a701664a079321cff519d8c7051d06/numpy-1.24.4-cp310-cp310-win32.whl#sha256=4c21decb6ea94057331e111a5bed9a79d335658c27ce2adb580fb4d54f2ad9bc cache_mode=OnlyIfCached
2023-10-02T23:41:08.315304Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/22/55/3d5a7c1142e0d9329ad27cece17933b0e2ab4e54ddc5c1861fbfeb3f7693/numpy-1.24.4-cp310-cp310-win_amd64.whl#sha256=b4bea75e47d9586d31e892a7401f76e909712a0fd510f58f5337bea9572c571e cache_mode=OnlyIfCached
2023-10-02T23:41:08.315636Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/a9/cc/5ed2280a27e5dab12994c884f1f4d8c3bd4d885d02ae9e52a9d213a6a5e2/numpy-1.24.4-cp311-cp311-macosx_10_9_x86_64.whl#sha256=f136bab9c2cfd8da131132c2cf6cc27331dd6fae65f95f69dcd4ae3c3639c810 cache_mode=OnlyIfCached
2023-10-02T23:41:08.315787Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/c0/bc/77635c657a3668cf652806210b8662e1aff84b818a55ba88257abf6637a8/numpy-1.24.4-cp311-cp311-macosx_11_0_arm64.whl#sha256=e2926dac25b313635e4d6cf4dc4e51c8c0ebfed60b801c799ffc4c32bf3d1254 cache_mode=OnlyIfCached
2023-10-02T23:41:08.315932Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/a7/4c/96cdaa34f54c05e97c1c50f39f98d608f96f0677a6589e64e53104e22904/numpy-1.24.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl#sha256=222e40d0e2548690405b0b3c7b21d1169117391c2e82c378467ef9ab4c8f0da7 cache_mode=OnlyIfCached
2023-10-02T23:41:08.316210Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/22/97/dfb1a31bb46686f09e68ea6ac5c63fdee0d22d7b23b8f3f7ea07712869ef/numpy-1.24.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=7215847ce88a85ce39baf9e89070cb860c98fdddacbaa6c0da3ffb31b3350bd5 cache_mode=OnlyIfCached
2023-10-02T23:41:08.316511Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/35/e2/76a11e54139654a324d107da1d98f99e7aa2a7ef97cfd7c631fba7dbde71/numpy-1.24.4-cp311-cp311-win32.whl#sha256=4979217d7de511a8d57f4b4b5b2b965f707768440c17cb70fbf254c4b225238d cache_mode=OnlyIfCached
2023-10-02T23:41:08.316642Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/d8/ec/ebef2f7d7c28503f958f0f8b992e7ce606fb74f9e891199329d5f5f87404/numpy-1.24.4-cp311-cp311-win_amd64.whl#sha256=b7b1fc9864d7d39e28f41d089bfd6353cb5f27ecd9905348c24187a768c79694 cache_mode=OnlyIfCached
2023-10-02T23:41:08.316786Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/11/10/943cfb579f1a02909ff96464c69893b1d25be3731b5d3652c2e0cf1281ea/numpy-1.24.4-cp38-cp38-macosx_10_9_x86_64.whl#sha256=1452241c290f3e2a312c137a9999cdbf63f78864d63c79039bda65ee86943f61 cache_mode=OnlyIfCached
2023-10-02T23:41:08.316953Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/a7/ae/f53b7b265fdc701e663fbb322a8e9d4b14d9cb7b2385f45ddfabfc4327e4/numpy-1.24.4-cp38-cp38-macosx_11_0_arm64.whl#sha256=04640dab83f7c6c85abf9cd729c5b65f1ebd0ccf9de90b270cd61935eef0197f cache_mode=OnlyIfCached
2023-10-02T23:41:08.317098Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/25/6f/2586a50ad72e8dbb1d8381f837008a0321a3516dfd7cb57fc8cf7e4bb06b/numpy-1.24.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl#sha256=a5425b114831d1e77e4b5d812b69d11d962e104095a5b9c3b641a218abcc050e cache_mode=OnlyIfCached
2023-10-02T23:41:08.317238Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/98/5d/5738903efe0ecb73e51eb44feafba32bdba2081263d40c5043568ff60faf/numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=dd80e219fd4c71fc3699fc1dadac5dcf4fd882bfc6f7ec53d30fa197b8ee22dc cache_mode=OnlyIfCached
2023-10-02T23:41:08.317383Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/d1/57/8d328f0b91c733aa9aa7ee540dbc49b58796c862b4fbcb1146c701e888da/numpy-1.24.4-cp38-cp38-win32.whl#sha256=4602244f345453db537be5314d3983dbf5834a9701b7723ec28923e2889e0bb2 cache_mode=OnlyIfCached
2023-10-02T23:41:08.317498Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/69/65/0d47953afa0ad569d12de5f65d964321c208492064c38fe3b0b9744f8d44/numpy-1.24.4-cp38-cp38-win_amd64.whl#sha256=692f2e0f55794943c5bfff12b3f56f99af76f902fc47487bdfe97856de51a706 cache_mode=OnlyIfCached
2023-10-02T23:41:08.317650Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/9a/cd/d5b0402b801c8a8b56b04c1e85c6165efab298d2f0ab741c2406516ede3a/numpy-1.24.4-cp39-cp39-macosx_10_9_x86_64.whl#sha256=2541312fbf09977f3b3ad449c4e5f4bb55d0dbf79226d7724211acc905049400 cache_mode=OnlyIfCached
2023-10-02T23:41:08.317919Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/14/27/638aaa446f39113a3ed38b37a66243e21b38110d021bfcb940c383e120f2/numpy-1.24.4-cp39-cp39-macosx_11_0_arm64.whl#sha256=9667575fb6d13c95f1b36aca12c5ee3356bf001b714fc354eb5465ce1609e62f cache_mode=OnlyIfCached
2023-10-02T23:41:08.318066Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/8f/27/91894916e50627476cff1a4e4363ab6179d01077d71b9afed41d9e1f18bf/numpy-1.24.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl#sha256=f3a86ed21e4f87050382c7bc96571755193c4c1392490744ac73d660e8f564a9 cache_mode=OnlyIfCached
2023-10-02T23:41:08.318189Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/7a/7c/d7b2a0417af6428440c0ad7cb9799073e507b1a465f827d058b826236964/numpy-1.24.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=d11efb4dbecbdf22508d55e48d9c8384db795e1b7b51ea735289ff96613ff74d cache_mode=OnlyIfCached
2023-10-02T23:41:08.318481Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/18/9d/e02ace5d7dfccee796c37b995c63322674daf88ae2f4a4724c5dd0afcc91/numpy-1.24.4-cp39-cp39-win32.whl#sha256=6620c0acd41dbcb368610bb2f4d83145674040025e5536954782467100aa8835 cache_mode=OnlyIfCached
2023-10-02T23:41:08.318760Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/63/38/6cc19d6b8bfa1d1a459daf2b3fe325453153ca7019976274b6f33d8b5663/numpy-1.24.4-cp39-cp39-win_amd64.whl#sha256=befe2bf740fd8373cf56149a5c23a0f601e82869598d41f8e188a0e9869926f8 cache_mode=OnlyIfCached
2023-10-02T23:41:08.318910Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/a4/fd/8dff40e25e937c94257455c237b9b6bf5a30d42dd1cc11555533be099492/numpy-1.24.4-pp38-pypy38_pp73-macosx_10_9_x86_64.whl#sha256=31f13e25b4e304632a4619d0e0777662c2ffea99fcae2029556b17d8ff958aef
 cache_mode=OnlyIfCached
2023-10-02T23:41:08.319053Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/42/e7/4bf953c6e05df90c6d351af69966384fed8e988d0e8c54dad7103b59f3ba/numpy-1.24.4-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=95f7ac6540e95bc440ad77f56e520da5bf877f87dca58bd095288dce8940532a cache_mode=OnlyIfCached
2023-10-02T23:41:08.319171Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/fc/dd/9106005eb477d022b60b3817ed5937a43dad8fd1f20b0610ea8a32fcb407/numpy-1.24.4-pp38-pypy38_pp73-win_amd64.whl#sha256=e98f220aa76ca2a977fe435f5b04d7b3470c0a2e6312907b37ba6068f26787f2 cache_mode=OnlyIfCached
2023-10-02T23:41:08.319473Z  INFO rattler_installs_packages::http: executing request url=https://files.pythonhosted.org/packages/6b/80/6cdfb3e275d95155a34659163b83c09e3a3ff9f1456880bec6cc63d71083/numpy-1.24.4-cp310-cp310-macosx_10_9_x86_64.whl.metadata#sha256=c0bfb52d2169d58c1cdb8cc1f16989101639b34c7d3ce60ed70b19c63eba0b64 cache_mode=NoStore
Resolved environment:
- PyWavelets

Name        Version
numpy       1.24.4
pywavelets  1.4.1

e.g.
executing request url=https://files.pythonhosted.org/packages/6b/80/6cdfb3e275d95155a34659163b83c09e3a3ff9f1456880bec6cc63d71083/numpy-1.24.4-cp310-cp310-macosx_10_9_x86_64.whl.metadata#sha256=c0bfb52d2169d58c1cdb8cc1f16989101639b34c7d3ce60ed70b19c63eba0b64

Is this expected? It appears to be slowing rip down when the wheels aren't cached and it's having to do a lot of backtracking, I noticed it particularly when trying this spec on Python 3.8 (Windows or Linux):
"numpy==1.21.6" "cython==0.29.28" "scipy>=1.4.0" "torch>=1.7" "torchaudio" "soundfile" "librosa==0.10.0.*" "numba==0.55.1" "inflect==5.6.0" "tqdm" "anyascii" "pyyaml" "fsspec>=2021.04.0" "aiohttp" "packaging" "flask" "pysbd" "pandas" "matplotlib" "trainer==0.0.20" "coqpit>=0.0.16" "pypinyin" "mecab-python3==1.0.5" "jamo" "bangla==0.0.2" "k_diffusion" "einops" "transformers"

It is expected, although we can probably filter out a bunch of artifacts (related to #3). The idea is that when we try to gather metadata for a particular version of a package we check if we have any artifact already cached. (Note the cache_mode=OnlyIfCached). This only queries the disk cache for existence and should be fairly quick. The metadata for all artifacts "should" be the same so this seems like a nice optimization.
Looking at your logging statement it appears it takes 5ms (08.319171 - 08.313789) to check the cache which seems reasonable. We can probably also do concurrent requests but I'm not sure that will be much faster.

I also noticed you run the debug build, for measuring performance it might be worth it to run the binary in release mode:
cargo r --release -- PyWavelets

With #42 we now only consider wheels that are relevant to the installed python interpreter.

Oh wow, that was fast, I was going to write a quick script to see how much real world performance inpact it had.
While 5ms isn't much on its own, you potentially it many thousand of time.

While 5ms isn't much on its own, you potentially doing it many thousand of times for a complex backtrack.

Thats true! I imagine that in the future we could simply parallelize these things. Closing for now since #42 has been merged.


Rails generator for resources

Initial attempt at creating a Rails generator for JSONAPI resource files

Cool, thanks. Looks like a good start. This is now creating a resources directory in the project. We should add this to the .gitignore file or clean up the files after the tests.
Also it seems the resources are replaced if the generator is run again. I don't use the Rails generators much so I don't know if this is standard. But it seems risky as you could overwrite some of your work if it accidentally run it again (which I'm sure I would do at times with a missed keystroke in selecting command history).
I don't know if generators usually go this far, but it would be cool to interrogate the model and setup the default relationships and attributes. That's certainly not needed at first, but it seems useful.

Thanks for the feedback. If you are inheriting from the standard rails generator it looks like get a confirmation to overwrite.
I agree it would be cool to have more integration. I think the next step would be to add parameters for attributes.

I just tested it out in a real project and it does warn you if the file exists. So no issue there. However it created the resource directory in the root of the project, and by convention the directory should be app/resources.

Nice idea @kevintraver. I don't use generators all that often, but I think that there a friendly feature for new users of jsonapi-resources.

I've fixed the teardown/cleanup issue, and improved the tests.

Thanks!

@kevintraver @lgebhardt any plans to add this generator into the Rails config.generators hooks so that it happens automatically (if configured to do so) when generating a rails resource? rails g resource user for example, would also generate a jsonapi resource.


[SPARK-4924] Add a library for launching Spark jobs programmatically.

This change encapsulates all the logic involved in launching a Spark job
into a small Java library that can be easily embedded into other applications.
The overall goal of this change is twofold, as described in the bug:

Provide a public API for launching Spark processes. This is a common request
from users and currently there's no good answer for it.

Remove a lot of the duplicated code and other coupling that exists in the
different parts of Spark that deal with launching processes.

A lot of the duplication was due to different code needed to build an
application's classpath (and the bootstrapper needed to run the driver in
certain situations), and also different code needed to parse spark-submit
command line options in different contexts. The change centralizes those
as much as possible so that all code paths can rely on the library for
handling those appropriately.

Test build #25118 has finished for   PR 3916 at commit 656374e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25118/
Test FAILed.

Test build #25138 has started for   PR 3916 at commit a7936ef.

This patch merges cleanly.

Test build #25138 has finished for   PR 3916 at commit a7936ef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25138/
Test PASSed.

Test build #25265 has started for   PR 3916 at commit 53faef1.

This patch merges cleanly.

Test build #25265 has finished for   PR 3916 at commit 53faef1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25265/
Test FAILed.

Test build #25271 has started for   PR 3916 at commit bb5d324.

This patch merges cleanly.

Test build #25271 has finished for   PR 3916 at commit bb5d324.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25271/
Test PASSed.

Test build #25350 has started for   PR 3916 at commit f26556b.

This patch does not merge cleanly.

Test build #25351 has started for   PR 3916 at commit fc6a3e2.

This patch merges cleanly.

Test build #25351 has finished for   PR 3916 at commit fc6a3e2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25351/
Test FAILed.

Test build #25350 has finished for   PR 3916 at commit f26556b.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25350/
Test PASSed.

Hey @vanzin I left a ton of minor comments. I haven't got through the whole thing yet but on a high level the structure makes a lot of sense.

I'm going to continue reviewing this. By the way it would be good for @sryza to look at this too since you wrote much of the original Spark submit code.

 I tried running this myself after build/sbt clean assembly and it says
Failed to find Spark launcher in /Users/andrew/Documents/dev/spark/andrew-spark/launcher/target/scala-2.10.
You need to build Spark before running this program.

Is there a special way with which I'm supposed to build this?

re: the missing jar, you probably need to run package aside from just assembly, since the latter doesn't cause the launcher jar to be built. If there's any trick on SparkBuild.scala I can use to automatically do that, please let me know.

Unrelated question. I notice that SparkLauncher doesn't have the full set of options found in SparkSubmitArguments. If the user wants to launch Spark from their own java applications, what is the recommended way for them to set, say, driver memory or the YARN queue name?

Test build #25561 has started for   PR 3916 at commit 61919df.

This patch merges cleanly.

HI @andrewor14 , thanks for all the comments. I updated the patch and I think I covered everything. Mainly, I fixed the issue with new lines that you asked about, and also now sbt assembly should work without having to do sbt package too.

I notice that SparkLauncher doesn't have the full set of options found in SparkSubmitArguments

My thinking is that aside from the exposed APIs, everything else would be set using SparkLauncher.setConf(). I even though about removing some other methods (like setMaster()) but decided to leave the most common ones easily accessible.

Test build #25561 has finished for   PR 3916 at commit 61919df.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25561/
Test FAILed.

Hmm, tests pass localy. I'd expect all of those to fail (not just the three in the last run) if there was a problem with my code.

Test build #25571 has started for   PR 3916 at commit 54cd4fd.

This patch merges cleanly.

Just had a look through this.  Overall, things look reasonable.  Had a few nits in the code and a few broader comments here:
Did you consider using ProcessBuilder for building the commands in AbstractLauncher?  It abstracts over the Linux and Windows ways of launching processes and handles some of the stuff that it looks like you're handling manually.
I think it's weird to expose constants for configuration property names in LauncherCommon.  If we want Spark to start exposing constants for these, we should do it somewhere in core with a class name that's more specific.  I'm worried that people are going to start referring to these constants when creating new SparkConfs, and it'll look weird to have them coming from LauncherCommon.

Did you consider using ProcessBuilder for building the commands in AbstractLauncher?

ProcessBuilder doesn't help here at all. The platform-specific branches aren't related to how to run a process, but how to return a string that can be executed from a bash script on Linux/Mac or batch script on Windows.
The code that launches processes directly (SparkLauncher.launch()) uses ProcessBuilder and doesn't have any platform-specific code.

If we want Spark to start exposing constants for these, we should do it somewhere in core

This library cannot depend on core.

I'm worried that people are going to start referring to these constants

That's probably a good thing. It's a centralized place where they're defined, and thus it means that people will have to think before changing them.

That's probably a good thing.

I don't necessarily dispute this.  I believe I've pushed for this in the past.  But I think it's weird that "LauncherCommon" is what they need to import to get them.  Something like "AppConfigurationKeys" would be better.  I also think it might be better to add these in a separate patch if we choose to do so.

Test build #25574 has started for   PR 3916 at commit 7ed8859.

This patch merges cleanly.

But I think it's weird that "LauncherCommon" is what they need to import to get them.

Well, it's a little weird, but you can do import static org.apache.spark.launcher.SparkLauncher.*; to get those constants too.
I chose this approach to minimize the number of imports people need (generally they just import SparkLauncher and be done with it, and access constants via SparkLauncher.FOO). But I'm not against introducing a second class with just constants if people prefer that.

I think those tests have been flaky.

Test build #25571 has finished for   PR 3916 at commit 54cd4fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25571/
Test PASSed.

Test build #25574 has finished for   PR 3916 at commit 7ed8859.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25574/
Test PASSed.

Test build #25582 has started for   PR 3916 at commit 44cd5f7.

This patch merges cleanly.

Test build #25582 has finished for   PR 3916 at commit 44cd5f7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25582/
Test PASSed.

Test build #25831 has started for   PR 3916 at commit e570fb5.

This patch merges cleanly.

Test build #25831 has finished for   PR 3916 at commit e570fb5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25831/
Test PASSed.

Test build #25852 has started for   PR 3916 at commit eae4d8e.

This patch merges cleanly.

Test build #25852 has finished for   PR 3916 at commit eae4d8e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25852/
Test PASSed.

At some point we should put some documentation in the .md files about the library and how to use it.

Test build #25904 has started for   PR 3916 at commit 7cff919.

This patch merges cleanly.

Test build #25904 has finished for   PR 3916 at commit 7cff919.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25904/
Test FAILed.

Test build #26031 has started for   PR 3916 at commit 23aa2a9.

This patch merges cleanly.

Test build #26031 has finished for   PR 3916 at commit 23aa2a9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26031/
Test PASSed.

Hey @vanzin I thought about this patch more and I think it would be a little risky to rush it into 1.3.0 when the code freeze is in a few days. Admittedly we don't have a ton of reviewers on this patch, but since this changes the entire gateway of how everyone submits their applications I would rather be more conservative here. Also it will be a lot of work QA'ing this in various modes and testing the various config settings (e.g. escaping various characters in config values and precedence), so I would prefer to push it back to the next release.

Hi @andrewor14, I understand, but can we please keep moving forward with the reviews? That way, we'll be ready to merge soon after branch-1.3 is cut, and we'll avoid this happening again later on.

Test build #26103 has started for   PR 3916 at commit ad03c48.

This patch merges cleanly.

Test build #26103 has finished for   PR 3916 at commit ad03c48.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26103/
Test PASSed.

Test build #26775 has started for   PR 3916 at commit b93692a.

This patch merges cleanly.

Test build #26776 has started for   PR 3916 at commit 46d46da.

This patch merges cleanly.

Test build #26775 has finished for   PR 3916 at commit b93692a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26775/
Test FAILed.

Test build #26776 has finished for   PR 3916 at commit 46d46da.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26776/
Test PASSed.

Test build #27164 has started for   PR 3916 at commit 2061967.

This patch merges cleanly.

Test build #27164 has finished for   PR 3916 at commit 2061967.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27164/
Test FAILed.

Test build #27246 has started for   PR 3916 at commit de81da2.

This patch merges cleanly.

Test build #27246 has finished for   PR 3916 at commit de81da2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27246/
Test PASSed.

@vanzin about half way through reviewing... will pick up tomorrow

HI @pwendell , thanks for all the feedback!
I pushed some updates that implement most (or all) you've commented on. That required renaming and moving a bunch of things, but the underlying code is not all that different. So hopefully it will not be hard to figure out what's going on after having seen the old patch.

Test build #27749 has started for   PR 3916 at commit 95ddfa8.

This patch merges cleanly.

Test build #27749 has finished for   PR 3916 at commit 95ddfa8.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27749/
Test FAILed.

Test build #27750 has started for   PR 3916 at commit 8ec0243.

This patch merges cleanly.

Test build #27750 timed out     for PR 3916 at commit 8ec0243     after a configured wait of 120m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27750/
Test FAILed.

Test build #27786 has started for   PR 3916 at commit 304333a.

This patch merges cleanly.

Test build #27786 has finished for   PR 3916 at commit 304333a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27786/
Test PASSed.

Hey @vanzin okay took another quite long look (sorry, was delayed this week due to strata) and I have embarrassingly few useful comments given how long I looked at it. Overall, the changes you made all seem great. Unfortunately since spark-submit as it stands before your patch is a fairly wide interface, it's difficult for me to conceive of every possible combination of settings/environments/etc, etc that could be broken by this. I think we'll just have to merge it and hope it gets a substantial amount of user testing.
The main thing left for me is the proposal to expose this as a public interface to Spark for application developers. The main issue I see is that it just won't be backwards compatible as it stands now, and this is something where I think compatibility is pretty important. One main use case I see here is some third party application wants to have a nice way of submitting Spark applications for user clusters that already have Spark installed, so they bundle this up with their app. At least, that's the main reason I've heard users directly ask for this feature. If they have to grab a newer version of this for every different Spark version users might have installed, that's a big pain for them. And it's the exact opposite API guarantees as the rest of Spark. And there is an alternative that is backwards compatible which is for them to just script around spark-submit. So users are in a weird place where there are two options and neither is strictly optimal.
To me that seems like a blocker to exposing it in the current form without exploring more options.
If we want to make this backwards compatible, we'd need to minimize the interface between the publicly exposed library and the supplied Spark distribution. If the interface from the user app is that we fork a subprocess when they call launch(), could that subprocess just invoke the spark-submit script in the distribution, which will then itself compute the classpath, etc? We'd just have to detect whether to invoke the windows or unix version, but that seems pretty doable. Could that work?
As a point of process, I'd be happy to explore various options for making this work by just merging a version where this is private and then building on top of that, to avoid staleness and allow for ample testing. I really do think its a good idea to have a programmatic version of spark-submit.

@pwendell I see what you mean about compatibility. Let me play with the code a bit, it might not be hard to do something like that as part of this patch.

Test build #27866 has started for   PR 3916 at commit 43008a7.

This patch merges cleanly.

Test build #27866 has finished for   PR 3916 at commit 43008a7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27866/
Test PASSed.

Test build #27893 has started for   PR 3916 at commit e2d4d71.

This patch merges cleanly.

Test build #27893 has finished for   PR 3916 at commit e2d4d71.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27893/
Test FAILed.

Test build #27906 has started for   PR 3916 at commit c12d84b.

This patch merges cleanly.

Test build #27906 has finished for   PR 3916 at commit c12d84b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27906/
Test FAILed.

@vanzin I'm done with this round of review. A majority of my comments are pretty minor. The bigger ones have to do with making sure the things that don't need to be public are not public, and how the pyspark shell is launched.

Test build #27936 has started for   PR 3916 at commit 0031a8e.

This patch merges cleanly.

Test build #27936 has finished for   PR 3916 at commit 0031a8e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27936/
Test FAILed.

Test build #27958 has started for   PR 3916 at commit 7e66c18.

This patch merges cleanly.

Test build #27958 has finished for   PR 3916 at commit 7e66c18.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27958/
Test PASSed.

Test build #27971 has started for   PR 3916 at commit 4c19196.

This patch merges cleanly.

Test build #27972 has started for   PR 3916 at commit 6184c07.

This patch merges cleanly.

Test build #27972 has finished for   PR 3916 at commit 6184c07.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27972/
Test FAILed.

@vanzin @pwendell As of the latest changes I think this PR is almost ready to be merged. An obvious blocker is that the SparkLauncher needs to also handle Windows, but we can fix that easily.
One bigger thing I wanted to clarify is the following: The scope of the public API this PR currently exposes is narrowed to the extent that SparkLauncher calls a particular distribution's spark-submit. IIUC the only backward compatibility requirement this entails is that we don't remove or change any of the fields that the SparkLauncher sets (e.g. version, master, app name). Is this correct? This seems like a narrow enough interface to keep unchanged in future versions of Spark such that merging this will be safe.

retest this please

Test build #27977 has started for   PR 3916 at commit 6184c07.

This patch merges cleanly.

SparkLauncher works on Windows. Did you see anything that indicates otherwise? I even managed to get all unit tests under launcher/ to pass on Windows (although I didn't test after the latest changes).
The only things that need to be kept for compatibility going forward are:

SparkLauncher's public API. You can change the internal fields all you want.
spark-submit's public API. It needs to keep accepting command lines the way it does today.

Test build #27971 has finished for   PR 3916 at commit 4c19196.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27971/
Test PASSed.

It seems the change to run things through spark-submit did cause a regression on Windows since the batch interpreter treats = characters especially. Let me try to fix it.

Test build #27977 has finished for   PR 3916 at commit 6184c07.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27977/
Test PASSed.

Sorry I haven't had time to look at this in detail, one thing I was looking for though was some documentation for the high level api.  I realize java doc will document most of it, but are we going to add some documentation to say  this is available and point users to javadoc?  Perhaps to deploy guide.

@tgravescs that can definitely be added, but I'd rather do it separately at this point. Also, regarding javadoc, I'm not sure how it's generally published for Spark: the "unidocs" generated from java classes seem to all be empty (while scala documentation is fine, and separately the generated javadocs are also fine). Is the unidoc stuff broken for java code?

I'm ok with it being separate, although in general I think its bad practice. I've seen some features go in without documentation which I think as a user is a major pain.

Test build #28016 has started for   PR 3916 at commit 5f4ddcc.

This patch does not merge cleanly.

Test build #28017 has started for   PR 3916 at commit 00505f9.

This patch does not merge cleanly.

Test build #28018 has started for   PR 3916 at commit b1d86b0.

This patch merges cleanly.

Test build #28016 has finished for   PR 3916 at commit 5f4ddcc.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28016/
Test PASSed.

Test build #28017 has finished for   PR 3916 at commit 00505f9.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28017/
Test PASSed.

Test build #28018 has finished for   PR 3916 at commit b1d86b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28018/
Test PASSed.

Test build #28200 has started for   PR 3916 at commit 28cd35e.

This patch merges cleanly.

Test build #28200 has finished for   PR 3916 at commit 28cd35e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28200/
Test FAILed.

Test build #28205 has started for   PR 3916 at commit e2367d2.

This patch merges cleanly.

Test build #28205 has finished for   PR 3916 at commit e2367d2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28205/
Test PASSed.

FYI: There is a question on SO, I believe, about the type of functionality being added in this PR.

Hey Marcelo,
I think this is really close to being ready to merge. I noticed some extra output (a '0') when running usage on spark-shell:
> ./bin/spark-shell --help
0
Usage: ./bin/spark-shell [options]
Spark assembly has been built with Hive, including Datanucleus jars in classpath.

I think whether this should be an exposed API is still a question. However, I'm fine to merge v1 of this as an exposed API and get feedback. It would be helpful to hear from potential users whether this is useful to them... so far I haven't heard much in that direction.

Test build #28401 has started for   PR 3916 at commit 897141f.

This patch merges cleanly.

Test build #28401 has finished for   PR 3916 at commit 897141f.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28401/
Test FAILed.

Test build #28403 has started for   PR 3916 at commit a1b8af1.

This patch merges cleanly.

Test build #28403 has finished for   PR 3916 at commit a1b8af1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28403/
Test FAILed.

Test build #28404 has started for   PR 3916 at commit 3b28a75.

This patch merges cleanly.

Sorry to spam the PR with comments about Bash word splitting, @vanzin. I think what you have Bash-wise looks good, but as a matter of consistency I just went through and marked the places where Bash word splitting bugs are theoretically possible.
As a habit, I think we should always quote Bash output (whether of variables, subshell, etc.), though I can see if in some cases you feel it would be unnecessary.

Test build #28407 has started for   PR 3916 at commit 2ce741f.

This patch merges cleanly.

Test build #28404 has finished for   PR 3916 at commit 3b28a75.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28404/
Test PASSed.

Test build #28407 has finished for   PR 3916 at commit 2ce741f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28407/
Test PASSed.

@andrewor14 are you good with this one? I'd like to merge it soon!

Yeah this LGTM. There are a few other minor comments I left regarding visibility but they don't have to block this PR from going in. I haven't tested whether all the cases (Windows, YARN, PySpark...) are maintained as before and I won't have the bandwidth to do so this week, but we can always do that separately.

FYI I ran our test suite with this change (covers standalone/client, yarn/both, spark-shell and pyspark, Linux only) and all looks ok. Also did some manual testing on Windows.

Test build #28447 has started for   PR 3916 at commit 18c7e4d.

This patch merges cleanly.

Test build #28447 has finished for   PR 3916 at commit 18c7e4d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28447/
Test PASSed.

Okay cool - LGTM I will pull this in. I just did some local sanity tests, built with Maven and ran (a few) Maven tests. We'll need to keep any eye on the Maven build tomorrow to see if this is succeeding there, since it's not caught by the PRB. Thanks @vanzin for all the work on this.


cssnano is discarding sourcemap comment

I'm using clean-css to get rid of repeated selectors, and cssnano to minify.
clean css adds a sourcemap.

cssnano removes it

I've even tried this with the --no-discardComments feature in the CLI and it still removes the sourcemap comment, so I'm not even sure if this is the right place to open this issue...

Sourcemaps are handled by PostCSS, you'll need to configure those.
https://github.com/postcss/postcss/blob/master/docs/source-maps.md

Why is cssnano discarding any comments whatsoever...?

It's not cssnano that does it, PostCSS takes care of sourcemaps. I have no control over their removal with this plugin. The documentation wasn't clear on that so I have updated it.
https://github.com/ben-eb/postcss-discard-comments#output

Alright thanks for updating docs. :+1:


lock node version and fix recaptcha variable name

What: lock node version and fix recaptcha variable name
Why: to use the node LTS version and keep the variable name consistent
How:

lock node version to 14.15.0
change recaptcha variable name to GATSBY_RECAPTCHA_V3_SITE_KEY

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


feat: add DateSelector dateFormat prop

Summary
Related Issues or PRs

How To Test

Screenshots (optional)

This seems useful — I'd just recommend updating DatePicker.stories.tsx to demonstrate how to use it.

Go for it!

This seems useful — I'd just recommend updating DatePicker.stories.tsx to demonstrate how to use it.

Go for it!

Pardon the accidental close, here's the commit

Any update on this?

Any update on this?

Just waiting on code reviews from our engineers (for this and the other PRs), pardon the delay

I fixed a form validation bug. Please review again.

I rebased this again. Any updates on when someone will review this?

It's been a few more months since the last time I checked on this. Would someone please review it?


Callback to Agent about episode end.

An end-of-episode signal to the agent would be beneficial. This could help unify inference (run_battle) with training. It could also help with MCTS and alerting it to reset the search tree.
In BaseAgent, this could look like:
def episode_end(reward):
"""Fill in here"""
pass

@tambetm, aye?

I'll give it a shot ASAP. I still have to catch up with the latest commits in master :).

I'll give it a shot ASAP. I still have to catch up with the latest commits in master :).

Merging this into #63 as it's more active.


Create Use Cases Pages

Can we also add a link in the header "Use Cases" that links to the top of this section?


Shouldn't key names use double quotes, instead of single?

If I copy the output of the search result ("print(videos)") and paste it into a JSON editor, the editor fails to parse the output, because the name is single-quoted. That is, it's like {'name': 'value'}. Shouldn't it be {"name": "value"}?

It returns a dict, not json, to convert you can use json.dumps()