[SPARK-13325][SQL] Create a 64-bit hashcode expression

This PR introduces a 64-bit hashcode expression. Such an expression is especially usefull for HyperLogLog++ and other probabilistic datastructures.
I have implemented xxHash64 which is a 64-bit hashing algorithm created by Yann Colet and Mathias Westerdahl. This is a high speed (C implementation runs at memory bandwidth) and high quality hashcode. It exploits both Instruction Level Parralellism (for speed) and the multiplication and rotation techniques (for quality) like MurMurHash does.
The initial results are promising. I have added a CG'ed test to the HashBenchmark, and this results in the following results (running from SBT):
Running benchmark: Hash For simple
  Running case: interpreted version
  Running case: codegen version
  Running case: codegen version 64-bit

Intel(R) Core(TM) i7-4750HQ CPU @ 2.00GHz
Hash For simple:                    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------
interpreted version                      1011 / 1016        132.8           7.5       1.0X
codegen version                          1864 / 1869         72.0          13.9       0.5X
codegen version 64-bit                   1614 / 1644         83.2          12.0       0.6X

Running benchmark: Hash For normal
  Running case: interpreted version
  Running case: codegen version
  Running case: codegen version 64-bit

Intel(R) Core(TM) i7-4750HQ CPU @ 2.00GHz
Hash For normal:                    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------
interpreted version                      2467 / 2475          0.9        1176.1       1.0X
codegen version                          2008 / 2115          1.0         957.5       1.2X
codegen version 64-bit                    728 /  758          2.9         347.0       3.4X

Running benchmark: Hash For array
  Running case: interpreted version
  Running case: codegen version
  Running case: codegen version 64-bit

Intel(R) Core(TM) i7-4750HQ CPU @ 2.00GHz
Hash For array:                     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------
interpreted version                      1544 / 1707          0.1       11779.6       1.0X
codegen version                          2728 / 2745          0.0       20815.5       0.6X
codegen version 64-bit                   2508 / 2549          0.1       19132.8       0.6X

Running benchmark: Hash For map
  Running case: interpreted version
  Running case: codegen version
  Running case: codegen version 64-bit

Intel(R) Core(TM) i7-4750HQ CPU @ 2.00GHz
Hash For map:                       Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------
interpreted version                      1819 / 1826          0.0      444014.3       1.0X
codegen version                           183 /  194          0.0       44642.9       9.9X
codegen version 64-bit                    173 /  174          0.0       42120.9      10.5X

This shows that algorithm is consistently faster than MurMurHash32 in all cases and up to 3x (!) in the normal case.
I have also added this to HyperLogLog++ and it cuts the processing time of the following code in half:
val df = sqlContext.range(1<<25).agg(approxCountDistinct("id"))
df.explain()
val t = System.nanoTime()
df.show()
val ns = System.nanoTime() - t

// Before
ns: Long =<PHONE_NUMBER>

// After
ns: Long =<PHONE_NUMBER>

cc @cloud-fan (you have been working on hashcodes) / @rxin

Test build #51324 has started for PR 11209 at commit f14c50c.

Test build #51324 has finished for PR 11209 at commit f14c50c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51324/
Test FAILed.

@jodersky / @cloud-fan The tests also failed on my machine. It turns out I messed up the initialization order during some cleaning up. This is fixed and the tests should pass now.

Test build #51347 has started for PR 11209 at commit 54c818b.

Test build #51347 has finished for PR 11209 at commit 54c818b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51347/
Test PASSed.

Merged build finished. Test PASSed.

Test build #51366 has started for PR 11209 at commit 8b09882.

Test build #51367 has started for PR 11209 at commit 6e43d08.

I added a synthetic benchmark in order to check the performance. The performance should be the best when we hash large chuncks of data, in this case byte arrays of 8223 bytes. The array is chosen in such a way that xxHash64 and MurMur both have to deal with non-word aligned input.
I have calculates the speed by making the following calculation (please correct me if you feel that this approach is wrong):
(numRows * numIterations * rowSize) / AvgTime

If I do this for the largest case I get to 10,2 GB/s:
val bytesPerSecond = ((1L << 10) * (1L << 11) * 8223L) / 1.569D
val gbPerSecond = bytesPerSecond / (1024 * 1024 * 1024)
>> gbPerSecond: Double = 10.236167543021033

Test build #51366 has finished for PR 11209 at commit 8b09882.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51366/
Test PASSed.

Merged build finished. Test PASSed.

Test build #51367 has finished for PR 11209 at commit 6e43d08.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51367/
Test PASSed.

Test build #51568 has started for PR 11209 at commit 0554224.

Test build #51568 has finished for PR 11209 at commit 0554224.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class HashExpression[E] extends Expression
abstract class InterpretedHashFunction[E]
case class Murmur3Hash(children: Seq[Expression], seed: Int) extends HashExpression[Int]
case class PrintToStderr(child: Expression) extends UnaryExpression
case class XxHash64(children: Seq[Expression], seed: Long) extends HashExpression[Long]

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51568/
Test PASSed.

Test build #53079 has started for PR 11209 at commit 1b08cd9.

Test build #53079 has finished for PR 11209 at commit 1b08cd9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53079/
Test FAILed.

Test build #53107 has started for PR 11209 at commit 1acee97.

Test build #53107 has finished for PR 11209 at commit 1acee97.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53107/
Test PASSed.

Test build #53218 has started for PR 11209 at commit 79af847.

Test build #53218 has finished for PR 11209 at commit 79af847.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53218/
Test PASSed.

Test build #53767 has started for PR 11209 at commit fcb25c2.

Test build #53767 has finished for PR 11209 at commit fcb25c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class InterpretedHashFunction

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53767/
Test PASSed.

LGTM

Thanks for the review! merging to master!


Updated syntax and reworked layout

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

@michaelrappaz Well done! Thank you, and congrats on your first-time contribution!


Added recently published companion to Soft Skills

This book expands on the career stuff from Soft Skills.

Thanks for the contribution!
Did not know that it has been already published. I'd prefer to read it and evaluate first before adding to the list.
I'll let it hang here for some time 👍

Makes sense! Nice job on this guide.

I second this. Excellent book.

Many people forget the importance of soft skills these days. This is a good reminder.

Thank you for all your comments!
Adding it right away! 🚀


Update goreleaser to add latest tags

Fix https://github.com/open-telemetry/opentelemetry-collector-releases/issues/73
Signed-off-by: Bogdan Drutu<EMAIL_ADDRESS>

@codeboten the reason to upgrade was because of some bug in v2 (rendering the version). I will split that.

@jpkrohling @codeboten please review https://github.com/open-telemetry/opentelemetry-collector-releases/pull/221 first


Adding messaging-with-jms-queues guide

Purpose
Adding Ballerina guide to explain the process of 'messaging with JMS queues' through a real-world example

Hi,
The requested change has been made.
Changed the MB Client from 'WSO2 EI Message Broker' to 'Apache ActiveMQ'


Cannot execute myth -x command on Mojave 10.14 / Python 3.7.2

Description
I installed  Mythrill:
brew update
brew upgrade
brew tap ethereum/ethereum
brew install leveldb
brew install solidity
pip3 install mythril

Then tried to run the mythrill command-line:
myth -x my_smart_contract.sol

But got this error:
Traceback (most recent call last):
  File "/usr/local/bin/myth", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/mythril/interfaces/cli.py", line 293, in main
    enable_online_lookup=args.query_signature,
  File "/usr/local/lib/python3.7/site-packages/mythril/mythril.py", line 97, in __init__
    self.mythril_dir = self._init_mythril_dir()
  File "/usr/local/lib/python3.7/site-packages/mythril/mythril.py", line 130, in _init_mythril_dir
    copyfile(str(parent_dir / "signatures.db"), db_path)
  File "/usr/local/Cellar/python/3.7.2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/site-packages/mythril/signatures.db'

I already tried this fix, without success (same error message):
mkdir ~/.mythrill
pip install --upgrade git+https://github.com/tintinweb/ethereum-dasm.git@master

How to Reproduce
(See above)
Expected behavior
Mythrill command-line should run properly
Environment

Mythril version: Mythril version v0.19.11
Solidity compiler and version: Version: 0.5.2+commit.1df8f40c.Darwin.appleclang
Python version: Python 2.7.15 (But python3 points to Python 3.7.2)
OS and Version: Mojave 10.14

The fix version will be released soon. By then can you take the signatures.db file in mythril repo(https://github.com/ConsenSys/mythril-classic/blob/develop/signatures.db) and move it to ~/.mythril directory. That might fix it.


Extend Pauli twirling to end users

What should we add?
Qiskit has the code to Pauli twirl circuits  (up to a phase factor) within its unit tests. Other SDKs have this functionality builtin, e.g. pytket.tailoring.PauliFrameRandomisation.  This functionality should be exposed to the end users.

Christmas comes early...


added documentation for logging

added documentation for logging
page size increased 640px to 1024px, for better view
Little bit modifications on documentation

Looks good for me


A server is required

Hello, here is my config file:
server1:
	user: root
	password:
	port: 3306
	host: localhost
server2:
	user: root
	password:
	port: 3306
	host: localhost
template: templates/simple-db-migrate.tmpl
type: schema
include: all
nocomments: true

And here is the cmd I'm trying to run
./dbdiff server1.database:server2.database-prod output=./sql/simple-schema.sql
When my config file contain tabulations, it doesn't work (yaml can't contain tabulations). When I remove them, I'm getting the error A server is required.
There is no password for now on the database, does I need to set one to use DBDiff?
Thank you for your reply.

Last three error messages:
✖ Unkown kind of resources
✖ A server is required
✖ Missing input
These error messages don't point to the problem. They tell you nothing. They are as helpful as "something went wrong".

@SteamWind are you sure your config file is named .dbdiff (including period?)

Yes, sure.

@aschkenasy
Ditto here. This command throws the same error:  ./dbdiff server1.epos:server2.epos
Here's my .dbdiff (anonymized)
server1:
user: username1
password: password1
port: 3306
host: sql.myserver.com
server2:
user: username2
password: password2
port: 3306
host: <IP_ADDRESS>
template: templates/simple-db-migrate.tmpl
type: schema
include: all
nocomments: true

I have this issue too.

It worked for me by using spaces instead of tabs for the intendation in the .dbdiff file.

same issue 🤯

I was receiving the error "A server is required."
My problem was in .dbdiff I changed "server1" and "server2" to be my actual server names.
After I changed them back, it worked fine.

have anyone fixed the problem? I tried different intendation and my server names are default ones (server1 and server2) and I still see this error


Code cells do not run

Environment data

VS Code version: 1.54.0-insider
Jupyter Extension version (available under the Extensions sidebar): v2020.12.414227025
Python Extension version (available under the Extensions sidebar): v2021.1.502429796
OS (Windows | Mac | Linux distro) and version: Mac 11.1
Python and/or Anaconda version: 3.7
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
Jupyter server running: Local

Expected behaviour
Python code cells are executed OK
Actual behaviour
Code cells take infinite time to execute - no output is shown
Steps to reproduce:

Open the IDE
Make sure the Python environment is selected properly - Conda's base environment
Run a new notebook with a Python code cell
Run this cell

Logs
User belongs to experiment group 'NativeNotebookEditor'

~/opt/anaconda3/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2020.12.414227025/pythonFiles/pyvsc-run-isolated.py -c "import jupyter"
~/opt/anaconda3/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2020.12.414227025/pythonFiles/pyvsc-run-isolated.py -c "import notebook"
~/opt/anaconda3/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2020.12.414227025/pythonFiles/pyvsc-run-isolated.py -c "import ipykernel"
~/opt/anaconda3/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2020.12.414227025/pythonFiles/pyvsc-run-isolated.py jupyter kernelspec --version
~/opt/anaconda3/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2020.12.414227025/pythonFiles/pyvsc-run-isolated.py -c "import ipykernel"
~/opt/anaconda3/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2020.12.414227025/pythonFiles/pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
~/opt/anaconda3/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2020.12.414227025/pythonFiles/pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
~/opt/anaconda3/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2020.12.414227025/pythonFiles/pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
~/opt/anaconda3/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2020.12.414227025/pythonFiles/pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_launcher_daemon -v
~/opt/anaconda3/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2020.12.414227025/pythonFiles/pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_launcher_daemon -v
Started kernel Python 3.7.6 64-bit ('base': conda)
~/opt/anaconda3/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2020.12.414227025/pythonFiles/pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_launcher_daemon -v

Seems to be affecting Julia as well...

I'm seeing exactly the same thing with a nearly identical environment. Some additional notes:

This ('NativeNotebookEditor' with Code-Insiders on Mac) was working for me about a week ago .
My conda environment is working fine with the non-native Jupyter extension with Code (non-Insiders).
This is a recent clean install of Code-insiders.

Hi all, so sorry for the trouble. This is happening because there was a breaking change in the VS Code API in VS Code Insiders late last week, and the Jupyter extension team is still working on publishing a compatible update to the VS Code marketplace for insiders users. Only VS Code Insiders users should be impacted by this, if you are seeing the same behavior on VS Code stable please file a separate issue. We should have an update early next week due to the long weekend but in the meantime you should be able to work around this by downgrading VS Code Insiders. Again, very sorry about the trouble caused and we appreciate your patience.

downgrading VS Code Insiders

@joyceerhl How can I do it?

@peterschmidt85 Downgrading for Windows: https://github.com/microsoft/vscode-jupyter/issues/4796#issuecomment-779266138
Downgrading for Apple Silicon: https://github.com/microsoft/vscode-jupyter/issues/4796#issuecomment-779374887
Based on the above it seems like the non-silicon macOS download link is https://vscode-update.azurewebsites.net/1.53.0-insider/darwin/insider. Please let us know if that doesn't work.

Dupe of #4796


Hot reloading

I am not sure if this is the right project asking for it but is something like react's hot reloading possible with Angular 2 + Webpack? I don't think is possible by Angular`s design?
Edit: the mentioned project is replaced by babel-plugin-react-transform.
More Information here

Hm.... I know webpack has hot reload, but I get the feeling the functionality described in your links is more advanced than this.
A better place to ask is https://github.com/angular/angular, I think.

I agree with @filipesilva that this is better asked on Core, however depending on what happens there we might need to make adjustments as well.

seems that this is a webpack configuration issue?
https://webpack.github.io/docs/hot-module-replacement-with-webpack.html
so this is the right place for my question

Hot module replacement as provided by webpack allows an app to reload a module/file on the fly. That bit is definitely a configuration feature.
However, the way by which the application responds to having modules being reloaded is an app/framework issue.
In the case of react, the hot reload behaviour seems to have been achieved via heavy instrumentation of components. This makes sense, because their state needs to somehow be persisted and re-introduced on the newly reloaded components.
I am not aware of such an angular feature, but there might be something to it if you ask on the angular repo. The CLI could perhaps have webpack's hot reload, but it wouldn't do much on it's own.

Here is a boilerplate with HMR enabled. It does not save the state when hot reloading.
This is a store specific thing?  Can this be solved by using RxJs?

nice article: http://blog.mgechev.com/2015/10/26/angular2-hot-loader-hot-loading-tooling
to sum up: it works with architecture considerations or it needs additional tools which patches the components without reinitiializing them.
Are there any angular tooling repositories? I think it's time to get the right tools for angular 2 now.

I am not aware of any, but have not really looked at it. Would love to hear about it if you find them!

There's a recent article demonstrating the use of hot reloading using Webpack.  Now that angular-cli is moving to Webpack, this is probably a really good thing to integrate.

You can find the angular2-cli friendly seed with redux and hot reloading here.
https://github.com/VladWulf/angular2-redux-hmr-boilerplate


Adapt v2.1.2

Implement new features mentioned in the changelog of the API

[x] Check new docs
[x] Add new endpoints (some endpoints seem not be accessible due not achievable scopes)
[x] Add interfaces and enums
[x] Change existing endpoints to reflect changes
[ ] Test everything

All endpoints which are not accessible due to not achievable scopes are commented out for now


Pattern for waiting for server data to load?

With fluxible-router-component, a route component delays rendering until the done() of an action is complete.
fluxible-router switches the route as soon as NAVIGATE_START is called and the RouteStore isn't listening for NAVIGATE_SUCCESS.  What's the desired pattern to delay loading the component until server data returns?

There isn't a way right now other than to use your own RouteStore or extending the built-in one. This could be an enhancement to the current RouteStore though: maintaining pending route change state.

Now we have a isNavigateComplete property that is passed when NAVIGATE_SUCCESS or NAVIGATE_FAILURE is heard. We'll be using this in our fluxible.io site. You can see the example usage here: https://github.com/yahoo/fluxible.io/pull/120/files?diff=unified#diff-e7d9b2441bd9e93366aca6ba7a5eccdeR21

this is terrific, thanks - just upgraded and everything is working fine


How to add forum routes to 'web' route groups to avoid errors

i have succeeded in integrating laravel and forum but one thing is left, the web guard middleware enbles session in laravel 5.2, so when i run /forum i get this errors:
ErrorException in 6829460faf86e4b23bc8a19c8baa92dc4b035c9e.php line 7:
Undefined variable: errors (View: C:\wamp\www\agron\resources\views\vendor\forum\partials\alerts.blade.php) (View: C:\wamp\www\agron\resources\views\vendor\forum\partials\alerts.blade.php) (View: C:\wamp\www\agron\resources\views\vendor\forum\partials\alerts.blade.php)

@jerryhanks take a look: #97

This does not solve my problem, if any one has used the forum with laravel 5.2 then they should tell me.

I did, and I couldn't reproduce the issue.

which folder is it. One more thing, i have installed it and the only thing i see is a page with no link for creating a category or thread or post or comment/reply, how can i start using it.

After publishing config via artisan vendor:publish, you'll find the option in config/forum.frontend.php.
Are you logged in? If so, are you using Laravel's standard auth or a third party authentication package? The forum uses Laravel's built-in Auth classes.

I am loged in and I am using larval standard Auth,

I have seen a link to create a new category but i get this error after creating a new catgeory
FatalErrorException in BaseModel.php line 138:
Call to a member function parameterNames() on a non-object

This is weird, but I have a feeling you're somehow using old package files. I say that because the line number in the current BaseModel for the line in question is 144, not 138. It moved down as of 3.0.2 with the addition of @deprecated and @todo lines.
Make sure you're using the latest version and that error should go away.

I am using 3.0

How do I update to the latest packed cos I just installed this package according the commands in the readme.md.

You can do it manually, by removing these lines from composer.json:
"riari/laravel-forum": "~3.0",
"riari/laravel-forum-frontend": "~1.0",

Then run composer update and it will remove package from your application.
After that you can install forum again from documentation.


Refactor the podcast views

The podcasts page is very long with all the podcasts and every episodes of each of them. I think it'd be better to only show the podcast description and the latest episode and keep all the episodes of a podcast for a detailed page for each podcast.
Todo:

[ ] create a show view with all the episodes of a podcast
[ ] remove the all episodes list in the podcast index page

I think you close it in #1094


Update References for each SWC-CTK

Currently most of the SWC-CTKs are just using CertiK.io as a default value, which seems not quite professional.
Unlike Relationships, contents in References should be no more conceptional, but closely related to Solidity and Ethereum. Consider using some popular community blogs, official documents, EIPs, reports of major security firms, etc. to fill the References section.

updated partially for SWC-CTK issues in commit be39bd8cc9eeca5725e2dadb6576f109b819149c


Backend: Auto Approval - Use Application

Issue: https://github.com/bcgov/entity/issues/22348
Description of changes:

Auto-approve use application instead of registrations
Update application status along with registration when applicable
Currently relationship between Application and Registrations in the code is one-to-one. Whereas the RFC depicts this as many-to-one. Outlined in comments where and what changes will be needed to accommodate this.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the namex license (Apache 2.0).

@shaangill025 are you able to assign reviewers?


Cannot replicate the results!

I tried to replicate the Xceptionnet results but failed. I strictly followed the data pre-processing and training. However, it turns out the log loss on the DFDC public test set is 0.4 at best after trying multiple runs. While I used the pre-trained xceptionnet the test loss is 0.3. I have several questions:

It takes us a day to train each epoch. How to train 20 epochs within a day?
It saves the best model after running validation at each epoch. However, training one epoch takes a long time and running validation only once per epoch may not be enough. I found that the model overfits quickly and best validation most is not really the best test model (ckpt-1 may be better).
For xceptionnet, any reason it does not use imagenet as pre-train?
For the same setting, does the test loss differ a lot between different runs?

Is 0.3 logloss the exact value you got? That is even better than our Kaggle record without WS-DAN (0.3250). Cannot give specific suggestion without further evidence. We will try the xception replication in our environment and see what will happen.
It takes us a day to train each epoch. How to train 20 epochs within a day?
Are you talking about the Xception code or the WS-DAN code? The Xception code should not be that slow. It only samples around 10% frames in one epoch to save time and validate more often. Could it be your IO too slow?
It saves the best model after running validation at each epoch. However, training one epoch takes a long time and running validation only once per epoch may not be enough. I found that the model overfits quickly and best validation most is not really the best test model (ckpt-1 may be better).
As above mentioned, the code only samples around 10% frames in one epoch so validation is more frequent. And it is very possible that best validation is not the best test model, which almost every DFDC team suffered.
For xceptionnet, any reason it does not use imagenet as pre-train?
I checked with our members. The xception model did use imagenet pretrained weight to initialize (from: https://github.com/Cadene/pretrained-models.pytorch). Sorry, it is not reflected in the code. I will update it later.
For the same setting, does the test loss differ a lot between different runs?
Randomness in augmentation (and other parts) could impact the result. But intuitively we don't think it affect much.

Thanks for your answer. It clears up a lot of the questions in my mind. I have one additional question, in your csv file, the number of frames sometimes do not match with the actual frames of the video. How did you determine the number of frames in a video?

Also do you notice which epoch is usually best.pth? I my case it is ckpt-5.pth, should I train longer or it is the case that it converges at epoch 5?

Sorry, I should make it clear that 0.4 is our xceptionet result, but I take your model from Google drive, the xceptionet result is 0.3.
Also, I changed to xceptionnet pre-trained, and it does not make much a difference for the final log loss, which is still 0.4+. I did not change any of your code, so I am not sure what went wrong.
I notice that you set to a random image when such frame does not exist. Does it affect the final results?

Hi, another question i wanna ask is, I assume xception-hg-2.pth is the best.pth you saved when running train-xceptiopn.py? However, I found that the test log loss of ckpt-1.pth is usually smaller than best.pth, although they are both 0.4+.
For wsdan, could you let me know how you saved ckpt_x.pth and ckpt_e.pth? The code seems does not tell this at all.


Allow multiple amiSelectorTerms with AND

Description
What problem are you trying to solve?
I have multiple AMI's which are tagged with a specific tag to allow them to be launched, the Name tag is not populated (and cannot be), so need to use the name attribute as well to get the correct AMI on launch. Instead of the image being found with the tag AND the name attribute, just the tag is used. This has caused strange behaviour (including Windows images being picked instead of Linux) so it would be useful to be able to set an AND if required.
How important is this feature to you?
Very important, we are testing this out as a replacement for autoscaler, and this is a real requirement. Not having a more complex filter is a bit of a pain and makes picking the AMI much harder :(

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

You should be able to specify an AND requirement by using a single amiSelectorTerm
amiSelectorTerms:
- name: <my-ami-name>
  tags:
     my-tag: <my-value>

That resolved it, thanks very much


Fixed issue with taking entire fasta header as the key

This is to address KBASE-3041.  The same code was merged to develop and tested in CI, but since there are many other things in develop, we're patching this to master by itself to fix the issue with the production code.

@scanon, @danielolson5, @kkellerlbl Merging this fix to master so that it can be in the queue for a production deploy.

Please make a jira ticket, assign to me, add Dan/Shane as watchers
--
Keith Keller
Physical Biosciences Division
(sent from mobile)
On Nov 30, 2015 5:22 PM, "Matt Henderson"<EMAIL_ADDRESS>wrote:

@scanon https://github.com/scanon, @danielolson5
https://github.com/danielolson5, @kkellerlbl
https://github.com/kkellerlbl Merging this fix to master so that it can
be in the queue for a production deploy.
—
Reply to this email directly or view it on GitHub
https://github.com/kbase/transform/pull/188#issuecomment-160816557.

Ticket added
KBASE-3113 https://atlassian.kbase.us/browse/KBASE-3113
On Mon, Nov 30, 2015 at 5:38 PM, kkellerlbl<EMAIL_ADDRESS>wrote:

Please make a jira ticket, assign to me, add Dan/Shane as watchers
--
Keith Keller
Physical Biosciences Division
(sent from mobile)
On Nov 30, 2015 5:22 PM, "Matt Henderson"<EMAIL_ADDRESS>wrote:

@scanon https://github.com/scanon, @danielolson5
https://github.com/danielolson5, @kkellerlbl
https://github.com/kkellerlbl Merging this fix to master so that it
can
be in the queue for a production deploy.
—
Reply to this email directly or view it on GitHub
https://github.com/kbase/transform/pull/188#issuecomment-160816557.

—
Reply to this email directly or view it on GitHub
https://github.com/kbase/transform/pull/188#issuecomment-160819287.


fix: install pandoc in the a11y tests

Fix #1575
Just added the pandoc setup to the build yaml

it seems to work now: https://github.com/pydata/pydata-sphinx-theme/actions/runs/7002467262/job/19046293717?pr=1576


User/chheilma/edge for vscode update

Renamed all the elements instances, added new screenshots and explained the new functionality.

Docs Build status updates of commit 9b31167:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

microsoft-edge/visual-studio-code/microsoft-edge-devtools-extension.md
:x:Error

Details

:warning:Warning

Details

microsoft-edge/visual-studio-code/microsoft-edge-devtools-extension.md

Line 18, Column 1: [Error-invalid-image] Invalid attribute.

Line 0, Column 0: [Warning-output-path-conflict] Two or more files output to the same path 'microsoft-edge/visual-studio-code/elements-for-edge.raw.page.json': 'visual-studio-code/elements-for-edge.md [redirection]', 'visual-studio-code/elements-for-edge.md'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit a5d4a43:
:warning: Validation status: warnings

File
Status
Preview URL
Details

:warning:Warning

Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

.vscode/launch.json
:white_check_mark:Succeeded

microsoft-edge/toc.yml
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-headless.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-network.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-settings.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-windowed.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/microsoft-edge-devtools-extension.md
:white_check_mark:Succeeded
View

Line 0, Column 0: [Warning-output-path-conflict] Two or more files output to the same path 'microsoft-edge/visual-studio-code/elements-for-edge.raw.page.json': 'visual-studio-code/elements-for-edge.md [redirection]', 'visual-studio-code/elements-for-edge.md'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 2526ad0:
:warning: Validation status: warnings

File
Status
Preview URL
Details

:warning:Warning

Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

microsoft-edge/toc.yml
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-headless.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-network.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-settings.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-windowed.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/microsoft-edge-devtools-extension.md
:white_check_mark:Succeeded
View

Line 0, Column 0: [Warning-output-path-conflict] Two or more files output to the same path 'microsoft-edge/visual-studio-code/elements-for-edge.raw.page.json': 'visual-studio-code/elements-for-edge.md [redirection]', 'visual-studio-code/elements-for-edge.md'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 2ac102f:
:warning: Validation status: warnings

File
Status
Preview URL
Details

:warning:Warning

Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

microsoft-edge/toc.yml
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-headless.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-network.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-settings.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-windowed.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/microsoft-edge-devtools-extension.md
:white_check_mark:Succeeded
View

Line 0, Column 0: [Warning-output-path-conflict] Two or more files output to the same path 'microsoft-edge/visual-studio-code/elements-for-edge.raw.page.json': 'visual-studio-code/elements-for-edge.md [redirection]', 'visual-studio-code/elements-for-edge.md'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 303220e:
:warning: Validation status: warnings

File
Status
Preview URL
Details

:warning:Warning

Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

microsoft-edge/toc.yml
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-headless.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-network.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-settings.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-windowed.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/microsoft-edge-devtools-extension.md
:white_check_mark:Succeeded
View

Line 0, Column 0: [Warning-output-path-conflict] Two or more files output to the same path 'microsoft-edge/visual-studio-code/elements-for-edge.raw.page.json': 'visual-studio-code/elements-for-edge.md [redirection]', 'visual-studio-code/elements-for-edge.md'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit ce58f99:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

microsoft-edge/toc.yml
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/elements-for-edge.md
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-headless.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-network.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-settings.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/media/edge-devtools-for-vscode-windowed.png
:white_check_mark:Succeeded
View

microsoft-edge/visual-studio-code/microsoft-edge-devtools-extension.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

#sign-off and merged.


Is Device Extension uninitialized?

Please state explicitly whether the allocated device extension is uninitialized or zeroed. (for an example, see https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-exallocatepoolwithtag)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fd3fcb78-63b5-8f9b-cb81-3fef485c3417
Version Independent ID: 6a28639d-1a75-b02c-8e72-93377b2dc3ee
Content: IoCreateDevice function (wdm.h) - Windows drivers
Content Source: wdk-ddi-src/content/wdm/nf-wdm-iocreatedevice.md
Product: windows-hardware
Technology: windows-devices
GitHub Login: @tedhudek
Microsoft Alias: tedhudek

Hi @OZ1, thanks for the question and apologies for the delay.  I've just added the following to the doc page (should be live momentarily):
The device extension of the device object is zeroed.


How to do generic Fourier signals?

We already have the ability go pretty much use whatever spectrum we want through the FourierBasisGP class factory and we can do lots of different selections. However, right now the basis is fixed (i.e. for something like DM we would need a completely new signal factory or at least a lot of copied code).
I've been working on a  few things to make this more flexible and I'd like to know what you think. I've defined a new factory (the name could change):
def Fourier(func, ftype, **func_kwargs):

    class Fourier(object):

        def __init__(self, psr):
            self._psr = psr

        def __call__(self, **kwargs):
            kwargs.update(func_kwargs)
            return selection_func(func)(self._psr, **kwargs)

        @property
        def ftype(self):
            return ftype

    return Fourier

This uses some of the fancier stuff that we are using from the selection, that is, if the underlying funcs arguments are attributes of the pulsar class then it will use them. This way we can still use generic functions without having to specifically write them to read in the Pulsar class (just make the arguments attributes of the Pulsar class).
So this could interact with FourierBasisGP as follows
pl = base.Function(utils.powerlaw, log10_A=parameter.Uniform(-18,-12), gamma=parameter.Uniform(1,7))
basis_dm = Fourier(utils.createfourierdesignmatrix_dm, 'dm')
basis_red = Fourier(utils.createfourierdesignmatrix_red, 'red')
rn = FourierBasisGP(pl, basis_red, selection=selection)
dm = FourierBasisGP(pl, basis_dm, selection=selection)

for DM  and red power-law noise. Basically it would now require a spectrum and a basis. The type part of it is needed so that we could have different kinds. The cool thing is that we can also set arguments on the outside so if we wanted  like  Fourier(utils.createfourierdesignmatrix_red, 'red', fmin=1e-9, fmax=1e-7).
What do you all think?

So @vallis wrote on Slack the other day

I think the best approach is to decide what API we want, then figure out how to implement it. Thinking very generally, maybe we want an abstraction like BasisGP(priorFunction, basisFunction, selectionFunction)

Here basisFunction should return a vector and a matrix (the numerical "labels" of the basis vectors, and a matrix of the basis vectors). It's a good call to have basisFunction behave like Selection, in that it can access pulsar fields directly. But it should also work like a Function (e.g., carry parameters that can be overridden at sampling time, and whose names are specialized).

Can we write an object that does the job of both Function and Selection?

(The rest of our code will need changing because right now it assumes that Fmat never changes it. If we make it parameter dependent, we do need caching of some sort, but only on the parameters that change it.)

I think I have made the tools to do this:
First I have changed selection_func so that it can read in both standard arguments as well as keyword arguments. We may not need this but it is good to be more general.
I've also re-written the Function factory to be more selection-like. With these methods it should be possible to make the BasisGP class factory. I'll get on this soon but I wanted to give you all a heads up first to let me know what you think.
You can find an example notebook here, which should work with the current master branch as all the new functionality is defined in the notebook.

Before I get to Function, let me suggest a small tweak to selection_func. See the comment inline.
def selection_func(func):
    funcargs = inspect.getargspec(func).args

    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        targs = []
        pulsarargs = {}
        for ct, arg in enumerate(args):
            # this could lead to confusing errors if a positional argument
            # is given after a pulsar argument. An alternative could be to take
            # a "psr=..." keyword argument (instead of taking the pulsar as a
            # positional argument), and used that to fill named parameters
            # that are not obtained from *args or **kwargs

            if isinstance(arg, Pulsar):
                pulsarargs.update({funcarg: call_me_maybe(getattr(arg, funcarg))
                                   for funcarg in funcargs[ct:] if hasattr(arg, funcarg)})
            else:
                targs.append(arg)
        pulsarargs.update(**kwargs)
        return func(*targs, **pulsarargs)

    return wrapper

@vallis, that makes more sense. I'll make those changes.

Yup, it's still magic but perhaps less confusing

OK, here's how I would change Function:
# The idea to `Function` is to associated sampling parameters with user-defined functions.

def Function(func, name='', **func_kwargs):
    fname = name

    class Function(object):
        def __init__(self, name, psr=None):
            self._func = selection_func(func)
            self._psr = psr

            self._params = {}
            self._defaults = {}

            # divide keyword parameters into those that are Parameter classes,
            # Parameter instances (useful for global parameters),
            # and something else (which we will assume is a value)
            for kw, arg in func_kwargs.items():
                if isinstance(arg, type) and issubclass(arg, (parameter.Parameter, parameter.ConstantParameter)):
                    # parameter name template
                    # pname_[signalname_][fname_]parname
                    par = arg(name + '_' + ((fname + '_') if fname else '') + kw)
                    self._params[kw] = par
                elif isinstance(arg, (parameter.Parameter, parameter.ConstantParameter)):
                    self._params[kw] = arg
                else:
                    # we could also extract the value from parameter.ConstantParameter and store it here...
                    self._defaults[kw] = arg

        def __call__(self, *args, params={}, **kwargs):
            # order of parameter resolution:
            # - parameter given in kwargs
            # - named sampling parameter in self._params, if given in params dict
            #   or if it has a value
            # - parameter given as constant in Function definition
            # - default value for keyword parameter in func definition

            for kw, arg in func_kwargs.items():
                if kw not in kwargs and kw in self._params:
                    par = self._params[kw]

                    if par.name in params:
                        kwargs[kw] = params[par.name]
                    elif hasattr(par, 'value'):
                        kwargs[kw] = par.value

            for kw, arg in self._defaults:
                if kw not in kwargs:
                    kwargs[kw] = arg

            if self._psr is not None and 'psr' not in kwargs:
                kwargs['psr'] = self._psr

            return self._func(*args, **kwargs)

        @property
        def params(self):
            # if we extract the ConstantParameter value above, we would not
            # need a special case here
            return [par for par in self._params.values() if not
                    isinstance(par, parameter.ConstantParameter)]

    return Function

Then the API becomes
pl = Function(utils.powerlaw, log10_A=parameter.Uniform(-18,-12), gamma=parameter.Uniform(0, 7))

log10_Amp = parameter.Uniform(-10, -5)
log10_Q   = parameter.Uniform(np.log10(30), np.log10(3000))
t0        = parameter.Uniform(psr.toas.min(), psr.toas.max())
fourier_env = Function(createfourierdesignmatrix_env, t0=t0, log10_Amp=log10_Amp, log10_Q=log10_Q)

basis = fourier_env(psr.name, psr=psr)
spectrum = pl(psr.name, psr=psr)

params = {'B1855+09_log10_A': -14,
          'B1855+09_gamma': 4.33,
          'B1855+09_log10_Amp': -8,
          'B1855+09_log10_Q': np.log10(300),
          'B1855+09_t0':<PHONE_NUMBER>.5632496}

Tspan = psr.toas.max() - psr.toas.min()
F, f = basis(params=params, Tspan=Tspan)
psd = spectrum(f, params=params)

...and note this uses the modified selection_func, something like:
def selection_func(func):
    funcargs = inspect.getargspec(func).args

    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        targs = list(args)

        if len(targs) < len(funcargs) and 'psr' in kwargs:
            psr = kwargs['psr']

            for funcarg in funcargs[len(args):]:
                if funcarg not in kwargs and hasattr(psr, funcarg):
                    targs.append(call_me_maybe(getattr(psr, funcarg)))

        if 'psr' in kwargs and 'psr' not in funcargs:
            del kwargs['psr']

        return func(*targs, **kwargs)

    return wrapper

Did you actually run this? I get the error
  File "<ipython-input-157-0a9e4022e932>", line 74
    def __call__(self, *args, params={}, **kwargs):
                                   ^
SyntaxError: invalid syntax

Ah, I used a Python 3 feature there (keyword-only arguments). This should work
        def __call__(self, *args, **kwargs):
            params = kwargs.get('params',{})
            if 'params' in kwargs:
                del kwargs['params']

In either case one needs to call the function by giving params= explicitly, just as we would give psr= explicitly.

Ok I figured it was some Python 3 nonsense :)

Ok I've incorporated your changes and made a first stab at a BasisGP factory in my test notebook. It still needs caching but it at least seems to work for both Fourier and ECORR basis types. Let me know what you think.

Ok, I think I've gotten the caching working and we could probably use this same decorator for caching other things as well, like white noise get_ndiag methods when we fix the white noise. Checkout the notebook.
Here is the caching function
def cache_call(attr, limit=10):

    def cache_decorator(func):
        cache = {}
        cache_list = []

        def wrapper(self, params):
            keys = getattr(self, attr)
            key = tuple([(key, params[key]) for key in keys if key in params])
            if key not in cache:
                cache_list.append(key)
                cache[key] = func(self, params)
                if len(cache_list) > limit:
                    del cache[cache_list.pop(0)]
                return cache[key]
        return wrapper

    return cache_decorator

It takes a string argument to pull out a subset of parameters and it also has a limit so the cache never gets too big.

OK, this cache doesn't work with multiple signals because it defined the cache dictionary per class not per instance. I had to change it like this:
def cache_call(attr, limit=10):
    """Cache function that allows for subsets of parameters to be keyed."""

    def cache_decorator(func):

        def wrapper(self, params):
            keys = getattr(self, attr)
            key = tuple([(key, params[key]) for key in keys if key in params])
            if key not in self._cache:
                self._cache_list.append(key)
                self._cache[key] = func(self, params)
                if len(self._cache_list) > limit:
                    del self._cache[self._cache_list.pop(0)]
                return self._cache[key]
        return wrapper

    return cache_decorator

Now the cache is part of the instance. At the moment it is initialized when __init__ is called but we could initialize it in the cache if it isn't found to begin with. Does anyone know a better way of doing this?

This was taken care of in PR #95; however, now I'm thinking that it isn't general enough. The BasisGP class assumes that the prior on the GP coefficients is diagonal. It may be better if this class would allow for a dense matrix but check to see if it is 1-d or 2-d before inverting so we can save speed.

This is now fully implemented in PR #132.


Automatically derive {Clone,PartialEq,PartialOrd} when {Copy,Eq,Ord} are derived

Rendered

I'm a fan. This feels like an annoying papercut we should fix.

Thanks for writing this up! One thing to note is that as of right now generic types like Option<T> cannot use the short form
impl<T: Clone> Clone for Option<T> {
    fn clone(&self) -> Self { *self }
}

Because it may wrap non-Copy types. We would need to have specialization to get that fast path working.

I imagine that if we introduce specialization, we will add a single impl:
#[low_priority] // or whatever
impl<T: Copy> Clone for T {
    fn clone(&self) -> Self { *self }
}

and then just modify #[derive(Copy)] to no longer generate a Clone impl.
(Same for the other traits.)

Why not "automatically derive supertraits of derived traits" in general? So #[derive(Ord)] (for example) also derives PartialEq, Eq, and PartialOrd, not just the latter.

@glaebhoerl hmm, good point. #[derive(Ord)] probably ought to imply #[derive(Eq)] too.

I spiced up the title to make it clearer what is being requested. @bstrie, hope you don't mind.

@glaebhoerl seems like a great idea

This should fix https://github.com/rust-lang/rfcs/issues/441 if subtypes are all automatically derived. I think that's where @glaebhoerl originally suggested it.

I'm all for it. We still have time for breaking changes now before the final, so if we'd do it, now's the time.

@Kintaro This isn't a breaking change.

@arthurprs it is. See http://internals.rust-lang.org/t/feedback-requested-on-rfc-1028-potential-breaking-change/1847

@arthurprs It is, see this post on by @nikomatsakis on Discourse.
I think the customary approach in cases like this has been to gather some statistics on how much code would actually break? I suspect that it's not very much at all, and that this is worth doing.

Is it not possible to be intelligent enough to do: "Okay, we found a derive(Copy). Let's go look for a manual impl of Clone, and if we find one, use it, otherwise we autogenerate one".
With that logic, would that make this RFC a non-breaking change?

We can't avoid generating the impl entirely, but we could tag it with an attribute that would tell later stages to strip it if it would conflict with another impl.

@diwic It might be possible, and if it is, it should definitely be done. However, it may be that expansion of item decorators happens before the session has collected all trait implementations, in which case it couldn't be done.

I prefer it being explicit. Sure, in this case implicit might be ergonomically better here, but it doesn’t match the style of the language – it is mostly explicit.

@sfackler yes, that's specialization. If we were to add specialization in the future, we could add this backwards compatibilty.

@nikomatsakis I think he means adding a #[implicitly_derived] attribute, and then if there's a conflict in later stages, removing the conflicting implementation that has said attribute.

It wouldn't necessarily be specialization in this case, though, since the manual impl and the derived impl could be defined on the exact same bounds:
#[derive(Copy)]
struct Thing(i32);

impl Clone for Thing {
    fn clone(&self) -> Thing {
        *self
    }
}

So we'd need a separate concept of a "low priority" or "weak" implementation or something that'll be discarded if there's any conflict whatsoever.

I'm torn on this one.  Overall, I do think I'm in favor, especially if we can add specialization in the future.

I think "check if there's already a manual impl, and only derive one if there isn't" versus "always derive an impl, but throw it away if it turns out there's a manual one" is just an implementation detail - the two are observationally equivalent, and there's no need to expose the underlying mechanism to end-users directly. (If I'm not missing something.)

I totally support this. I've always found it annoying how I have to derive both Eq and PartialEq. I don't mind the breakage, especially since very little code will probably be broken (does anyone have statistics on how much would actually break?).

We don't need a breaking change, can't we expand deriving in a non-breaking way? That way we have time to do it during a normal release cycle too and not do it in a rush.
For example, what about #[derive(Ord="full")] to derive the whole shebang, or some similar backwards compatible expansion of the current feature.
I think we need to calmly look at nice generic solutions just like we used to. Always deriving multiple traits is less flexible, and useful in less situations, even though it is easier when it is applicable. Even just giving the trait group a new derive name is a simpler and more broadly useful solution: #[derive(TotalOrdering)].

@seanmonstar @sfackler
The fact that it's a more limited, special-case of specialization doesn't really make me feel better about it, to be honest. I'm definitely opposed to adding some last-minute magic into the trait system without a very strong justification. I want people to be able to understand deriving as simple shorthands for things they could have typed themselves. I don't think the breakage will be that bad if we just do this. In any case, we're trying to gather some numbers based on @brson's tool and PR https://github.com/rust-lang/rust/pull/23905.

In the space of non-breaking solutions, I propose an alias that covers at least Clone, {Partial,}Eq, {Partial,}Ord, Debug, and Hash - i.e. the set of derivable traits that would be useful to implement (especially for users of a library type, or at worst harmless) for most simple structs with a few integer or string fields.  What I'd call 'data-like' (as opposed to 'object-like') structs.

@bluss @comex yes, it's true that shorthand names are a reasonable compromise -- and maybe a better solution. The last time we went through the exercise, we failed to come to much consensus about what those shorthands ought to be, but perhaps we're in a better position now. Things are more stable, and there are a lot more crates we could grep through to try to figure out the most common combinations.

Another option would be to add #[derive_explicit(...)] for cases when you don't want auto-derived base traits.
As for the breaking change, can libsyntax detect common cases like #[derive(Copy, Clone)] in existing code and issue a deprecation warning instead of an error?

@vadimcn #[derive(Copy, Clone)] doesn't break. It's only if the Clone impl is manually implemented, but Copy is derived.

If specialization is planned at some point, I don't mind waiting for it, since that would be a backward compatible change.

Speaking as a user who is generally in favor of inference, I have to say I'm against this. At least so long as manually implementing any trait that is also implicitly derived results in an error. Manually implementing one trait should not bar you from deriving another. If this problem can be solved more generally in the future, then this would be a great addition at that time.

I am for this as long as the specialization gets implemented eventually. If there will never be a way to manually implement PartialEq, for example, and derive Eq, I am against this.

I think having thought this through I'm now feeling that it's not worth making a breaking change for this. Depending how things work out, we can either have some kind of shorthand notation or perhaps we'll be able to rely on specialization (we'd have to see).

I'd rather have a shorthand than this RFC. My code would be broken by it, and I don't see the benefit given that there is a lint (at least for Copy + Clone).

Logically, if I see something that says #[derive(Copy)] or #[derive(Eq)], I would assume it would also imply a partial comparison or clone, so I don't see that as a big problem. And that way, seeing the word Partial would indicate to me that the type doesn't have a full version faster and clearer than if you had to always specify both. And I can see how if specialization comes in, it would be able to be dropped in, so that's nice. So I'd be tentatively in support of this.

What's the point of a breaking change to save a few characters? Why not implement it in a non-breaking way (if you have a Clone implementation, let it be used in the Copy implementation) instead of breaking it?

I'm in favour of this. It's weird that we allow the user to even ask for non-sensical derivations (e.g. "only Ord"). I'm sympathetic to manual implementations becoming a bit more verbose, but those are the exception, and we should optimize for the common case. This change makes Rust "just work" better. Also it allows us to make stronger assumptions about derived implementations: if they derive(Ord) we know exactly how Ord is implemented. If we decide to introduce UnsafeOrd or whatever then we can silently upgrade all derive(Ord)'s to derive(UnsafeOrd)'s (iff they contain only UnsafeOrd's).
The fact that it saves key-strokes is simply gravy, but not the main point IMHO.

I think I prefer the shorthand idea. @aturon suggested something like #[derive(Ord*)], meaning "Ord and all of its supertraits". This is not legal syntax today, but I've been wanting to expand the "grammar" of attributes to permit arbitrary token trees for a while now...anyway, something like that seems like a simple and consistent rule we could easily add later, and it would keep the workings of derive pretty straightforward.

One option I haven't seen mentioned, forgive me if I missed it, would be to allow a user to specify which supertraits they wish to implement manually as part of the syntax.  Something along the lines of #[derive(Copy, -Clone)] would make it possible to explicitly supply a manual implementation of the supertrait by masking off the derived version.  It would remain breaking, but avoids forcing a manual implementation of the subtrait where a supertrait needs specialization.

After much deliberation, we've decided not to pursue this avenue, because it doesn't seem worth making a breaking change -- even a small one. Moreover, this change is not universally popular (e.g., 1, 2, 3) and may be pedagogically challenging, as it breaks the correspondence where deriving a trait is exactly equivalent to implement it (since implementing a trait does not automatically implement its supertraits). The alternative of creating shorthands seems to be a promising alternative, and might also go further in reducing verbosity by permitting groups of common traits to be group together. Therefore, I'm going to close this RFC as postponed under issue https://github.com/rust-lang/rfcs/issues/441. @bstrie, thanks for writing this up and @erickt, thanks for the preliminary implementation. Thanks everybody else for the comments and thoughts.