Fix Cassandra, improve config and tests

Motivation for this change

It's more intuitive and discoverable when you can configure set some options using Nix attributes - plus we can check it using some assertions.
Make it possible to set the variables in cassandra-env.sh.
Allow client connections by default because that's what users expect.
Fix tests and test more things
Make binaries work again (properly wrap them)

The test doesn't work - it doesn't work without this PR. I'll figure out why, when I have time. Seems like a race condition of the VMs starting, it's not a problem with the module itself, rather the tests.
[EDIT]: I think I fixed the tests, please have ofborg run them.
I tested manually on my system that the different configuration options actually have the desired effect.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

This PR now also fixes the tests for Cassandra.
They take 5 minutes to run (on my laptop) but that's fine for booting and shutting down 3 VMs and rebalancing a Cassandra cluster.

This is very necessary. Thanks Daniel.

I broke some of the binaries in #59090...
I misinterpreted the error when testing some of the binaries. I thought I just used it wrong parameters but turns out I broke some of them while making others work.
5041afc9690894f1a809ad6fc fixes it.

Please somebody have a look - Cassandra is currently broken on master.

Hi John,
One delay here is that there is no clear indication of who the cassandra expert is for NixOS. Looking at the git history for both the package and service is a bit sparse. If no one is an expert on cassandra it will take a fair bit of work to review and validate these changes. So the question is... who do we need to get involved for review of this PR before someone with the appropriate access will feel comfortable merging?

@GrahamcOfBorg build cassandra
@GrahamcOfBorg test cassandra

@JohnAZoidberg cassandra is missing from all-tests.nix.

Actually yeah, that makes me think I should add my new config options to the tests so that non-Cassandra people can trust that they do the right thing. (And also they're automatically checked.)

Does anybody know whether I can rely on the IPs staying like they are? Or can I force them to be like I assume?

Does anybody know whether I can rely on the IPs staying like they are? Or can I force them to be like I assume?

Looking through nixos/tests I'm inclined to believe setting a static IP might be a good solution.

I added some more tests that test almost all config options.

@fuzzy-id You rewrote the module - do you have any thoughts?

Looks like openjdk on arch is broken in master :frowning_face:
hash mismatch in fixed-output derivation '/nix/store/pc2krcggyflqbgpz6zlpzxr7hjyfdlif-OpenJDK11U-jre_aarch64_linux_hotspot_11.0.3_7.tar.gz':
  wanted: sha256:0hisjzgv29x2af0wvpy1c9j027qvv08cd9r9f0hg40wdczbbgxaa
  got:    sha256:0r45f358ih5xmygqm91g6nkq9d6nnjl8zz75kl4xbij00svzlcfy

Unrelated to this change, so I'm going to ignore those aarch64-linux errors.

On first sight this looks good and promising. @JohnAZoidberg did you run the tests with all versions of cassandra we currently have in nixpkgs?

No... Good point. It seems that the older versions don't work.
2.1 doesn't work because the password file has the wrong permissions and this version checks that - can we set permissions in the nix-store?
2.2 and 3.0 error with sh: cannot set terminal process group (-1): Inappropriate ioctl for device so I don't think it's the modules fault.

Please don't store passwords in the nix store! And no you cannot set permissions on files in the store, the nix store is always world readable!

Yeah, right - I knew that but I guess I was confused.
I added a warning not to use that on multi-user systems and added an option for providing a custom file.
I still kept it because in many cases you don't care and it's just easier to not have to keep track of a stateful file.
Tests work on all versions that we have in nixpkgs now - I verified that all versions we currently have are the ones supported by Apache.
The other error was unrelated and doesn't seem to cause any problems.

Are you okay with that password config?
Can you please have ofBorg run the test?

@JohnAZoidberg with all the awesome work you've been doing lately you should request access to use the  bot yourself too! https://github.com/NixOS/ofborg

Thanks! Did that yesterday ;)

@jpyen & @fuzzy-id as people with cassandra knowledge do you both approve the changes in this PR? Does everything seem reasonable in the changes? How about the test? Does the nixos test have good coverage? If you can provide in depth review it would be greatly appreciated.
@JohnAZoidberg looking over your (currently) 17 commits the only thing that jumps out is the massive number of options this module has. RFC0042 hasn't been accepted yet, and the reasoning you have for going this route seemed reasonable so the number of options can't really be held against you.
If the tests succeed, we get some reviews by people with cassandra knowledge, and no one else has any arguments against then I think we can merge this. This PR has sat for 24 days already with master being broken so hopefully we can wrap this up in the next few days. Thanks for all your hard work on nixpkgs lately :+1:

Seems like ofBorg timed out after 30 minutes but the tests being run for only about three. On my laptops it takes 6 minutes - after all it starts a cluster with 2 nodes, breaks one and starts a third one to replace it. Cassandra has to do a lot of things.
I'm not sure what to do.
Yeah, I've seen the PR. I haven't thought about it a lot yet.
But I don't think it quite fully applies to Cassandra because here we use two config files, many optional systemd units and lots of environment variables.
I did add some more options which are needed/recommend for a minimal Cassandra cluster - I don't have the intention of adding all of the hundreds of supported options. See: 16dda79 (I should probably add an URL of the docs)
I split it up into different commits on purpose so that it's exactly clear what each of them does. Otherwise any reviewer would be terribly overwhelmed.

I just ran the test locally and had success. I understand your reasoning on the options. You did a great job of making the commits digestible for a reviewer, much appreciated. At this point I just want to hear someone else with cassandra knowledge agree with the changes in this PR and state that they feel the test is representative+adequate.

Yup, start_native_transport is enabled by default so that clients can connect to Cassandra when you .enable = true it. Would be confusing otherwise. See: #50954
What do you think about this diff? Is the doc clearer?
   cassandraConfig = flip recursiveUpdate cfg.extraConfig
     ({ commitlog_sync = "batch";
        commitlog_sync_batch_window_in_ms = 2;
-       start_native_transport = cfg.allowCql;
+       start_native_transport = cfg.allowClients;
        cluster_name = cfg.clusterName;
        listen_interface_prefer_ipv6 = cfg.listenInterfacePreferIPv6;
        partitioner = "org.apache.cassandra.dht.Murmur3Partitioner";
@@ -212,13 +212,16 @@ in {
         Set to <IP_ADDRESS> for a single node cluster.
       '';
     };
-    allowCql = mkOption {
+    allowClients = mkOption {
       type = types.bool;
       default = true;
       description = ''
         Enables or disables the native transport server (CQL binary protocol).
-        This server uses the same address as the rpc_address, but the port it
-        uses is not rpc_port but native_transport_port.
+        This server uses the same address as the <literal>rpcAddress</literal>,
+        but the port it uses is not <literal>rpc_port</literal> but
+        <literal>native_transport_port</literal>. See the official Cassandra
+        docs for more information on these variables and set them using
+        <literal>extraConfig</literal>.
       '';
     };
     extraConfig = mkOption {

Yes, I didn't add them because they're very trivially addable using extraConfig and I want to add only the most important options in Nix.

Not sure about listen_interface_prefer_ipv6 this looks too specific to me.

You're right, it does. But DataStax include it in their "minimal properties needed for configuring a cluster" :shrug:
And a bit IPv6 propaganda can't hurt ;)

Yup, start_native_transport is enabled by default so that clients can connect to Cassandra when you .enable = true it. Would be confusing otherwise. See: #50954

This is strange. I was referring to the Cassandra docs which state that start_native_transport is true by default. Hence this option should be superfluous. BUT I just tested locally and noticed that it is indeed necessary. o_O Looks like this is a documentation bug…

What do you think about this diff? Is the doc clearer?
[…]

Yep. This looks very good to me.

I'm not convinced of the listen_interface_prefer_ipv6 option. It only makes sense if listen_interface is set, too. (But having an addiitonal test for this corner-case doesn't seem right neither.) Also note the docs:

If there is only one address it will be selected regardless of ipv4/ipv6.

Maybe you want to add this to the documentation to produce less surprises. In general, this option seems to have a sensible default, so I don't see the necessity to have it here.
All the rest looks very, very good to me. Thanks a lot! Tests look solid. The module itself looks cleaner. I especially appreciate the addiitonal possibility to configure JVM options.

I'm not convinced of the listen_interface_prefer_ipv6 option. It only makes sense if listen_interface is set, too. (But having an additional test for this corner-case doesn't seem right neither.) Also note the docs:

Yeah okay, I removed it. Like I previously said: We don't want to include every possible config in Nix.

This is strange. I was referring to the Cassandra docs which state that start_native_transport is true by default. Hence this option should be superfluous. BUT I just tested locally and noticed that it is indeed necessary. o_O Looks like this is a documentation bug…

I've wondered too... Yes, seems like the doc is wrong.
Thanks guys for the patience and thorough reviews :)

I'm not convinced of the listen_interface_prefer_ipv6 option. […]

Yeah okay, I removed it. Like I previously said: We don't want to include every possible config in Nix.

Perfect! I think this module makes a good compromise on exposed options now.

Thanks guys for the patience and thorough reviews :)

Dito. Good work!
From my point of view this is ready to get merged.

If either @jpyen or @contrun can weigh in as a third set of eyes/final review I would greatly appreciate.

@JohnAZoidberg This has sat for long enough. We're not getting any more review than we've already had. Do you mind fixing the conflict and then we'll merge? Thanks for your work and patience on this.

Oh, I was responsible for those conflicts :laughing:

Thanks @JohnAZoidberg

The NixOS manual doesn't seem to build with this merged:
# nixos-rebuild switch --builders ''
building Nix...
building the system configuration...
trace: warning: Option `programs.captive-browser.package' has no description.
these derivations will be built:
  /nix/store/d9ihpqcd0d6x5dswjfkwrsiwrv1jd2gk-options-db.xml.drv
  /nix/store/33p1av88g1y974qgjbmxz3a53nsv61vp-generated-docbook.drv
  /nix/store/w7zbd8q8akfdcy47vwplpz8n63kwi61b-nixos-manual-combined.drv
  /nix/store/y8z3qqwv7ivskf4m8rzqiz6jpqrgmk1k-manual-olinkdb.drv
  /nix/store/aslfagf5y3c267g41aj2kr3c9j2z8zvm-nixos-manual-html.drv
  /nix/store/dfsssp62vf9963yqdcq9wwc5axljxki2-nixos-help.drv
  /nix/store/rjpb8pnpk4m30dwbi245fpz4bycqb9x0-nixos-manual.desktop.drv
  /nix/store/yfkra26ay0lc76j06cbypj9dmk4c677l-nixos-manpages.drv
  /nix/store/y3qm2sy4y3czpisxdfn67g8vnpn5ccjl-system-path.drv
  /nix/store/n2k8cpgsmzs61v3jhfq4g1v694q1kyjp-dbus-1.drv
  /nix/store/sr7ywq644wvzcdaqm82mha8kf6nf0rw0-unit-dbus.service.drv
  /nix/store/h9dnqlg1pbkhnwx4040cff2z3bx4ab24-user-units.drv
  /nix/store/lija5f5blhlm19clnvlr8l98zpwh4clj-unit-polkit.service.drv
  /nix/store/p7xypymw1czr1rmkiy65rd8ki2xgy4bi-unit-dbus.service.drv
  /nix/store/wqzj1ycaw5va2iy3ddsjdbxckgb01hxj-unit-systemd-fsck-.service.drv
  /nix/store/wrig2rm34mbwq3sfyz5zpzii5xq04hda-system-units.drv
  /nix/store/w31lswqzbb21hgn4hy5inyq697kd7kw8-etc.drv
  /nix/store/0n3zi29zf8k7p1l46qkxy3qn0yh69qh0-nixos-system-ra-19.09.git.412d110.drv
building '/nix/store/d9ihpqcd0d6x5dswjfkwrsiwrv1jd2gk-options-db.xml.drv'...
intermediate.xml:12476: parser error : Opening and ending tag mismatch: literal line 12471 and para
</para></nixos:option-description><para><emphasis>Type:</emphasis> null or strin
       ^
intermediate.xml:12476: parser error : Opening and ending tag mismatch: literal line 12471 and nixos:option-description
</para></nixos:option-description><para><emphasis>Type:</emphasis> null or strin
                                  ^
intermediate.xml:12480: parser error : Opening and ending tag mismatch: para line 12470 and listitem
            </filename></member></simplelist></listitem></varlistentry><varliste
                                                        ^
intermediate.xml:12480: parser error : Opening and ending tag mismatch: option-description line 12470 and varlistentry
            </filename></member></simplelist></listitem></varlistentry><varliste
                                                                       ^
intermediate.xml:60118: parser error : Opening and ending tag mismatch: listitem line 12470 and variablelist
      </filename></member></simplelist></listitem></varlistentry></variablelist>
                                                                               ^
intermediate.xml:60118: parser error : Opening and ending tag mismatch: varlistentry line 12470 and appendix
lename></member></simplelist></listitem></varlistentry></variablelist></appendix
                                                                               ^
intermediate.xml:60120: parser error : Premature end of data in tag variablelist line 3

^
intermediate.xml:60120: parser error : Premature end of data in tag appendix line 3

^
unable to parse intermediate.xml
builder for '/nix/store/d9ihpqcd0d6x5dswjfkwrsiwrv1jd2gk-options-db.xml.drv' failed with exit code 6
cannot build derivation '/nix/store/33p1av88g1y974qgjbmxz3a53nsv61vp-generated-docbook.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/w7zbd8q8akfdcy47vwplpz8n63kwi61b-nixos-manual-combined.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/yfkra26ay0lc76j06cbypj9dmk4c677l-nixos-manpages.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/aslfagf5y3c267g41aj2kr3c9j2z8zvm-nixos-manual-html.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/y3qm2sy4y3czpisxdfn67g8vnpn5ccjl-system-path.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/0n3zi29zf8k7p1l46qkxy3qn0yh69qh0-nixos-system-ra-19.09.git.412d110.drv': 1 dependencies couldn't be built
error: build of '/nix/store/0n3zi29zf8k7p1l46qkxy3qn0yh69qh0-nixos-system-ra-19.09.git.412d110.drv' failed


Performance with many operations

Hello Jason,
we are planning to migrate our app storage adapter from "js-data-localstorage" to "js-data-localforage" because of some bugs in localStorage implementation on some browsers which we would like to support. But the problem is that we ran into some serious performance issues.
Both adapters in current version save these storage items for a resource:

item with map of all stored ids
item for each model in the resource

Our average storage now contains around 6000 items with total data around 4MB and the possibility is that the amount of items will be raising even more in time. We create/destroy all those data together as user logs in/out of our app (with createMany and destroyMany).
So I made some storage performance testing on target Android v5 tablet (Samsung Galaxy Tab 4 10.1) and here are the results:

IndexedDB 5000 items x 5kB => 151s (!!!)
IndexedDB 5 items x 5000kB => 2,5s
WebSQL 5000 items x 5kB => 111s (!!!)
WebSQL 5 items x 5000kB => 4s
localStorage 1000 items x 5kB => 9s
localStorage 10 items x 500kB => instant

Tests on other devices, like iOS and desktop browsers came basically the same. The time that operations take, drastically rises on number of items, not on their data size.
The conclusion is that the current approach with many storage items is not acceptable under this storage utilization. None of our clients will wait 2 extra minutes just for offline  The problem is in both localstorage and localforage adapters, but is much more affecting the localforage adapter in IndexedDB or WebSQL mode.
So what we should do to get rid of this problem?
I'm thinking of rewriting the whole adapter functionality so there is always 1 storage item for 1 resource. This item would basically contain the same data as current "ids map item" where ids would be substituted by the model data. Here is the example:
Key: some-base-path/resource
Value: {"foo":{"id":"foo","value":true},"bar":{"id":"bar","value":false}}

I'm aware of that performance with single model changes will suffer, because of a need to JSON stringify/parse the whole resource data into the single storage item. But the extra time should be much lower and acceptable as it is solving 2 minutes of waiting on "xxxMany" operations.
Maybe the best solution would be to add something like storageStrategy configuration, so you can choose on each resource if you want to use single or many storage items in adapter operations.
As I'm writing this post, I have another idea for storage organization. Maybe it should work "hybrid" and allow both single and many approach at the same time. For example, for a situation where I had storageStrategy: 'many' on existing resource and then I switch to storageStrategy: 'single' and I don't want to loose the old data. I might look something like:
Key: some-base-path/resource
Value: {"foo":1,"bar":{"id":"bar","value":false}}

Key: some-base-path/resource/foo
Value: {"id":"foo","value":true}

So what do you think of that all? Hope I haven't flooded you with information, but I was trying to explain you all the important ideas and causes.
This issue is very important for our application and we hope you can help us solve it. Would be nice, if you contribute with some code, or at least by letting us implement it and then merge into your adapter production version, so we don't lose further features and fixes. Implementation of this (or any other working) solution is inevitable for us.
Thanks for your interest and developing these superb plugins ;)

Almost 5 months passed without any reply. So this project is not maintained anymore?

It is, it's just lower priority than the 20 other projects I maintain.

I understand your situation. I reacted based on your "Looking for contributors!" appeal on homepage. I thought you will be interested in our help with developing this project. Maybe I hoped for quick evaluation or a few hints based on your knowledge of jsdata ecosystem. Well if you are not interested, then we will develop this feature on our own.

Your storageStrategy idea looks like it could work, I'd be happy to review your pull requests.

@radarfox are you still working on this? I would be interested in this - and revamping the localforage adapter to work with js-data v3. so we could join forces..


LabBreakers, Github Actions

Made those changes to the hosts files
Added another machine for the major version upgrade lab module

I know it looks like it failed the Build with Nuitka action, but that's just because the files were the same, so it didn't need to commit anything new. The actual build part and copying worked.
I probably will just build on my forked repo though, and then pull request to main to keep it simple, but we just wanted to test this working here as well.


WorldQuery derive State and Fetch types are still accidentally nameable

Bevy version
Latest commit as of writing (7d9cb1c)
What you did
#[derive(WorldQuery)]
pub struct Client<S: ClientState> {
    pub state: &'static S,
    // ...
}

pub trait ClientState: Component {}

What went wrong
error[E0404]: expected trait, found struct `ClientState`
  --> crates/example/src/client.rs:11:22
   |
11 | pub struct Client<S: ClientState> {
   |                      ^^^^^^^^^^^ not a trait
...
35 | pub trait ClientState: Component {
   |           ----------- you might have meant to refer to this trait
   |
help: consider importing this trait instead
   |
1  | use crate::client::ClientState;
   |

Additional information
To workaround it currently, we can prepend self:: to the struct bound:
#[derive(WorldQuery)]
pub struct Client<S: self::ClientState> {
    pub state: &'static S,
    // ...
}

Interesting, I'll look into this. Not sure if there's a clean way of solving this, but it should at least be possible to check for this footgun and emit a descriptive error message.

Similar: #1727.
I should have a fix for both of these up pretty soon.


Use native Quarkus Websockets

Enable atmosphere to use Quarkus websockets, as they are not jsr356 compliant anymore since 10.whatever version of Quarkus.
This results in Devmode Gizmo not working (it depends on websockets for live reload) and Push falling back to long polling.
Actual behavior
Atmosphere complains and says there is no websocket implementation available
Expected behavior
Atmosphere uses Quarkus native websockets
Hint
Add a new Quarkus specific class to https://github.com/vaadin/atmosphere/tree/master/modules/runtime/src/main/java/org/atmosphere/container

Quarkus documentation states that "The Quarkus WebSocket implementation is an implementation of Jakarta Websockets", so Atmosphere should work with JSR356Endpoint; maybe something has changed in the meanwhile.
Currently there are two issues with PUSH on Quarkus:

Quarkus websocket feature is enabled only if classes related to websocket are found; in detail at least a class implementing javax.websocket.server.ServerApplicationConfig, extending javax.websocket.Endpoint or annotated with javax.websocket.server.ServerEndpoint.
More details on https://github.com/urosporo/vaadin-quarkus-extension-parent/pull/11 and related issue
The Vert.x handler that processes websocket ugrade request has no knowledge about Undertow HTTP session, so Flow PushHandler always sends "Session expired" notification and the browser tried to reconnect again and again with no success.

As a temporary solution we can add to the extension a class to enable websocket deployment and a build step that will produce a build item with a modified version of VertxWebSocketHandler that handles only Vaadin PUSH request and tries to attach the HTTP session to the exchange of the upgrade request.


add missing enable_addons reference in gke blueprint for multitenant-…

Small fix for the GKE blueprint multitenant-fleet adding the missing enable_addons call to the underlying gke-cluster module call.

hi @jackspyder, thanks for catching this. Can you sign the CLA so we can merge this?

Hey @juliocc I've signed a corporate CLA, i'm the group owner, its publicly visible, my commit is with my corporate email which is also added to my github account (though not my primary mail). It was signed last night so maybe takes some time to show up? Sorry if im wasting your time on such a small change.

Hey @jackspyder, unfortunately the CLA is still showing as failed. Can you double check once again?

@juliocc finally the CLA is signed! So sorry about the delay for such a small change!

Awesome, thanks @jackspyder. Now you have to rebase you branch wi master so we can merge it.


Game switching from XBOX to PS3 emulation

Hi. I'm using this great tool for quite long time and till today there was no issues... So I installed Horizon Chase Turbo from yesterdays EPIC giveaway and gamepad keep switching from Xbox layout to PS3 layout. I cannot fix it. In options I tried to set XInput but seconds later after press button on gamepad I see PS3 is emulated. My gamepad is noname (kind of clone of PS3). Please help!

Try pressing "Add as affected device to hidguardian" in edit window and do not forget to update settings.json by pressing save congiguration in file menu.


Course Schedule

Enter your question -
There are N courses and they are labeled from 0 to N-1. You are also given a list of pairs (X, Y), indicates, you must take a course Y before X.
You need to answer if it is possible to finish all courses or not.
Enter link to the question(if question belongs to any online platform) -
https://leetcode.com/problems/course-schedule/
Tags for the question(eg - Array, Basic, Stack, etc.) -
Topological Sorting, Cycle detection in graph.

@ravi0213,
Kindly add your solution to "leetcode" folder.
Deadline - 07/10/2021


Antialiasing

Is antialiasing supported on OS X on Monaco Font?

It should be. Are you experiencing issues with font rendering? If yes, I would greatly appreciate a screen capture of the issue so I can diagnose it.

Actually what I meant is the possibility to disable the antialiasing in Atom. Please find attach my Mac Terminal prompt and the corresponding one in Atom. Both are in Monaco 10pt.

I see now what you are saying. I'll look into this and get back to you.


Each manga in Library needs to be migrated from old extensions to the new ones otherwise new chapter check fails

Describe why this should be added
Each manga in Library needs to be migrated from old extensions to the new ones otherwise new chapter check fails. Old ones show as obsolete anyways.
Just selecting the manga, clicking the migrate button, followed by search select same name in the results should be enough.
Maybe this info can be added as a note in README.
This is not a request per se but something I noticed.
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] If this is an issue with the app itself, I should be opening an issue in the app repository.
[X] I have updated the app to version 0.14.7.
[X] I have updated all installed extensions.
[X] I will fill out all of the requested information in this form.

i used tachiyomi 0.15.2. and i used the old tachiyomi extension which i already install. set it to trusted. and they worked fine without having migrating them

you can delete the old extensions and then the new ones will replace them automatically


Problem with Hibernate Typedefs

I've been using a Typedef Defintition together with Jasypt
to store Encrypted Database fields.
See the Typedef below.
While this works in JVM Mode and within Quarkus without further Hints.
It crashes in Spring Native Mode:
-- cut --
Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Invocation of init method failed;
nested exception is org.hibernate.boot.registry.classloading.spi.ClassLoadingException: Unable to load class [encryptedString]
Debugging into the EntityManagerFactory shows that it's indeed constructed via Reflection.
However a Typehint for EncryptedString does not solve the Problem
@TypeHint(types = {org.jasypt.hibernate5.type.EncryptedStringType.class}, access = AccessBits.ALL)
It doesn't matter if AccessBits.ALL is set or not ..
I see spring is complaining specifically about the "encryptedString" type .. but I don't know how to declare this ..
Any hints would be appreciated.
-- cut package-info.java---
@TypeDefs
({
@TypeDef(
name="encryptedString",
typeClass= EncryptedStringType.class,
parameters= {
@Parameter(name="encryptorRegisteredName", value="hibernateStringEncryptor")
}
)
})

Ok, i think i at least found the source of the problem ..
Interesting enough when compiling in native mode without building the image with paketo and not even having graalvm installed, it is possible to track down the error during integrations tests .. with fast build times.
So internally the SimpleValuebinder just does "classForname" with "encryptedString" .. which is the TypeDef Name and NOT the Type Class ...
Of course this cannot work .. having a suspicion .. i removed the package-info.java alltogether and restarting the application in JVM mode .. same error ..
So i guess basically this means that the package-info.java is with the TypeDefs gets ignored or removed ...

ok .. i could solve it by annotating the business object with all informations
and not using the package-info.json
@Type(type="org.jasypt.hibernate5.type.EncryptedStringType", parameters = @org.hibernate.annotations.Parameter(name="encryptorRegisteredName", value="hibernateStringEncryptor"))
private String secret;

I leave it up to you guys if you want to fix the problem with the package-info.java or not ...
The workaround is now documented here.


Hide neomerx package behind our own interface

So the plan for 2.0 is to hide the neomerx/json-api package behind our own encoder interface. Ideally we should not use anything in that package except encoding, and it should be an internal thing of our encoder interface.
If we create our encoder via a factory, we can then support multiple versions of neomerx/json-api. I.e. detect which one is installed and use the right wrapper encoder.
The reason for doing this is it means we become more immune to changes in that package, some of which can have quite significant impacts. However we do want to keep using that package because the encoding it does is very well implemented and the package has a great focus on speed and efficiency.
Ideally our new approach should align with the Eloquent resource implementation that is native in Laravel. That will be more familiar to Laravel developers. I.e. the JSON API resource Schema class becomes one of those (but retain the schema name as resource is a reserved word in PHP).
However to reduce upgrade burden our encoder should support the resource schema being a neomerx schema. I.e. so there's no requirement to upgrade existing schemas immediately.

Completed in the new package, see https://laraveljsonapi.io


Missing etcher-CLI examples

CLI examples should be added to:
https://github.com/resin-io/etcher/blob/master/docs/CLI.md

Is this just a generic request for documentation, or is there also some part of the Etcher CLI that you need help with?

It's not just for me.
I mean examples like in the following link:
https://www.bountysource.com/issues/45325226-etcher-cli-help-output-is-misleading
How to set the source file and destination drive, with or without verification, unmount or not when flashing finished.

Haha, that link you posted there is actually just a copy of #1452 ;-)
We've been focusing most of our efforts on the Etcher GUI so far, as that has much greater usage, but we'll try to add some polish (and documentation) to the Etcher CLI at some point soon.


no module named HtmlTestRunner

HtmlTestRunner version:
Python version:
Operating System:

Description
im getting above error in the code , when i use the import statement
What I Did
import HtmlTestRunner
pip install html-testRunner

Hi, I create an environment in python 2.7 and 3.5, install the library, import it and worked.

can you please share the sample class/program

I use the same that is in the readme. Do this:

Create a virtualenv and activate it.
Run pip install html-testRunner
Then copy the example in the readme, save it to a file test.py and run it python test.py


Use  unwatch between multi and exec, and throw NullPointException when executing Exec

Expected behavior
Unwatch can be exec before execution
Actual behavior
Use  unwatch between multi and exec, and throw NullPointException when executing Exec
Steps to reproduce:
    jedis.watch("key");
    Transaction transaction = jedis.multi();
    jedis.unwatch();
    transaction.set("key","value");
    transaction.exec();

Jedis version:
2.9.0
Java vcersion:
1.7

Resolved by #2031 and #2032


Generated files path

One nice option would be to specify generated files path.
For example, you have backend area and want your view to be in backend/clients section, not only in /clients section.
Also folder/namespace for controllers and models.

That is supported currently you can simply use an underscore to create the separation Book_Author https://laracogs.com/docs/services/crud/#snake-name-example-book_author


ios app integration causes memory overload (0.11 ios integration, 0.58 UI)

Every day or so my watch will crash and show a New interpreter error: LOW_MEMORY,MEMORY error in red text on the bottom of the screen, followed by all watch functions failing until I restart the watch. I've tracked it down to the iOS integration, because when I do a memory dump from the watch most of it is this message:
Bangle.ancsMessageQueue = [
  {
    "event": "add",
    "uid": 18, "category": 0, "categoryCnt": 18, "silent": true, "important": false,
    "preExisting": true, "positive": false, "negative": true },
  {
    "event": "add",
    "uid": 19, "category": 0, "categoryCnt": 19, "silent": true, "important": false,
    "preExisting": true, "positive": false, "negative": true },
  {
    "event": "add",
    "uid": 20, "category": 0, "categoryCnt": 20, "silent": true, "important": false,
    "preExisting": true, "positive": false, "negative": true },
  {
    "event": "add",
    "uid": 21, "category": 0, "categoryCnt": 21, "silent": true, "important": false,
    "preExisting": true, "positive": false, "negative": true },
  {
    "event": "add",
    "uid": 22, "category": 0, "categoryCnt": 22, "silent": true, "important": false,
    "preExisting": true, "positive": false, "negative": true },
  {
    "event": "add",
    "uid": 23, "category": 0, "categoryCnt": 23, "silent": true, "important": false,
    "preExisting": true, "positive": false, "negative": true },
  {
and so on and so forth for over 1500 lines, taking up about 80% of the memory dump.

I'm assuming these are past notifications, and also that they must be iOS related because of the reference to ANCS, and I'm not sure why there's so many of them. Maybe the iOS integration isn't clearing out old notifications? I assume a fix would involve erasing these events when new ones come in, or erasing old events every few hours. Also, googling this issue yields zero hits and no useful information.

Thanks for tracking this down! It's an odd one - it looks like maybe NRF.ancsGetNotificationInfo can maybe end up never returning sometimes... Do you still get messages in that state, or were they not arriving?
I guess if the iPhone disconnected while ancsGetNotificationInfo was being called, that would do it.
I've just made some changes to the iOS app (in the development app loader) which should help, and I also pushed a new cutting edge Bangle.js firmware that should at least try and 'reject' the promise returned by ancsGetNotificationInfo if there are any problems.
Maybe you could give those a try and see if they help?


use icon element types instead of strings internally

Overview
This PR brings all of our internal components (with the exception of the Alert/InlineAlert) to pass React.ElementType to icon-receiving props instead of string literals.
It also updates support so consumers can pass icons like icon={CogIcon} in addition to icon={<CogIcon />}
I updated the codemod (and tests!) to automate this migration.
Screenshots (if applicable)

Testing

[x] Updated Typescript types and/or component PropTypes
[x] Added / modified component docs
[x] Added / modified Storybook stories

Some docs around how to run this would be great!

@akleiner2 it's documented in the codemod file itself how to run it.


RAT-367: Enable running old junit tests during the build with junit5

This fixes https://issues.apache.org/jira/browse/RAT-367 and now also the older tests are run on each build.

Thanks for the catch, I'll add a changelog entry accordingly.


fix: quoting database name

Description

This fixes the issue we were having when database name had mixed-case letters. Quoting ensures the casing is preserved.
TESTs
Run both jaffle shop and pytest tests. Confirmed users can override the value in dbt-project.yml
Checklist

[x] I have run this code in development and it appears to resolve the stated issue.
[x] This PR includes tests, or tests are not required/relevant for this PR.
[x] I have updated CHANGELOG.md and added information about my change.
[x] I have removed any print or log calls that were added for debugging.
[x] If this PR requires a new PyPI release I have bumped the version number.
[x] I have verified that this PR contains only code changes relevant to this PR.
[x] If further integration tests are now passing I've edited firebolt.dbtspec to account for this.
[x] I have pulled/merged from the main branch if there are merge conflicts.
[x] After pulling/merging main I have run pytest on the included or updated firebolt.dbtspec.

mypy issue in CI is due to specifically python=3.7 and dbt-core=1.3.0, I filed a separate ticket to investigate https://packboard.atlassian.net/browse/FIR-16420


Added check-suite checking

This was left over from the port of the other Dockerfile, it isn't needed here.

Wait no - we might need it to check if the check-suite has passed.

Now it checks to see if the check-suite is passing or not

Don't we want to report failures? Also let's move all the logic to hooks.sh.

Yes, good idea.  My thoughts is we use the other action to check if the suite is passing or not, and if it is, run bash ./hooks.sh did-not-fail

Alright I think I fixed it all up, should now be ready for review.

What are you trying to achieve here? It's not clear from the PR title or description in the first comment.

I'm trying to make it so it can check if the suite passed or not

Removed it and squashed.


Add CC RawOrder and Order models

Context
The device supplier provided some raw order data that we wish to show to settings
Changes proposed in this pull request
Add the ability to import from CSV to the RawOrder model and covert this to usable Order objects.
Guidance to review
In local dev run the rake tasks:
Ingest
ORDERS_FILE_PATH=/local/path/to/csv/file bundle exec rails import_orders:ingest

Convert
bundle exec rails import_orders:convert

 Order data ingestion


fix(toggleterm): start_in_insert not working

IMO this is a good patch (taken from #913) and should be merged.

I initially assumed this to be an issue with the nightly version, but that's not the case. The start_in_insert doesn't work because persist_mode is set to true. persist_mode retains the mode from when you last exited toggleterm.nvim. If you exit the terminal using wincmd, then you would reenter in insert mode. In my case, I exit the terminal by pressing q after double-tapping . Consequently, toggleterm remembers that my previous mode was normal, and it sets the mode back to normal when I reenter toggleterm.
toggleterm.nvim Notice: "If using persist_mode terminal mappings should be changed to use wincmd instead otherwise persist mode will not work correctly."


Nerve has incorrect annotation

This is on the naz_changes branch.

@NEbrahimi ??

Closing as fixed by commit 47f5b84.


"This field may not be null." for models.CharField declared with blank=True

There is an inconsistency of client- and server-side validation for CharField fields with blank=True.
The generated templates accept such an object, but the synchronization fails. The outbox synchronization process sends these fields as "", but the server responds with "400 Bad Request" with a JSON structure containing "This field may not be null." messages.

Yes, this is a bug in wq.db's ModelSerializer, which converts all incoming "" to None before running the validation.  A temporary workaround would be to create a custom ModelSerializer that restores the blank value:
# myapp/serializers.py
from wq.db.rest.serializers import ModelSerializer

class MyModelSerializer(ModelSerializer):
    def to_internal_value(self, data):
        result = super().to_internal_value(data)
        if result.get('fieldname', None) is None:
            result['fieldname'] = ""
        return result

# myapp/rest.py
from wq.db import rest
from .serializers import MyModelSerializer
from .models import MyModel

rest.router.register_model(
    MyModel,
    serializer=MyModelSerializer,
)

The longer term fix is a bit more complicated.  The issue narrows down to the fact that wq.db's ModelSerializer uses the HTML JSON Forms parser to handle potentially nested records.  This processing happens early, so that by the time the validators run, the data is treated as if it had been submitted via a JSON API.  Since JSON differentiates between null and "", but HTML does not, I had to make an arbitrary decision how to handle empty strings when converting from an HTML Form submission to JSON (see this function).  I chose to treat all empty strings as null, since that was the more common interpretation in my usage.  The downside is that blank=True doesn't work at all.
Django REST Framework itself does not suffer from this constraint, because it has access to both the field properties and the structure of the input data at the same time (see this function).  When is_html_input() is run on a dictionary that has already been processed by HTML JSON forms, it returns False, so the allow_blank handling code doesn't run.
If you aren't using nested serializers you technically don't need HTML JSON Forms at all, though I don't currently have an easy way to disable the functionality.

Another workaround: I noticed that if I set value of these non-nullable text fields to " " (a space), they are stored in the database as empty strings (without conversion to NULL).


[AAP-5025] Audit view rule details page

Audit view rule details page

@AlexSCorey - this PR is still a draft, was not yet submitted for review. Will make the requested changes.TY!

@AlexSCorey  - thank you fr the review, it is appreciated. It was a draft PR at the time of your review, though. I always remove all the console logs before marking the PR as ready for review.
I addressed most of the comments above.


[Story] Receive email confirmation of successful submission

Epic
Epic issue includes overview, complete list of user stories, technical and content requirements, user testing task ideation, and current design mock-ups.

Sign and Submit

User story

As a filer, I want to receive email confirmation that my submission was successful, so that I know the filing process is complete.

Acceptance criteria

Receive an email with confirmation ID and platform specific language ( submit ID, timestamp, beta language)

Identify details of alert message:

Who sends: noreply@cfpb
Subject: [BETA]......
Who receives the alert? Which contacts are main line vs. CC. The main submitter, associated accounts, primary contact
Primarily select initial submitter and then request who else received the alert via forwarding
Body: Basic email text

Tasks
Team members should create tasks and add links (below) prior to sprint planning.
UX

[x] Write initial user story

Review against task analysis for accuracy and relevance
Add a link to the story from the task analysis

[x] Validate user story with the team

Review for specificity and completeness

Content

[x] https://github.com/cfpb/sbl-ux/issues/23

Back end

[x] https://github.com/cfpb/regtech-mail-api/issues/45
[x] https://github.com/cfpb/regtech-mail-api/issues/52
[x] https://github.com/cfpb/sbl-filing-api/issues/495

title:
HMDA filing accepted!
body:
HMDA filing accepted!
Congratulations, you have successfully completed your HMDA filing for 2022!
Your data and signature were received and recorded on {timestamp}. Your receipt number for this submission is {lei-year-quarter-timestamp}

The following reflects what we implemented on beta (and what we plan to implement on production when the time comes):
Beta implementation: Alert and Email

Alert
Email

Production implementation: Alert and Email

Alert
Email


Videos are not deleted per "Number of Days to keep Videos" setting

Expected behavior

Videos should be deleted if older than "Number of Days to keep Videos".

Actual behavior

All videos are retained regardless of "Number of Days to keep Videos" setting.

Steps to reproduce

Set "Number of Days to keep Videos" to any integer.  Videos are not erased once the days have surpassed.

The same issue is happening to me. I was about to open this issue also.
@frenck
Can you take a look into it? Thanks

Hi,
The issue stills there. How can I help to solve this?

@ascillato You can't...
https://community.home-assistant.io/t/community-hass-io-add-on-shinobi-pro/49767/98?u=frenck

This add-on has been deprecated some time ago already and has now been removed from the add-on store.
If you are looking for an alternative: https://github.com/hassio-addons/addon-motioneye


Undo/Redo buttons do not update word count

Describe the bug
The word count is not updated after using the Undo and Redo buttons.
To Reproduce
Steps to reproduce the behavior:

Open or create a scene with some text in it.
Select and delete a few words. The word count decreases.
Press Undo button.
Observe that the word count does not go back.
Start typing again: the word count gets updated and is accurate.

Expected behavior
The word count should be updated (immediately or with a delay) after pressing Undo or Redo buttons.

Seemed like they didnt even change any scenes in the store. Got fixed with https://github.com/TorstenDittmann/OmniaWrite/commit/d1d3bb55ff665a3a4375752ce9f1cad3dc323b53
Nice catch 🥳


Update README.md

Expanded readme to add more information on how GenderMag Recorders Assistant reduces gender disparities in software.

name: Pull request template
about: GenderMag Recorders Assistant project

What does this implementation fix? Explain your changes.

Updating the readme makes it easier for contributors to understand the how of the tool in addition to the what (understanding how the tool reduces gender disparities vs. just what it accomplishes).

Does this close any open issue? (Give issue id from issue tracker)

No

Include any related logs, error, output file etc.

N/A

Did you test on Mac and Windows?

N/A

Did you test the full GM process and verify that the code does not contribute any unexpected errors/bugs?

N/A

Did you follow the style guidlines and verify that the code does not have any incorrect formatting?

Yes

If possible, provide a screenshot of the functionality.

N/A

Any other information?

None

I think this is a great change! One nit-picky thing about this wording: "there are some gender differences on where men and women are more likely to fall on the spectrums" When we describe the facets and gender, we try to avoid language that just talks about "men and women" as gender is also a spectrum. An alternative description might be: "facets statistically cluster around genders; for example, Abi's facet values are those frequently seen in women"

I think this is a great change! One nit-picky thing about this wording: "there are some gender differences on where men and women are more likely to fall on the spectrums" When we describe the facets and gender, we try to avoid language that just talks about "men and women" as gender is also a spectrum. An alternative description might be: "facets statistically cluster around genders; for example, Abi's facet values are those frequently seen in women"

Thank you for the suggestion! That makes a lot of sense to change the wording to reflect gender as a spectrum. I can make that change.


Modify GxIT URL for eu subdomains

Hi eu team!
Hope all is ok for you. Is it possible to modify a GxIT URL like https://c0f1771e21d7ee60-83b672c1b14a476180d38bdedbe5360c.interactivetoolentrypoint.interactivetool.ecology.usegalaxy.eu/sample-apps/SIG/ coming from subdomains to point for example https://c0f1771e21d7ee60-83b672c1b14a476180d38bdedbe5360c.interactivetoolentrypoint.interactivetool.usegalaxy.eu/sample-apps/SIG/ non subdomains oriented URL ? This will allow direct display of GxIT content instead of having an intermediate step related to security and https

The better solution is to use the path-based proxying that was added recently in Galaxy (giving more parity with the original GIEs).
If you can get your IT working with path-based proxying then, it won't be an issue, and it's easier for admins everywhere :) Currently the DNS based proxying is more or less impossible for most admins :(

whowho, thank you Helena, so if I understand, we can manage it on the GxIT side ? Do you have an example just to be sure to understand ?
THANK YOU §!!! And have a wonderfull day !!

No, unfortunately dan didn't provide any examples or documentation :(
https://github.com/galaxyproject/galaxy/pull/8596/files is the PR, I think you need to set requires_domain=False in your XML to start.
the interactivetools_prefix needs to be set (doesn't much matter)
entry_point_prefix = self.app.config.interactivetools_prefix

then it will generate a URL like:
rval = self.app.url_for(f'/{entry_point_prefix}/access/{entry_point_class}/{entry_point_encoded_id}/{entry_point.token}/')

but I'm not sure how this is communicated to the container, or if it is. If it is not, we need to do so, and then the container needs to know how to proxy this (some examples are around, I can help here.)
If you can test the first part (set requires_domain=false) and provide to me the environment variables passed into the container, I can help update your container.


MdDialog error when opened onInit

Bug, feature request, or proposal:
Error in MdDialogContainer_Host.html, Line 1

Im calling my openModal function on my ngOnInit, just for testing purposes, and I got this error.
ERROR Error: ExpressionChangedAfterItHasBeenCheckedError: Expression has changed after it was checked. Previous value: 'undefined'. Current value: 'dialog'. It seems like the view has been created after its parent and its children have been dirty checked. Has it been created in a change detection hook ?
What is the expected behavior?
Remove the error
What is the use-case or motivation for changing an existing behavior?
I could have a modal opened onInit in the future in one of my side projects, I will be a pain if there are errors doing this
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
<EMAIL_ADDRESS>@angular/cli: 1.3.1
os: win32 x64
Is there anything else we should know?
This is my first time doing this and I have no idea that there are errors if you will do this onInit, although i did this for testing purposes i...

This is already being tracked in #5268. Take a look over there. The using of setTimeout is a possible a workaround for you.

@julianobrasil I see thank you

@julianobrasil I see thanks

Closing as a duplicate of #5268.


Divider Component Bug

Problem description
Sometimes the Divider Component display a 2px thickness, when no custom styles have been applied.
Steps to reproduce
         <List>
               <ListItem disabled={true}>

Trade Miles

{tradeInfo.miles}

                </ListItem>
                <Divider/>
                <ListItem disabled={true}>

Trade Payments

{tradeInfo.payment}

                </ListItem>
                <Divider/>
                <ListItem disabled={true}>

Trade Payoff

{tradeInfo.payoff}

                </ListItem>
                <Divider/>
                <ListItem disabled={true}>

Down Payment

0

                </ListItem>
                <Divider/>
                <ListItem disabled={true}>

Must Haves

0

                </ListItem>
            </List>

Versions

Material-UI: v0.15.4
React: V15.3.1
Browser: Chrome

@mbusta305 That looks like an issue with a manual screen zoom. Are you at the right scale (100%)?

I am at the correct scale, maybe I wasn't earlier but I have not zoomed in or out all day but it is working now. I Guess I can close this but this isn't the first time I've had this happen but then again I did not check if I was at the right scale then either.

Duplicate of #14815.


Add a Hypothesis plugin

This patch adds a plugin which teaches Hypothesis how to generated examples of Pydantic's custom field types (which closes #2017).  Note that there is no runtime impact or dependency; this module is imported by Hypothesis so it only exists at test-time and only when Hypothesis is both installed and actually being used.

[x] Unit tests for the changes exist
[x] Tests pass on CI and coverage remains at 100%
[x] Documentation reflects the changes where applicable
[x] changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)

I've ended up dropping the integration for URLs, and for constrained lists and sets... but everything else is ready to go!

Why not support constrained lists and sets?
In short, I couldn't get them to play nicely with Hypothesis' type registry - because the runtime objects are subclasses of a parametrized generic type, and aren't always distinguished by our introspection logic 😭

Why I decided to leave our URLs (with code)
I spent a long time tweaking support for URLs, but ultimately gave it up: it's easy to generate valid URLs, unless you want to generally really strange ones which will find lots of bugs.  I decided that it was better to make users register their own - and make the tradeoff explicit - than to cover up the complexity and have automatic but weak tests.  This is a standard design choice for Hypothesis, sadly, so I've included my code so far below for the benefit of any future contributor who wants to pick it up.
def idna_encodable(s: str) -> bool:
    # We only need this because the regex patterns aren't fully precise; but
    # rejection sampling is a LOT easier to implement than precise patterns.
    try:
        s.encode('idna')
    except Exception:
        hypothesis.reject()  # type: ignore[no-untyped-call]
    return True

@resolves(pydantic.AnyUrl)
def resolve_anyurl(cls):  # type: ignore[no-untyped-def]
    domains = st.one_of(
        st.from_regex(ascii_domain_regex(), fullmatch=True),
        st.from_regex(int_domain_regex(), fullmatch=True).filter(idna_encodable),
    )
    if cls.tld_required:

        def has_tld(s: str) -> bool:
            assert isinstance(s, str)
            match = ascii_domain_regex().fullmatch(s) or int_domain_regex().fullmatch(s)
            return bool(match and match.group('tld'))

        hosts = domains.filter(has_tld)
    else:
        hosts = domains | st.from_regex(
            r'(?P<ipv4>(?:\d{1,3}\.){3}\d{1,3})' r'|(?P<ipv6>\[[A-F0-9]*:[A-F0-9:]+\])',
            fullmatch=True,
        )

    return st.builds(
        cls.build,
        scheme=(
            st.sampled_from(sorted(cls.allowed_schemes))
            if cls.allowed_schemes
            else st.from_regex(r'(?P<scheme>[a-z][a-z0-9+\-.]+)', fullmatch=True)
        ).filter(idna_encodable),
        user=st.one_of(
            st.nothing() if cls.user_required else st.none(),
            st.from_regex(r'(?P<user>[^\s:/]+)', fullmatch=True).filter(idna_encodable),
        ),
        password=st.none() | st.from_regex(r'(?P<password>[^\s/]*)', fullmatch=True).filter(idna_encodable),
        host=hosts,
        port=st.none() | st.integers(0, 2 ** 16 - 1).map(str),
        path=st.none() | st.from_regex(r'(?P<path>/[^\s?]*)', fullmatch=True).filter(idna_encodable),
        query=st.none() | st.from_regex(r'(?P<query>[^\s#]+)', fullmatch=True).filter(idna_encodable),
        fragment=st.none() | st.from_regex(r'(?P<fragment>\S+)', fullmatch=True).filter(idna_encodable),
    ).filter(lambda url: cls.min_length <= len(url) <= cls.max_length)

st.register_type_strategy(pydantic.AnyUrl, resolve_anyurl)
st.register_type_strategy(pydantic.AnyHttpUrl, resolve_anyurl)
st.register_type_strategy(pydantic.HttpUrl, resolve_anyurl)
st.register_type_strategy(pydantic.PostgresDsn, resolve_anyurl)
st.register_type_strategy(pydantic.RedisDsn, resolve_anyurl)

def gen_url_models():
    class AnyUrlModel(pydantic.BaseModel):
        anyurl: pydantic.AnyUrl

    class AnyHttpUrlModel(pydantic.BaseModel):
        anyhttp: pydantic.AnyHttpUrl

    class HttpUrlModel(pydantic.BaseModel):
        http: pydantic.HttpUrl

    class PostgresDsnModel(pydantic.BaseModel):
        postgres: pydantic.PostgresDsn

    class RedisDsnModel(pydantic.BaseModel):
        redis: pydantic.RedisDsn

    yield from (AnyUrlModel, AnyHttpUrlModel, HttpUrlModel, PostgresDsnModel, RedisDsnModel)

@pytest.mark.parametrize('model', gen_url_models())
@settings(suppress_health_check=[HealthCheck.filter_too_much, HealthCheck.too_slow])
@given(data=st.data())
def test_can_construct_urls_model(data, model):
    # This is a separate test because we want a minimal health-check exemption
    instance = data.draw(st.from_type(model))
    assert isinstance(instance, model)

`Literal[None]` requires Python 3.7+ or a recent version of Hypothesis
Because I literally just fixed that this evening - since it requires a backport on a security-only Python version, we delayed working out how to support it until we knew it would actually be used before the 3.6 EOL later this year.

Amazing work @Zac-HD! So glad to see this PR open!
Could you please check the coverage to make sure it remains at 100% please?

@PrettyWood - thanks!  I'm looking forward to seeing what people do with it :smiley:
There was one deliberately unreachable line (assertion that we returned from a loop body), which I've unrolled so we get 100% coverage without needing lots of pragmas.

OK @samuelcolvin - I think I'm done, and have reverted the #2155 patch in favor of a test-only solution :smile:

@Zac-HD I was wondering, would it be possible to add generic support for constrained types like ConstrainedStr. For example:
st.register_type_strategy(
    ConstrainedStr,
    lambda T: st.from_regex(T.regex, fullmatch=True) if T.regex is not None else st.text(min_size=T.min_length, max_size=T.max_length)
)

Unfortunately, that type strategy is not picked up on for subclasses of ConstrainedStr:
class MyString(ConstrainedStr):
    regex = "[A-Z]{2,8}"

st.from_type(MyString).example()  # Returns empty string

This doesn't need to hold back this PR, but it is related so I thought I'd post it here.

I was all ready to explain why we couldn't, but in fact there is a way... it's just a little more complicated than you might expect.  First, the obvious approach of registering ConstrainedStr doesn't do anything for child classes.  That's correct, if unfortunate in this case - otherwise we could just return builds(object) for everything!
So the trick is to register a strategy for each child class when that class is created, which can be done from the __init__ method of a metaclass.  The second trick is to ensure that this is a noop unless the user is already using Hypothesis, and that can be done with a WeakSet and a plugin (see https://github.com/Parquery/icontract/pull/181/files plus https://github.com/mristin/icontract-hypothesis/pull/5/files for an example).
I'll update this PR in a week or two, or if you'd like to merge it sooner I can open a follow-up instead 😁

This looks good. I'm happy to merge as soon as tests are passing.

Thanks @samuelcolvin :grin:
I think it'll take another design session; the fundamental concept is working well but there's a few issues still to iron out - conlist() and conset() have a type parameter and will require some indirection, the ConstrainedNumberMeta metaclass introduces a few others, and to finish up I'll need to pay some more attention to test-time performance.  But it's on track and I'm looking forward to shipping it :rocket:

That's great, let me know when it's ready to merge and I'll do so.

great, let me know when you think this is ready to merge.

@samuelcolvin - I'm ready to merge 🚀

see description above for details about the limited support for URLs, conset(), conlist(), and Literal[None]-on-py36
the docs declare that this will ship in Pydantic 1.8 - is that correct?

But overall I'm very happy with where we've ended up 😁

@samuelcolvin - I'm ready to merge 🚀

see description above for details about the limited support for URLs, conset(), conlist(), and Literal[None]-on-py36
the docs declare that this will ship in Pydantic 1.8 - is that correct?

But overall I'm very happy with where we've ended up 😁

Great job again @Zac-HD! I'll probably use it as soon as v1.8 is released 🚀
For conset and conlist, I think it will be possible to support them in the plugin once rewritten with Generic[T]. We are currently waiting for cython to support it

Great job again @Zac-HD! I'll probably use it as soon as v1.8 is released 🚀
For conset and conlist, I think it will be possible to support them in the plugin once rewritten with Generic[T]. We are currently waiting for cython to support it

Thanks for the review @PrettyWood!  I've added your suggestions, along with even more comments to explain what's happening 😄

Thanks for the review @PrettyWood!  I've added your suggestions, along with even more comments to explain what's happening 😄

this is awesome, thank you so much! 🚀 🙏 🥳
I'm working through PRs now, v1.8 coming soon.

Woohoo!  Pydantic is also awesome, I'm so excited about combining it with Hypothesis 😁
Can't wait to see what people do with this either :shipit:


SPIKE: Data Hub figment for production solution?

We are currently using datahub.figment.io Cosmos cloud nodes.  Will this solution work going forwards?
Connect with Conscience- he has best insight
Also, chat with Marley about custom dedicated node.  Could he host for Osmosis specifically?
Do we want to use GRPC if possible?
Currently GRPC is not supported for Osmosis
A/C:
Create issues necessary to enable app to connect to cosmos/osmosis nodes

Reached out to figment on Discord:
Yash - Figment — 01/09/2022
@wegfox @adamfox I've to check with our team for the availability of gRPC endpoint on these networks. I'll keep you posted

https://discord.com/channels/741351331222126663/741354053736333333/929940610255847454

Reached out to Yash again:
https://discord.com/channels/741351331222126663/741354053736333333/936670858238373888

Response from Yash:
https://discord.com/channels/741351331222126663/741354053736333333/936672562153398392
"hey @adamfox sorry for the delay in getting back, I missed it. currently there're no such plans of having gRPC on Osmosis & Secret very soon"

figment doesn't support gRPC so will leverage unchained nodes.


Reduce timeout for SmartLogic API token requests and log failures

Our Nginx logs report a series of failed requests for SmartLogic API tokens via the ses-token route.
<IP_ADDRESS> - - [06/Feb/2020:18:39:13 +0000] "GET /ses-token/ HTTP/1.1" 499 0 "https://boardagendas.metro.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
<IP_ADDRESS> - - [06/Feb/2020:18:39:13 +0000] "GET /ses-token/ HTTP/1.1" 499 0 "https://boardagendas.metro.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
<IP_ADDRESS> - - [06/Feb/2020:18:39:13 +0000] "GET /ses-token/ HTTP/1.1" 499 0 "https://boardagendas.metro.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
<IP_ADDRESS> - - [06/Feb/2020:18:39:13 +0000] "GET /ses-token/ HTTP/1.1" 499 0 "https://boardagendas.metro.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
<IP_ADDRESS> - - [06/Feb/2020:18:39:13 +0000] "GET /ses-token/ HTTP/1.1" 499 0 "https://boardagendas.metro.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"

When multiple requests to SmartLogic hang, it can cause all requests to the site to time out.
As recommended by requests, we should set a timeout so these requests don't hang indefinitely, and also log timeouts so we can report them to Metro.

Given the concurrency of these requests, we should also check that our front-end code doesn't issue duplicate requests for a token.

FYI @shrayshray a fix for this is in and deployed to the production site.


SDA-7231 Update rosa describe for hypershift

This PR changes the following describe messages for hosted clusters:

"Red Hat hosted" -> "ROSA Service Hosted".
Split Availability between Control Plane and Data Plane.

$ rosa describe cluster -c lponce-local-01

Name:                       lponce-local-01
ID:                         227cb83dnhfbfab3v1j4quoqqbrd187r
External ID:                1d58dd7c-c5fe-4616-8c48-e8c5928d065a
Control Plane:              ROSA Service Hosted
OpenShift Version:          4.12.6
Channel Group:              candidate
DNS:                        lponce-local-01.fhww.hypershift.sdev.devshift.net
AWS Account:<PHONE_NUMBER>89
API URL:                    https://ae1c7b56c59c042938d385dafcf730fe-35bfeacc9ef5fa6a.elb.us-west-2.amazonaws.com:6443
Console URL:                https://console-openshift-console.apps.lponce-local-01.fhww.hypershift.sdev.devshift.net
Region:                     us-west-2
Availability:
 - Control Plane:           MultiAZ
 - Data Plane:              SingleAZ
Nodes:
 - Compute (Autoscaled):    3-4
 - Compute (current):       3

/lgtm


Add CircleCI Deploy Option

This PR will address #350 to add a circle configuration (meaning a hidden folder, .circleci with a config.yml file) to do the following:

build a container from a user specified repository with notebook
deploy to Docker Hub

The base could additionally be customized by the user to do more testing or extraction of metadata, and this should be considered a base or skeleton. To help with this, I have also made the following changes to documentation:

I added a deploy.rst page, rendered from the index.rst. I usually do markdown so please yell at me if I mucked up some formatting :)
I replaced the (broken) Data Science Notebook repository with the other example that is working.

What you might want to do is to connect this repo to CircleCI and then define the required environment variables and create the Docker Hub repository (read the deploy.rst instructions) so that you can build and deploy an example to go with this repository. if you wish, you can also add a filter to only deploy on master merge!
For more verbose description, please see the issue here ! https://github.com/jupyter/repo2docker/issues/350#issuecomment-405194344

Short thought: should we move .circleci to somewhere else (maybe examples/?) to make it clear that this is an example of how you would setup your repository if you wanted to use repo2docker to continuously build the image for your repo?
For repo2docker we don't really want to build a docker image all the time and might want to use circleci for other things. At which point it would become confusing which parts of the circle CI config are for repo2docker testing and which are about setting up this continuous build thing?

This is awesome! Some short(ish) thoughts below:
Should we move .circleci to somewhere else (maybe examples/?) to make it clear that this is an example of how you would setup your repository if you wanted to use repo2docker to continuously build the image for your repo?
Is this the simplest possible circleCI config we can cook up? For me the level of complexity is Ok, but I am thinking of someone who is new to this whole container and CI thing. Every extra line that does something which is "nice to have" but not really required adds complexity and potential for cargo cult'ing. If there are steps we can remove I'd maybe make a "basic" version and a "doing it really well".
For the repo2docker repository we don't really want to build a docker image all the time and might want to use circleci for other things. At which point it would become confusing which parts of the circle CI config are for repo2docker testing and which are about setting up this continuous build thing?

With the changes that are coming to circle (orbs) you will actually be able to maintain separate files, so I don't think it would be confusing. But it's up to you! I'm not a jupyter organization person so I can't make a new repository with a working example, which I think is essential. I would suggest that it doesn't hurt now to have this (because it's useful and needed) and either decide to keep here (because it's not confusing when orbs are released) or when you do set up circle here (and decide you don't like having both) to create the dummy repo. It's definitely up to you!

Multiple Workflows

I think of myself as a proficient Circle CI and travis user, and yet I have to grab the docs all the time when I do reconfigure one of them. So I was wondering if we could have a directory that I can copy into my existing repository that "does the right thing" without having to study the example here to remove the bits which are for testing repo2docker and aren't needed for my repo.

A New Separate Repository

I think this is a good idea. We could place it in the https://github.com/binder-examples/ organisation where we already have quite a few binder related example repositories. We could configure a cron job for that repo so that we get builds once a day/week so we notice when a change to repo2docker breaks this setup. (Cron builds would be a cool advanced feature anyway ;) )
I would want people to take the example config from somewhere and add it to their existing repository. Not start from cloning repo2docker and modifying it to add their work.

@betatim with the changes I made (removing the run) there is no extra work needed other than defining credentials in the CircleCI web interface. The user clones, connects, and defines those variables, that's as simple as I can get it! (no needing to edit the config file, I agree that is always challenging.)
Sure that works! I'm not sure how I can help if the repository doesn't exist though... ideas?

Okay DOCKER_TAG looks good! See the updated build here:
https://circleci.com/gh/vsoch/repo2docker/22
And the tag is now very specific:
https://hub.docker.com/r/vanessa/repo2docker/tags/
This new strategy ensures that every single deployed image is maintained on Docker Hub (as opposed to having many "latests" with different sha sums) so it's a neat idea.
For next steps - if you want to make me a repository to issue PR to where an example (connected to circleci) could be depployed, I'd be happy to remove the .circleci folder from here, and put it over there. The docs can still be added for deploy.rst here, but they will point the user to the other respository. The other repository can be exclusively for many examples of different tests/ other one might use for a circleci deployment, and we can add them as users request/ need.

For next steps - if you want to make me a repository to issue PR to where an example (connected to circleci) could be depployed, I'd be happy to remove the .circleci folder from here, and put it over there. The docs can still be added for deploy.rst here, but they will point the user to the other respository. The other repository can be exclusively for many examples of different tests/ other one might use for a circleci deployment, and we can add them as users request/ need.

I think that is the option I'd champion. Let's see if any of the others have comments. If we decide to do this I'll create a repo in binder-examples.

woohoo! I'm down!

Let's leave this one open - I'll remove the config.yml from here, and update the docs to be referencing the binder-examples. That way, the user can still find what he/she is looking for even if the files aren't hosted alongside.

I think the proposal you folks came to sounds like a great way to go.

I created https://github.com/binder-examples/continuous-build. The name is not perfect, suggestions welcome.
👍  to keeping docs here in this PR and the CircleCI config in https://github.com/binder-examples/continuous-build

on it!

@betatim I've updated this pull request in the following ways:

the entire circle config folder is removed, and the jupyter team can add something back when desired to write tests for repo2docker (I agree this is the more appropriate approach to take!)
I have updated the deploy.rst docs to now reference the new "continuous-build" folder.
I don't think the name is bad at all, it says exactly what it is and I like it.

I have also opened a pull request to the other repository that will include the "default" configuration example, along with (in the future) configurations with other bells and whistles.

The pull request is here
The (same) deployment as before from the new repo is here.

It's pretty cool that I didn't need to redefine the environment variables, I could simply import them from my previous project! This is really cool for organizations because it means that (hidden credentials) to deployment places could be shared with others in the organization without actually sharing them. The last few letters of variables ARE shown however, so this should be done with care (if at all). If you would like to add me as a contributor to the other binder-examples repo, I'd be glad to monitor and help users with custom / new configurations. We will have a little recipe book of templates!

Update for those following this PR - the repo with just the recipes is now merged and ready for use, and we can update the docs here (per this PR) and then (if jupyter has other forms of communication / docs) share with users, and start to brainstorm how to capture metadata, test, etc.

I agree, creating a separate issue would be a good idea.

@jzf2101 see https://github.com/binder-examples/continuous-build/ ! We have one more PR in to set a default REPO_NAME so the circle build doesn't fail given that the user forgets to define one. I had originally imagined we would want it to fail, but probably not.

oh yes that's a great idea! +1 - maybe put this as an issue on the other repo (unless you already did, haha).  I think it could work both ways actually, given that there is a travis and circle builder template, for example, the files are isolated (e.g. travis.yml vs .circleci so you would want to have certainty of testing from the same base (argument for keeping one repo). On the other hand, let's say we have a bunch of just circle templates, it would be more clear to have separate repos for each (and just take care to test them separately as appropriate).

Let's move the discussion of naming and whether or not to have multiple CIs in one repo to https://github.com/binder-examples/continuous-build/issues/5

Did you need any more clarification on my part? We have this documentation linked from the continuous build repo, so we should get deployed soon!

Nope, just slow because it was the weekend :) Merged!

Looks great! https://repo2docker.readthedocs.io/en/latest/deploy.html  and Happy Monday :)

Thanks again for all the work and review here! Please ping me if I can help with anything in the future.


Update camel to version 4.5

Please describe the task that needs to be done
Update to the latest Camel version to receive the upstream recent fixes
Prerequisite of: https://github.com/KaotoIO/kaoto-next/issues/755

Closing it because we're on Camel 4.6 now


Support BehaviorTab starting out paused

If say START_PAUSED=1, BehaviorTab starts out in 'paused' mode, eg. doesn't run the behavior right away, maybe not even connects to the tab? Or maybe just connects and doesn't start behavior yet, until it gets the pubsub start message, whatever is simpler.

Yep, can confirm it already starts out paused, waits until the start message has been received!


HIDPI on mac

HIDPI in imgui is still a mess, if you have a monitors with different DPI's on mac, this happens when you move a window to the higher DPI display.

The easiest work around is to just disable retina framebuffers in glfw
glfwWindowHint(GLFW_COCOA_RETINA_FRAMEBUFFER, GL_FALSE);

I've gotten hidpi working somewhat smoothly on Windows in one of my branches. It's still a mess.
glfw enables DPI awareness by default on Windows.  I wish glfw had an equivalent hint for windows.

@markreidvfx I switched it to use the GLFW + Metal backends (see https://github.com/jminor/raven/commit/b653fc88407b49908e450919eec109664774ccb8 and subsequent changes) and this seems to be fixed now. That also future-proofs this vs Apple's plan to eventually drop support for OpenGL.
FYI, I left the GLFW + OpenGL build for use on Windows (though I haven't tested that since the change). I don't have access to a Windows machine with a high-dpi monitor, but if that has a similar issue, maybe we could try one of the more native Windows backends?

I'll try and port it to the DirectX + win32 backends.


Option to keep original name after update/replace item.

Sometimes I update an item with a translated name and I want to keep it instead of changing it to the CPR Compendium name. I added a setting in the 'General' section to retain the name.

It's not intended to overwrite the name, the item updater should just pull the old name, a setting isn't needed.

Correct, just change that instead.


added extensions and paths to exclude

(fonts, dbs, ides, builds, icons and common tests)

Looks good! Thanks so much for your contribution!

I put in but consider removing "test" from testPaths as it may be too generic

Agreed, I was actually wondering the same. Going forward, I intend to treat tests as follows:

If a value for the flag -tests is provided, then only this path is used (and your added function is ignored)
If -tests is not provided, then your function is used

Moreover, I think it makes sense to provide an extra flag, e.g. an -include flag, to explicitly specify folder names that should be included in case the tool automatically excludes something interesting. For example: Say the app is called "tester" or something, then -include tester would include all paths with "tester" in it despite the word "test" in it

I should have now solved the concern regarding "test" being too generic. Previously, e.g. a folder called "testimonals" would have been deleted. With the new check in place (https://github.com/fw10/veracode-javascript-packager/blob/main/utils.go#L86), only a folder called e2e, test, __tests__ should be removed now.


Alternative implementation of ParamOverrides

I've re-implemented ParamOverrides, following the suggestion made in #85 about sharing parameter objects. This PR isn't meant for merging at this stage, I'd just like some comments on the basic idea.
This new implementation passes all the tests (including one I added to check that dynamic overrides work), although the implementation's not yet complete (just missing some minor methods that the current ParamOverrides has). However, what I'd like to know is: is the whole idea crazy? If it's not crazy, is there a way to make the implementation simpler? I've left extra comments in the code for now to explain what I was thinking.
In case it's easier to look at the class in one piece:
https://github.com/ceball/param/blob/bug85_newParamOverrides/param/parameterized.py#L1580
(Also, note that I haven't considered speed.)

Changes Unknown when pulling 134ef81ee847f5c9d4480e89e9c1d0a177524fa6 on ceball:bug85_newParamOverrides into ** on ioam:master**.

Also, what do you think about this? Say a Parameterized instance a has a mutable parameter, e.g. b = param.List([1,2]). If someone creates a ParamOverrides and sets that parameter to a new list either during initialization (ParamOverrides(a,{'b':[3,4]})) or afterwards (ParamOverrides(a).b=[3,4]), no problem. But if someone edits the mutable parameter without first having set it (ParamOverrides(a).b[0]=3), they'll actually be editing the parameter on the overridden instance (a.b will be [3,2] in this case). To me that's probably ok with suitable documentation, but with a more complex implementation it would be possible to do better for instantiate=True mutable parameters.
(I think the above behavior is the same for the existing ParamOverrides.)

Seems ok to me, when finished.  Is it really necessary to make a new class with all the parameters?  I.e., is there no easy way just to point to the parameters as they are in the overridden class?  I haven't tried either approach, so I don't know.
I don't think the mutable object behavior you describe is worth putting in a special case for; people already have to know to deal with mutable objects differently in general.

On Sun, Feb 15, 2015, at 07:23 PM, James A. Bednar wrote:

Seems ok to me, when finished.  Is it really necessary to make a new
class with all the parameters?  I.e., is there no easy way just to point
to the parameters as they are in the overridden class?

A parameterized instance doesn't have pointers to its parameter objects,
except via its class. The new class is where I store the pointers to the
overridden parameterized instance's parameter objects. No new parameter
objects are created (except the standard parameters of Parameterized,
since ParamOverrides is now a subclass of that).
Is this what you meant, or did you mean could I avoid having a new class
by storing elsewhere the pointers to the parameter objects (i.e. doing
something different from a normal Parameterized)?
Or am I on the wrong track completely?

You're the expert on Param, not me, so I'd usually be the one on any wrong track!
What I meant was that I don't see why you need a new class to store the pointers to the parameterized instance's parameter objects.  Doesn't overridden's class already exist, with precisely the right set of parameter objects?  So I'm imagining (very vaguely, so that's as much detail as I can give) somehow pointing to the existing class, so that we don't have to make a new one.  But there could easily be technical reasons why this is not feasible.

Changes Unknown when pulling e0f2d05765ed7b9b896ae0d67bb36b450039b3ae on ceball:bug85_newParamOverrides into ** on ioam:master**.

Coverage increased (+0.61%) to 69.11% when pulling b000a46003e9e0e0c82b917e4fc10a71401390b9 on ceball:bug85_newParamOverrides into e4a7720417b9f6efb51888b7bb0de0e3db7a6aeb on ioam:master.

Coverage increased (+0.61%) to 69.11% when pulling 7d4a622d8ac605e7984b9fa74603549a43adb7e4 on ceball:bug85_newParamOverrides into e4a7720417b9f6efb51888b7bb0de0e3db7a6aeb on ioam:master.

Is there any chance of finishing this up? The fact that ParamOverrides doesn't actually validate types is a major issue.

When resuming this work, please consider addressing or closing:

#36
#134
#173

I think we've established that ParamOverrides has moved from something we're trying out to see if we want, to something that we use, so it's time to finish this! ;) I'll bring this PR up to date.
@jbednar:

What I meant was that I don't see why you need a new class to store the pointers to the parameterized instance's parameter objects. Doesn't overridden's class already exist, with precisely the right set of parameter objects? So I'm imagining (very vaguely, so that's as much detail as I can give) somehow pointing to the existing class, so that we don't have to make a new one. But there could easily be technical reasons why this is not feasible.

Yes, the overridden's class already exists, and yes it has exactly the right set of parameter objects. But...
Parameter objects are stored on the class. Different instances of ParamOverrides will override the parameters of different parameterizeds and will hence need different parameters.
Your question is about having no new class, but in that case the ParamOverrides class would end up with links to parameters on multiple different parameterized classes i.e. it would not have the right parameters for any one parameterized class.
My implementation is one new class for each ParamOverrides instance created, to store the links to the overridden's parameters. I did that because it's easy to do. Each new class is a subclass of overridden's class. However, it would instead be possible to create only one new class per type of overridden. I don't see how it's possible to do it with fewer classes than that while keeping the implementation of ParamOverrides inheriting from Parameterized. But to have only one new class per type of overridden, we'd need to keep the classes around.
Unless I missed something? Quite possible!

Coverage increased (+1.06%) to 75.705% when pulling 3de2729cb0371c69b259630bf8607a13e3710830 on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+1.06%) to 75.705% when pulling 3de2729cb0371c69b259630bf8607a13e3710830 on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+1.06%) to 75.705% when pulling 3de2729cb0371c69b259630bf8607a13e3710830 on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+1.06%) to 75.705% when pulling 3de2729cb0371c69b259630bf8607a13e3710830 on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+1.06%) to 75.705% when pulling 0a8686e8faee3cb630677382bf1834812ef02306 on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+1.06%) to 75.705% when pulling 0a8686e8faee3cb630677382bf1834812ef02306 on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+1.06%) to 75.705% when pulling 0a8686e8faee3cb630677382bf1834812ef02306 on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+1.1%) to 75.769% when pulling 52a80a3b2bcbaed58f357aa15119e289d8f2c5bb on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

What's left?

[ ] add missing methods noted in code TODO
[ ] test with holoviews
[ ] test with topographica?
[ ] test performance?
[ ] fixed #134?
[ ] fixed #173?
[ ] address #36?

Coverage increased (+2.2%) to 76.859% when pulling 53666f78854b3ba7bab6cba659f8297d34ae9f3f on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+2.3%) to 76.977% when pulling 8b2aee5054db6263f5d93df697c4386b5015ddcd on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+2.3%) to 76.977% when pulling a0c5da326fa6fa81398a1639ad27183ce5842f9d on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+2.3%) to 76.977% when pulling c36d507fc4b04ccc231956e01a607e2e0e8a42ca on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+2.6%) to 77.29% when pulling 2c675fe1d0f1d7472ee9460098d2022c34d3c78a on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+2.6%) to 77.29% when pulling 682a627fc60e1c332c35223057e51af25c7ba0e6 on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Coverage increased (+2.6%) to 77.27% when pulling 6214ee475e54ea1802b007d344c79295a9dc2fb6 on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

I'm very happy to see these improvements to ParamOverride but I'm not sure about the decorator and corresponding underscore attribute described in #36.

@jbednar: Maybe you could review this again? I think we could merge right after the next param release and then see what happens for a while. I've tested with holoviews but not yet topographica (the two things most likely to exercise paramoverrides the most).
Notes:

ParamOverrides is less like a dict than before (when it was a dict!). I've added what I think is the minimum dict-like behavior to support holoviews. We could make it emulate dict completely, though.

ParamOverrides is at the same time slightly more dict-like than before, because now it raises KeyError rather than AttributeError when a name's not found via dict-like access.

Because ParamOverrides now inherits from Parameterized, the name parameter will effectively always be overridden i.e. in typical usage p.name will not be the name of the overridden object but of the ParamOverrides object. That could be confusing.

Keep the overloads decorator or not (see #36)? If keeping, what about its name (and the name of "_overrides")?

I haven't tested performance :( The following comment used to exist: we'd like __init__ to be fast because it's going to be called a lot

Fixes #134 and #173.

@jlstevens:

I'm not sure about the decorator and corresponding underscore attribute described in #36.

Yep, I agree. Wrote my comment above at the same time as yours. After initially liking the way it seems to make things look a bit nicer, I think I prefer the explicit p = ParamOverrides(...).

Another note: #176 (issue #104) should be merged first or at the same time, or else holoviews will have spurious failing tests next time holoviews' travis runs (it takes param from github, I think).
For reference:
======================================================================
ERROR: Assert histogram shares colormapper
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/cball/code/ioam/holoviews/tests/testplotinstantiation.py", line 839, in test_side_histogram_cmapper
    plot = bokeh_renderer.get_plot(img.hist())
  File "/Users/cball/code/ioam/holoviews/holoviews/core/element.py", line 41, in hist
    individually=individually, dimension=d, **kwargs)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 1881, in __new__
    return inst.__call__(*args,**params)
  File "/Users/cball/code/ioam/holoviews/holoviews/core/operation.py", line 125, in __call__
    self.p = param.ParamOverrides(self, params)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 1758, in __init__
    Parameterized.__init__(self,**dict_)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 1041, in __init__
    self._setup_params(**params)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 978, in override_initialization
    fn(parameterized_instance,*args,**kw)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 1441, in _setup_params
    setattr(self,name,val)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 537, in __set__
    self._check_value(val)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 534, in _check_value
    raise ValueError("String '%s' only takes a string value: %s"%(self._attrib_name,val))
ValueError: String 'dimension' only takes a string value: None

======================================================================
ERROR: test_side_histogram_no_cmapper (tests.testplotinstantiation.TestBokehPlotInstantiation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/cball/code/ioam/holoviews/tests/testplotinstantiation.py", line 827, in test_side_histogram_no_cmapper
    plot = bokeh_renderer.get_plot(points.hist())
  File "/Users/cball/code/ioam/holoviews/holoviews/core/element.py", line 41, in hist
    individually=individually, dimension=d, **kwargs)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 1881, in __new__
    return inst.__call__(*args,**params)
  File "/Users/cball/code/ioam/holoviews/holoviews/core/operation.py", line 125, in __call__
    self.p = param.ParamOverrides(self, params)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 1758, in __init__
    Parameterized.__init__(self,**dict_)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 1041, in __init__
    self._setup_params(**params)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 978, in override_initialization
    fn(parameterized_instance,*args,**kw)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 1441, in _setup_params
    setattr(self,name,val)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 537, in __set__
    self._check_value(val)
  File "/Users/cball/code/ceball/param/param/parameterized.py", line 534, in _check_value
    raise ValueError("String '%s' only takes a string value: %s"%(self._attrib_name,val))
ValueError: String 'dimension' only takes a string value: None

Coverage increased (+2.5%) to 77.154% when pulling 4bb95b6d9c370ad7fc1c5123943f02a1e9d734eb on ceball:bug85_newParamOverrides into 131227a603050889f1be57a1ec8109626226726c on ioam:master.

Testing with topographica has revealed two things missing from this implementation vs the previous one: (1) support for pickling, and (2) support for arbitrary (non parameter) attributes on parameterized function instances.

Topographica's tests also show a use case I'd completely forgotten about: p=ParamOverrides(...); f(**p). I think p has to be a dict for that to work (i.e. p can't just have the right methods). Or maybe it's just got to be a Mapping (abstract base class) - can't remember if that's in python 2.6 or not. Either way I think that makes it difficult for the implementation proposed in this PR to be backwards compatible.
E.g. from imagen:
class GaussianCloud(Composite, TimeAware):
    ...
    def __call__(self,**params_to_override):
        p = ParamOverrides(self,params_to_override)
        ...
        return super(GaussianCloud,self).__call__(**p)

Also, missing param_keywords() method.

Summary so far! :(
Currently, we use ParamOverrides like this: p = ParamOverrides(overridden, overrides), where overridden is a Parameterized instance and overrides is a dict. Typically we use this in the __call__ method of overridden.
I think we want p to behave a lot like overridden. When you try to set things on p, you should get param's usual behaviors (e.g. value checking). When you get things from p, you should get param's usual behaviors (e.g. dynamic values). For example, this should work:
class A(param.Parameterized):
    x = param.String(default='hi')
    y = param.Number(default=0.5, bounds=(0,1))

a = A()

try:
    param.ParamOverrides(a, dict(x=1))
except ValueError:
    pass
else:
    raise

param.ParamOverrides(a, dict(y=lambda:0.75)).y == 0.75

try:
    param.ParamOverrides(a, dict(y=lambda:1.5)).y
except ValueError:
    pass
else:
    raise

If ParamOverrides doesn't work just like param normally does I think it's confusing. We have at least one example of that already: "ParamOverrides stops Filename search_paths from working" (https://github.com/ioam/param/issues/134), but there'll be others. I've also found myself wanting to use the normal methods available on Parameterized, too. E.g. "ParamOverrides: support get_param_values()" (https://github.com/ioam/param/issues/173). How far should we go? I think ParamOverrides should be just like the overridden instance.
Do you agree with the above? If you think it's too much, and we should instead have ParamOverrides support only a specific, smaller set of things, then please say.
I thought the simplest and easiest way to achieve the above would be for ParamOverrides to inherit from Parameterized. I felt that alternative suggestions would result either in ParamOverrides being some kind of reimplementation of Parameterized, or in ParamOverrides requiring more magic to access features of Parameterized via overridden than exists in the proposed ParamOverrides-inherits-from-Parameterized implementation.
I created what I think is a bit of an ugly solution, but it works (in that it fixes an important gap in parameter checking, particularly for parameterized functions). I knew I could easily add anything necessary to make the new implementation backwards compatible with the previous one (except for the part about catching parameter errors i.e. the reason for this PR), giving us something working right now (the priority), while hoping in the future we might be able to simplify things/find an entirely better solution. However, it turns out that there's one thing I probably can't do. As noted above (https://github.com/ioam/param/pull/92#issuecomment-328301714), you can only use a Mapping for kwargs. In topographica and imagen (but not holoviews), that appears be done with ParamOverrides a couple of times (e.g. in a __call__ method: p=ParamOverrides(...); ...; super(...).__call__(**p).
So what should I do? Add a method to get a dictionary for use in kwargs and update imagen and topographica and apologize that ParamOverrides may start generating two types of error for your old code (genuine parameter errors, and an error if you try to use it for kwargs without updating your code)? Or maybe make the new ParamOverrides even trickier than it already is by also inheriting from Mapping? Or have I gone down a miserable technical rabbit hole, and can someone suggest an altogether simpler approach?
Jim previously suggested this:

maybe we should provide a totally different way to do this that's less awkward? E.g. can we offer a method decorator that pushes the current dictionary of parameter values onto a stack, lets the method do what it likes, then pops the original values back off when the method returns?

However, I would be worried about changing "the state of the parameters". You might look at a __call__ method and decide it doesn't modify anything so you can use it "safely" from multiple threads (or in e.g. a generator), without realizing what's happening to the parameters. It might be particularly surprising for someone to find calling 'a function' with some arguments in one place could affect the function's parameters in another. E.g.
class A(param.ParameterizedFunction):
    x = param.Number(default=0)

                   # pushes __dict__ onto stack and replaces with temp copy into which
    @override_dict # ** params are set; pops __dict__ back off stack on return
    def __call__(self, **params):
        # some long-running operation e.g. a network operation...

a = A.instance(x=0)

with concurrent futures ThreadPool()) as executor: # (I can't quite remember how that goes)
    executor.submit(a) # will likely have a.x = 0
    executor.submit(a, x=10)
    executor.submit(a) # will likely have a.x = 10 because a(x=10) will still be running

Closing this as stale. We are likely to harvest some code from here, but it would require significant work.