Feature/bsk 222 integrator refactor

Tickets addressed: bsk-222 (https://github.com/AVSLab/basilisk/issues/222)
Review: By commit
Merge strategy: Merge (no squash)

Description
The ticket suggested moving most of the current integrator functionality to a parent class, which was the approach followed in this PR. This parent class is in charge of collecting the state data from dynamic objects, calling equationsOfMotion to obtain the required derivatives, and combining this information into the new state. An adaptive version of the RK schemes is also supported. These classes support parallel integration of multiple dynamic objects by "flattening" the state information of every dynamic object into a single map.
Note that step sizes are computed slightly different in the adaptive methods, thus simulation results obtained using these integrators might change slightly after this PR. User interfaces remain the same or have been expanded, which means that there are no backwards compatibility issues.
Verification
The fixed-step integrators produce exactly the same results as before, and thus existing tests pass on them. The tests for adaptive integrators were updated to reflect the slight changes in how the step-size control is implemented.
Documentation
The module documentation LaTeX was updated to reflect how the testing and new step size algorithm have changed. A new scenario has been added that showcases the differences between integrators. Other documentation remains the same.
Future work
Implement other RK schemes now that this is very easy to do. Possibly, consider implementing other kinds of integrators, such as Adams-Bashforth or implicit RK.

I cannot assign reviewers, so I'll tag @joaogvcarneiro @schaubh @patkenneally

Builds for Linux are failing because the code is using C++14 features (std::make_unique, but possibly more). Interestingly, Windows builds seem to work without issue, which probably means we are using some inconsistent compiler flags.
Should I refactor the C++14 features into C++11 code, or could we expect updating to C++14 (or even 17 or 20) in the near-ish future?

Should I refactor the C++14 features into C++11 code, or could we expect updating to C++14 (or even 17 or 20) in the near-ish future?

I would suggest holding off on a refactor. Recently, I proposed advancing to C++14 (at least, but ideally C++17) because I too have wanted language features such as std::make_unique. In general our use of raw pointers in the code base is not ideal and there are places where ownership should be explicitly encoded to clearly communicate the intent/design.
In my experiments, not a lot needs to change to move to 14 or 17. In fact nothing code base-wise. The changes are primarily Cmake compiler directives (as you eluded to). So let's look at putting together a branch that does this. If everything checks out, we can merge that branch and then merge in this branch after. What do you think of that proposal?

So let's look at putting together a branch that does this. If everything checks out, we can merge that branch and then merge in this branch after. What do you think of that proposal?

@patkenneally  Sounds good, see https://github.com/AVSLab/basilisk/issues/274

@patkenneally it seems like Linux is still failing to build (see action). I've tried to find possible causes, but I haven't come up with anything.

Found the issue! Turns out the windows compiler (Visual Studio 16 2019 / MSVC 19) is more permissible than the linux compiler (GCC 9.4) in some aspects, which meant that some code was acceptable for windows but not linux.
For anyone interested: there were two separate issues, both related to subclassing templated classes. The first was related to using type definitions defined in the parent class:
template <typename T>
struct Base { using type = int; };

template <typename T>
struct Derived : public Base<T>
{
    type mem;
};

In this case, MSVC correctly finds the definition of type in Base<T>, so this compiles. GCC, on the other hand, raises an undefined type warning. The other problem happens for calling a parent constructor from the subclass of a templated class:
template <typename T>
struct Base {};

template <typename T>
struct Derived : public Base<T>
{
    Derived() : Base() {};
};

In this case, MSVC understands that : Base() is calling the parent constructor, while GCC needs Base<T> to be used.
Check out this godbolt https://godbolt.org/z/nqd3TG4e9 to see how MSVC compiles while GCC doesn't for these cases.

We might want to change the way we're testing these integrators. We currently use numeric values based on another simulation using Mathematica. I suggest simulating for n periods and testing the position and velocity at periapsis and apoapsis since we know when they happen and what the result should be. Since the initial conditions are the same, these truth values should be the same for all integrators. We need to lower the timestep enough for the lower-order integrators so the tests still pass.

I agree this could be an additional test, and it is a very good smoke test for checking that the integrators are working as expected. However, I see the value in also keeping the current approach. It is possible for us to make implementation errors that are subtle enough to make integration work for a low time step, but that are not as performant as the proper integrator. For example, we might define the coefficients of an integrator wrong (instead of A_{2, 1} being 0.45 we set it to 0.4). An error as small as that will hurt the performance of the integrator, but it will likely still converge to the analytical result for a small enough time step.

The first commit should be split into two: the first part would be the extendedStateVector, and the second part would be the svIntegratorRungeKutta.

Agree, the fact that I used "Additionally" in the commit description should have been a clear signal that it's probably best split.

Regarding doing new integrator tests, the nice thing about the current setup is that the tests run very fast, and we are checking the values for a handful of steps to ensure each value is what we expect.  If we doing do spot checks are apoapses and periapses, we will have to integrated the states of a longer time period to which can drastically increase the time to run these tests.  I'm not seeing any benefit to running such slower test beyond that the truth values are computed on the fly rather than crafted offline.  I would like to stay with our current testing method for now.


Cleanup PyCon 2016 titles

@Daniel-at-github
Can you remove the speakers' names and file formats from the titles of these videos?
For example: Brian Warner - Magic Wormhole- Simple Secure File Transfer - PyCon 2016.mp4 could be Magic Wormhole- Simple Secure File Transfer.
http://pytube.org/pycon-us-2016/brian-warner-magic-wormhole-simple-secure-file-transfer-pycon-2016mp4.html

Done in #57

Awesome! Thanks!


Vulkan: Enabling any MSAA level hangs the editor indefinitely

Godot version
4.0.dev
System information
Windows 10, Vulkan, GTX1080 (460.89)
Issue description

As the title states, enabling MSAA at any quality level hangs the editor indefinitely.
Steps to reproduce
N/A
Minimal reproduction project
No response

Yes, I can also confirm experiencing this too on Windows 10, GTX 1080 (471.11)

@Calinou @missembyr Tested this again today after pulling this PR #49973 and appears to be resolved, so it may be okay to close this now.

Closed by #49973


Date range picker

It's frustrating when we can't zoom in and see more precise details for smaller/larger time periods.
I'd like to be able to zoom in to specific time ranges and scroll sideways to view more. The trades demo Details view looks like it has this feature but the picker is not actually implemented. When different ranges are selected, nothing happens.

https://github.com/denniscm190/trades-demo/blob/main/Trades/Details.swift#L44

Hello EstiT,
Currently, you cannot zoom in/out with gestures but you can update the chart to show a specific time frame. This is something you have to implement by your own since it's out of the scope of this framework. However, I'll try to help you:

Add a picker
Add .onChange(of:perform:) modifier
Implement a function that updates your @State variable that feeds your chart when the value that you captured with the .onChange(of:perform:) changes.


Random crashes (subtitle-related?)

For some reason whenever I play with this mod it often crashes at random times. Not Enough Crashes cannot identify a mod that's causing the crashes, but the log seems to indicate something related to subtitles. I first thought it was because of Hydrogen or Presence Footsteps, but the crash still happens even without those mods, even with the latest Java version, Fabric Loader and Fabric API.
Crash log: https://paste.ee/p/Bo51e
Installed mods:
Fabric Loader
Fabric API
Fabric Language Kotlin
Canvas Renderer
Lithium
Hydrogen
Human Reborn
Earth2Java
Decorative
Cave Biomes
Promenade
Decorative
Adorn
On Soul Fire
Wildfire Female Gender Mod
Winged
Better Dropped Items
ExtraSounds
Croptopia
FallingTree
Mod Menu
Elemental Creepers
Pettable
Command Macros
Inmis
Dawn API
Better Nether
Culinaire
Emotecraft
Mo Structures
Stoneholm
Cloth Config 5.0.34
LambDynamicLights
Trinkets
VanillaPlus Biomes
All the Blocks you Want
Blockus
WTHIT
Megane
Not Enough Crashes
Sentimentality 3
Xaero's Minimap
Let me know if there's anything else I need to add.

Also sorry if the link acts weird, try copying and pasting it

I am unable to recreate this crash, unfortunately, which makes it pretty hard to track down a cause. Is there any specific situation where this happens?

I am unable to recreate this crash, unfortunately, which makes it pretty hard to track down a cause. Is there any specific situation where this happens?

I tried reinstalling the mod to see if the jar was corrupted, but even with the new jar I still get the same crash. However, it seems to me that it can happen when there is a large amount of SFX with subtitles playing at the same time, possibly overloading the subtitle HUD. For me its prone to happen most when I am falling or flying down to a cave with a lot of vegetation or water, due to the different plethora of sounds it generates. I feel like this may be an issue on Mojang or Java's end, but the crash log generated seems to be kind of vague.
Another crash log from reinstalled mod.
https://paste.ee/p/gbcEk

This should be fixed in the next version, which I'll try and publish in the next few days after I fix #20.


Issues gotten from static analyzer

I ran static analyzer Svace and got critical issues:

After having been compared to a nil value at lookup.go:329 by calling function 'v6.Info.GetField' at lookup.go:259, pointer 'value[0]' is dereferenced at lookup.go:267.

After having been compared to a nil value at lookup.go:329 by calling function 'v6.Info.GetField' at lookup.go:299, pointer 'value[0]' is dereferenced at lookup.go:305.

After having been compared to a nil value at lookup.go:329 by calling function 'v6.Info.GetField' at lookup.go:361, pointer 'value[0]' is dereferenced at lookup.go:367.

After having been compared to a nil value at lookup.go:329 by calling function 'v6.Info.GetField' at lookup.go:377, pointer 'value[0]' is dereferenced at lookup.go:383.

After having been compared to a nil value at lookup.go:329 by calling function 'v6.Info.GetField' at lookup.go:393, pointer 'value[0]' is dereferenced at lookup.go:399.

After having been compared to a nil value at lookup.go:329 by calling function 'v6.Info.GetField' at lookup.go:409, pointer 'value[0]' is dereferenced at lookup.go:415.

After having been compared to a nil value at lookup.go:329 by calling function 'v6.Info.GetField' at lookup.go:425, pointer 'value[0]' is dereferenced at lookup.go:431.

After having been compared to a nil value at lookup.go:329 by calling function 'v6.Info.GetField' at lookup.go:441, pointer 'value[0]' is dereferenced at lookup.go:446.

There is no guarantee that method GetField returns not empty (including nil) slice. And there is no checks for for getting value as empty slice in methods GetAny, GetMap, GetBool, GetInt, GetUint, GetFloat32, GetFloat64 and GetString. This potentially may cause panics.
Please add checks for getting empty slice where they need to be.

version 6.26.4

https://github.com/brianvoe/gofakeit/blob/master/lookup.go#L260 GetField will return error if field not found.

https://github.com/brianvoe/gofakeit/commit/163b487242aee020e15867381c23ce70ed61fc57


Move elv integration to component and bump pypca

Breaking Change:
All config for the ELV/PCA component/integration should be removed and replaces with:
elv:

Optionally the exact port for the serial interface can be specified (default is: /dev/ttyUSB0)
Description:
Corrected the setup of the elv integration and it's switch platform. Bumped pypca to version 0.0.5.
Documentation PR: home-assistant/home-assistant.io#9995
Example entry for configuration.yaml (if applicable):
elv:
  device: '/dev/ttyUSB0'

Checklist:

[x] The code change is tested and works locally.
[ ] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.
[x] I have followed the [development checklist][dev-checklist]

If user exposed functionality or configuration variables are added/changed:

[x] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[x] [The manifest file][manifest-docs] has all fields filled out correctly. Update and include derived files by running python3 -m script.hassfest.
[x] New or updated dependencies have been added to requirements_all.txt by running python3 -m script.gen_requirements_all.
[ ] Untested files have been added to .coveragerc.

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

Please update your opening post with the template provided. You've now just deleted them, that is not ok.
Documentation PR: https://github.com/home-assistant/home-assistant.io/pull/9995
Please do the same update the documentation PR as well and link the PR's in the openings post.
Thanks.

@fabaff  for which files exactly? For the whole /elv directory?

Go the changes tab here on github. In the top panel for each file changes in the elv package you can see that it says that permissions changes from 644 to 755.

@majuss are you planning to continue here?

Yes of course! I was 3 weeks on vacation so it will take some time.

If you're not planning to continue in the near future, please close the PR and you can reopen yourself when ready. We're trying to decrease our open PR buffer. Thanks!

Please solve the merge conflicts.

Can be merged when build passes and merge conflicts are solved.

So now the black formatter just fails, I tried to run it in a local venv but I just get "NotImplementedError: This is still being implemented."

So what to do now? Can anybody help me out :)?

Please run black from the root of the project and target the whole homeassistant directory.
black --fast homeassistant

Still getting the same "not implemented" error message..

Can you post the error message? Something is probably not correct in your environment.

I created a complete new venv and tried it again:
Traceback (most recent call last):
  File "/home/pi/contrib/venv2/bin/black", line 10, in <module>
    sys.exit(main())
  File "/home/pi/contrib/venv2/lib/python3.7/site-packages/black.py", line 5, in main
    raise NotImplementedError("This is still being implemented.")
NotImplementedError: This is still being implemented.

Does it work if you specify the path to a single file instead of a directory when calling black?

Nope same error.

So now I simply formatted everything with an online black formatter... seems like the azure/CI pipeline is broken somewhere...


Render bug with Uploaded Track preview images

Firefox 48.0.1 on MacOS 10.11.6
Depending on the width of the browser window, the preview images for each track disapear.
Three attached images - narrower (works), medium (missing preview thumbnails), larger (works).

and a small video (wrapped inside a zip file so GH will allow me to upload it) showing the problem.
osv-render-thumbnails.zip

ps. Same problem in Chrome 51.0.2704.103 (64-bit) on MacOS

I noticed the problem yesterday.  I wasn't seeing images on Chrome and Firefox but they work properly on Android Chrome.  I increased the size of Chrome and Firefox because they were not full screen and the images appear.


Kubernetes Cluster IP performs Source NAT, contradicting docs: k8s.io/docs/tutorials/services/source-ip/

This is a Bug Report

Problem:
The documentation of when Kubernetes performs source NATing contradicts what I see in my deployment.
Here is a description of my issue:
I am having a networking issue in Kubernetes.
I am trying to preserve the source IP of incoming requests to a clusterIP service, but I find that the requests appear to be source NAT'd. That is, they carry the IP address of the node as the source IP rather than the IP of the pod making the request. I am following the example for cluster IPs here: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-clusterip
but I find that the behavior of Kubernetes is totally different for me.
The above example has me deploy an echo server which reports the source IP. This is deployed behind a clusterIP service which I request from a separate pod running busybox. The response from the echo server is below:
CLIENT VALUES:
client_address=<IP_ADDRESS>
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://<IP_ADDRESS>:8080/
SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001
HEADERS RECEIVED:
connection=close
host=<IP_ADDRESS>
user-agent=Wget
BODY

The source IP <IP_ADDRESS> belongs to the node. I expected to see the address of busybox which is <IP_ADDRESS>.
Does anyone know why SNAT would be enabled for a clusterIP? It's really strange to me that this directly contradicts the official documentation. (edited)
All of this is running on the same node. The node is running in iptables mode. I am using microk8s.
My microk8s version:
Client:
  Version:  v1.2.5
  Revision: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
Server:
  Version:  v1.2.5
  Revision: bb71b10fd8f58240ca47fbb579b9d1028eea7c84

Proposed Solution:
Either correct the documentation and properly describe how to disable source NAT for cluster IPs, or alter Kubernetes behavior to match the documentation
Page to Update:
k8s.io/docs/tutorials/services/source-ip/

microk8s 1.2.5

@rjw245 I'd like to understand the steps I'd follow to replicate this. If you provide sample command lines, eg curl http://thing/, mention where to run each of these.
Would you be willing to add that detail?

You can just follow the steps on the kubernetes documentation page I linked
https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-clusterip
First set up the source-ip-app deployment as shown at the top of the page. Then follow the instructions under the section for ClusterIP, starting with exposing the deployment with a ClusterIP service. It also describes how to test with BusyBox.

I've confirmed this does not happen with minikube. So make sure to use microk8s when testing.

I've submitted an issue to microk8s here:
https://github.com/ubuntu/microk8s/issues/956

It sounds like the documentation is correct and that Microk8s isn't behaving as expected. I've not checked that though.

I agree, perhaps this can be closed and we can investigate further in the microk8s bug

/close
Free free to reopen if we still see it a documentation issue.


labelsfilter: add reserved labels to default identity label list

Fix #14100
When using custom identity-relevant-label-list in whitelist mode, if
reserved:host (or reserved:.*) label is not included in the custom
list, the cilium_host endpoint will lose its reserved:host label.
When rolling back the default configuration, that is, removing the specified
label list and restarting the agent, cilium agent will raise errors like
following:
level=warning msg="Regeneration of endpoint failed" .. error="Exposing new BPF failed: invalid LXC MAC: invalid MAC address "
level=error msg="endpoint regeneration failed" ..  error="Exposing new BPF failed: invalid LXC MAC: invalid MAC address "

And subsequently, all pods' traffic on this node will be interrupted.
This is because the agent relies on this label to distinguish cilium_host
endpoint from normal endpoints, and the former has no lxcMAC.
Add reserved labels to the default label list could solve the problem.
Signed-off-by: ArthurChiao<EMAIL_ADDRESS>

@pchaigno please have a review, thanks!

@aanm  https://github.com/cilium/cilium/blob/master/pkg/endpoint/endpoint.go#L809
	// If host label is present, it's the host endpoint.
	ep.isHost = ep.HasLabels(labels.LabelHost)

@aanm https://github.com/cilium/cilium/blob/master/pkg/endpoint/endpoint.go#L809
	// If host label is present, it's the host endpoint.
	ep.isHost = ep.HasLabels(labels.LabelHost)

That is odd given that ep.HasLabels checks for all labels, including information labels. Can you try this patch instead?
diff --git a/pkg/endpointmanager/manager.go b/pkg/endpointmanager/manager.go
index ff5f530a2..10964ac6c 100644
--- a/pkg/endpointmanager/manager.go
+++ b/pkg/endpointmanager/manager.go
@@ -504,7 +504,7 @@ func (mgr *EndpointManager) AddHostEndpoint(ctx context.Context, owner regenerat
        // Give the endpoint a security identity
        newCtx, cancel := context.WithTimeout(ctx, launchTime)
        defer cancel()
-       ep.UpdateLabels(newCtx, epLabels, nil, true)
+       ep.UpdateLabels(newCtx, epLabels, epLabels, true)
        if errors.Is(newCtx.Err(), context.DeadlineExceeded) {
                log.WithError(newCtx.Err()).Warning("Timed out while updating security identify for host endpoint")
        }

cc @pchaigno

Hi @aanm, just tested your patch, the problem still exists.

Reserved labels shouldn't never be excluded for any reason since they are reserved. @ArthurChiao or @pchaigno where in the code are we checking if cilium_host is reserved?

To determine if an endpoint is the host endpoint, we also may check the security ID directly: https://github.com/cilium/cilium/blob/641c0f9b3072a014c9541ecaa4e00a2b24c98d97/daemon/cmd/datapath.go#L267
In some cases, we may rely on the security labels to set that security ID: https://github.com/cilium/cilium/blob/641c0f9b3072a014c9541ecaa4e00a2b24c98d97/pkg/identity/identity.go#L250-L251
:point_up: that might be why we get the reported bug when changing security-relevant label filters.
I think we should ensure the reserved labels are never excluded by our filtering logic (filterLabels). Probably best to warn the user if they try to exclude reserved labels. It's also not enough to add it to the list of default filters because that list will be overwritten when loading custom filters from a file (cf. readLabelPrefixCfgFrom).

It's also not enough to add it to the list of default filters because that list will be overwritten when loading custom filters from a file (cf. readLabelPrefixCfgFrom).

Nice point! @pchaigno

It would probably also make sense to apply @aanm's suggested change, even if it doesn't fix the problem at hand.

@ArthurChiao Are you still planning to work on this?

Yes, I'd like to, but i'm not sure what's the suggested way to fix this. Can you detail more? Thanks! @pchaigno

We need to ensure reserved:.* labels are never excluded by either (1) modifying filterLabels or (2) adding reserved:.* to validLabelPrefixes in ParseLabelPrefixCfg. The second approach is simpler and probably enough (users would still be able to explicitly exclude specific reserved labels, but at this point they are just looking for trouble).

I'll try the 2nd one and update this PR then. Thank you! @pchaigno

Hi @pchaigno , I have refactored the patch, with changes:

Logs some messages to help users understand how label-list-parsing works
Add reserved:.* to default label list
Warn users if they overwrite the default label list with --label-prefix-file but not include reserved:.* in both label file and --labels=<list>. I appended a test file in the commit message.

But there is one thing I still don't quite understand: you suggested adding reserved:.* to validLabelPrefixes in ParseLabelPrefixCfg(), instead of in defaultLabelPrefixCfg(), could you please explain more?
Any comments welcomed! Thanks!

test-me-please

test-me-please

Sorry for the delay in reviewing

It's OK, enjoy your holidays :smiley: @pchaigno
I've merged the suggested changes to the patch, thank you!

Sorry for the delay in reviewing

It's OK, enjoy your holidays :smiley: @pchaigno
I've merged the suggested changes to the patch, thank you!

My carelessness! Just fixed it :) @pchaigno

My carelessness! Just fixed it :) @pchaigno

test-me-please

test-me-please

Marking for backport into v1.8 and v1.9 as it can affect the host firewall on those versions.


Provide the default callback to enhance

Describe the problem
I've seen people try to add a loading state via enhance a couple times. They do something like this:
<form use:enhance={() => {
  loading = true;
  return () => {
    loading = false;
  }
}}>

However, they don't realize that once you return a callback from the function passed to enhance, the default behavior after a form submission no longer applies and you have to do the following yourself:

run invalidateAll if the result was a success
run applyAction if the current location matches or the result was a redirect/error

This logic currently happens in the default callback.
This means if you want to extend enhance, e.g. to add a loading state, you have to duplicate that logic, even if your customizations are minimal:
<form use:enhance={() => {
  loading = true;
  return ({action, result}) => {
+    if (result.type === 'success') {
+      await invalidateAll();
+    }
+
+    if (
+      location.origin + location.pathname === action.origin + action.pathname ||
+      result.type === 'redirect' ||
+      result.type === 'error'
+    ) {
+      applyAction(result);
+    }

    loading = false;
  }
}}>

(You could remove the check around applyAction and always apply it, but I don't like how that diverges you from the default enhance behavior, perhaps without even realizing it).
You can abstract this into a custom action that wraps enhance to reduce the boilerplate, but you won't get any updates that are made to the default callback behavior, e.g. the changes in https://github.com/sveltejs/kit/pull/6828.
All you wanted to do was toggle a loading state, but now you have to maintain the default enhance behavior as well.
Of course, there are cases where you want to customize the invalidate or applyAction behavior. However, I suspect those cases are more the exception than the norm.
Describe the proposed solution
Provide an additional argument to the callback returned from enhance. Name is bikesheddable, but for the purposes of this issue I'll call it defaultBehavior. The implementation of this method is the same as fallback_callback. People who want to add some logic after a form submission and preserve the default behavior could call this method instead of recreating the logic themselves. This way, if the default behavior is updated, custom actions calling this method get it for free.
The example above would then look like this:
<form use:enhance={() => {
  loading = true;
  return ({action, result, defaultBehavior}) => {
    defaultBehavior();
    loading = false;
  }
}}>

If people want to write enhance callbacks that always do the default callback behavior, then they could create their own action that calls defaultBehavior without having to track any updates to the behavior.
Alternatives considered
One alternative is to keep the status quo and require the user to duplicate the default behavior if desired.
Another alternative is to call fallback_callback automatically after the returned callback runs, but also pass a cancel arg to the returned callback that would prevent that behavior if desired.
<form use:enhance={() => {
  loading = true;
  return ({action, result, cancel}) => {
    // invalidate & applyAction called automatically, unless cancel is called
    loading = false;
  }
}}>

This makes the default case simpler. However, this has a few downsides:

it's a breaking change for people returning a callback that didn't want invalidateAll and applyAction called
because we need to run the provided callback to find out if cancel was called, there isn't a good way to run code after invalidation & applying the action has completed. Supplying defaultBehavior gives the user control over when invalidation happens.

Importance
would make my life easier
Additional Information
No response

Although there is already a PR,
But I like cancel() better, only I would call preventDefault().
This is more like the standard behavior of HTML element handlers.
When you describe your own behavior you have a lot of code, it's not difficult to add one more argument and break the default behavior. But when you just need to set a single variable, with defaultBehavior() you have to specify an extra argument in the destructor and make an extra function call every time.

I'm not sure I follow. When you return a callback function, the default behavior already is invoked. It's only that when you provide a callback function, that you lose that default behavior without an easy way to get it back. This feature request proposes giving the callback function another argument so you can easily get that default behavior back.

Of course, preventDefault() is a breaking change, but I can try to do PR.
There is another way that may not be breaking.
<form use:enhance={( done ) => {
  loading = true;
  done(() => {
      loading = false;
  })
}}>

We will not rename cancel to preventDefault - this sounds unrelated to this issue.
The done function is interesting, but what happens if you call it and return a callback function?
<form use:enhance={( done ) => {
  loading = true;
  done(() => {
      loading = false;
  });
  return () => {
     // what happens here? What happens with done?
  }
}}>

I think the semantics are more clear and easier to understand with an additional argument to the callback function

We will not rename cancel to preventDefault - this sounds unrelated to this issue.

I meant the other cancel() .
The one specified in this issue in Alternatives considered, which is proposed to be passed to the result callback.


Remove preinstall script

This pull request removes the preinstall script from the package.
Closes: #315

coverage: 100.0%. remained the same
when pulling fbab1eff8a1b62913ef2729be408d7b9633d375b on remove_preinstall_script
into 231ca43aa49f49e74f6592480a1506dc9f554df4 on master.


.NET Developers have a first class experience working with GraphQL

GraphQL has been gaining traction over the last few years across a variety of platforms. We plan to investigate the space and find ways to improve the experience with .NET. This will involve working with the community as well as internal teams to understand and support the existing ecosystem. It may also involve specific investment from Microsoft, either in the form of contributions to existing work or in developing complimentary pieces in the Microsoft stack. Examples of this include but are not limited to:

IntelliSense in Visual Studio products for .graphql schemas and definitions
Interactive exploration of GraphQL endpoints similar to what GraphiQL provides
Services for exposing GraphQL endpoints and supporting schema stitching

These are not commitments but lines of exploration the team will investigate to enhance the GraphQL experience.

[Using "team:libraries" for stuff that touches the core libraries team]

[Using "team:libraries" for stuff that touches the core libraries team]

Really excited to see this. I have heard of one organisation that is moving to node over .net because .net's graphql offerings does not have feature parity with the apollo implementation in node.

Bulk closing .NET 6 epics and user stories. If you think this issue was closed in error, please reopen the issue and update it accordingly.


class TimeDeltaField

new class handles datetime.timedelta data

Duplicate of #649. Thank you, but I do not want to add custom handling of timedeltas to the core library. Feel free to use this field in your own code.


fix(notify): correctly filter those recipients who join the domain

这个 PR 实现什么功能/修复什么问题:
fix bug Introducted from #10650

[x] 功能、bugfix描述
[x] 冒烟测试

是否需要 backport 到之前的 release 分支:

release/3.7

/area notify
/cc @zexi

Codecov Report

Merging #10972 (5a39fc1) into master (c26ef91) will increase coverage by 5.37%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master   #10972       +/-   ##
===========================================
+ Coverage    5.81%   11.18%    +5.37%
===========================================
  Files         899      147      -752
  Lines      138815    16932   -121883
===========================================
- Hits         8071     1894     -6177
+ Misses     130038    14845   -115193
+ Partials      706      193      -513

Flag
Coverage Δ

aFlag
11.18% <ø> (+5.37%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/compute/models/globalvpcs.go

pkg/multicloud/ucloud/host.go

pkg/util/stringutils2/i18n.go

pkg/cloudprovider/resources.go

pkg/compute/models/groupguests.go

pkg/compute/models/timer.go

pkg/util/samlutils/encrypt.go

pkg/multicloud/ucloud/utils.go

pkg/compute/models/loadbalancerlisteners.go

pkg/baremetal/utils/disktool/disktool.go

... and 742 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c26ef91...5a39fc1. Read the comment docs.


Should be able to use env var to set spring.cloud.stream.bindings.input

I'd like to be able to use:
-e spring_cloud_stream_bindings_input=myqueue

when deploying to Docker/Lattice, this works for other Spring Boot config properties like
-e spring_redis_host=myhost

That should already work since it's exactly what the ReceptorModuleDeployer does:
https://github.com/spring-cloud/spring-cloud-data/blob/master/spring-cloud-data-module-deployers/spring-cloud-data-module-deployer-lattice/src/main/java/org/springframework/cloud/data/module/deployer/lattice/ReceptorModuleDeployer.java#L69
The parameters contain the binding values as populated here:
https://github.com/spring-cloud/spring-cloud-data/blob/master/spring-cloud-data-core/src/main/java/org/springframework/cloud/data/core/ModuleDefinitionBuilder.java#L87
The only difference is those are dot-delimited, but earlier they used underscores yet with uppercase (if that makes a difference).

These work:
java -Dspring.cloud.stream.bindings.output=myqueue -jar target/time-source-1.0.0.BUILD-SNAPSHOT-exec.jar

java -jar target/time-source-1.0.0.BUILD-SNAPSHOT-exec.jar --

spring.cloud.stream.bindings.output=myqueue
but these don't:
SPRING_CLOUD_STREAM_BINDINGS_OUTPUT=myqueue java -jar target/time-source-1.0.0.BUILD-SNAPSHOT-exec.jar

spring_cloud_stream_bindings_output=myqueue java -jar target/time-source-1.0.0.BUILD-SNAPSHOT-exec.jar

This is reproducible, but specific to spring-cloud-stream-modules, i.e. running the SCSM time-source:
SPRING_CLOUD_STREAM_BINDINGS_OUTPUT=foo java -jar target/time-source-1.0.0.BUILD-SNAPSHOT-exec.jar
will display the described behaviour, however, running the sample from within spring-cloud-stream:
SPRING_CLOUD_STREAM_BINDINGS_OUTPUT=qux java -jar target/spring-cloud-stream-sample-source-1.0.0.BUILD-SNAPSHOT-exec.jar
will work as expected.

Seems like the key difference is whether the value is set outside the OS environment. In other words, the SCS examples that still it set via the application.yml will allow overriding via environment, the others won't. Since all the other Boot overriding mechanisms (System properties, command-line arguments) work, I would say it's a Boot issue.

Ok, it seems that this is caused by the way in which Spring bean OS environment binding happens for nested properties (binding in this case). The SystemEnvironmentPropertySource will be strictly queried for SPRING_CLOUD_STREAM_BINDING (which does not exist as such), and does not support nesting. A quick solution would be to replace the current defaulting mechanism that returns a value if the property is not set with a prepopulated PropertySource (which will be presumably overriden correctly).

For the record, I'm trying to get the equivalent to http | log running on cf and this doesn't work there either.  The following was tried (stream-cloud-streams-bus is the name of my redis service on CF):
$ cf push cc-http -p http-source/target/http-source-1.0.0.BUILD-SNAPSHOT-exec.jar --no-start
$ cf bs cc-http stream-cloud-streams-bus
$ cf se cc-http spring_cloud_stream_bindings_output http_out
$ cf start cc-http
$ cf push cc-log -p log-sink/target/log-sink-1.0.0.BUILD-SNAPSHOT-exec.jar --no-start
$ cf bs cc-log stream-cloud-streams-bus
$ cf se cc-log spring_cloud_stream_bindings_input http_out
$ cf start cc-log
```

It turns out that for non-String property value mapping (list, map etc.,) spring-boot's RelaxedDataBinder would consider mapping the property name tokens into data-binding [only if the property names are delimited by .] (https://github.com/spring-projects/spring-boot/blob/master/spring-boot/src/main/java/org/springframework/boot/bind/RelaxedDataBinder.java#L502)
Hence, for this case, we need to specify the property this way:
SPRING_CLOUD_STREAM_BINDINGS.OUTPUT=myqueue java -jar target/time-source-1.0.0.BUILD-SNAPSHOT-exec.jar
spring_cloud_stream_bindings.output=myqueue java -jar target/time-source-1.0.0.BUILD-SNAPSHOT-exec.jar

Note the . for bindings.output in both the cases for the property name after prefix spring.cloud.stream on ChannelBindingProperties.
Even if we set the system property this way:
java -Dspring_cloud_stream_bindings_output=myqueue -jar target/time-source-1.0.0.BUILD-SNAPSHOT-exec.jar

that wouldn't work because of _ for bindings_output.
I believe it makes sense to have such a differentiation (to have . delimited property names for collections etc.,) as the property name itself could have _ for direct string mapping.

but that means it's unusable as a standard env var:
$ export FOO_BAR.oops
-bash: export: `FOO_BAR.oops': not a valid identifier

Fixed by @mbogoevici with commit 83123c3b97ee731e448d99df26b0f0bf8e517c81


What are use-cases for more binaries? (32-bit and ARM)

Quick Summary: Some people want to use Elm on 32-bit machines.
This meta issue is meant to gather relevant requests to get an idea of what the situation is where folks are using that kind of computer:

https://github.com/elm/compiler/issues/1878 - 32-bit Windows at Work
https://github.com/elm/compiler/issues/1962 - 32-bit Linux
https://github.com/elm/compiler/issues/2010 - ARM64 - Rasberry PI 4
https://github.com/elm/compiler/issues/2038 - 32-bit C code used with npm on Windows?
https://github.com/elm/compiler/issues/2117 - ARM in cloud computing and CI

Open a brand new issue if you would like to tell us about more scenarios.

The closest thing we have to "build from source" instructions is the Dockerfile for creating our portable linux binary. It only aims to work on a specific version of Alpine Linux, but perhaps it can be useful to people trying to build binaries for other platforms in the meantime.
https://github.com/elm/compiler/blob/master/installers/linux/Dockerfile


broadcast_to method

CableReady::Channels exposes a public broadcast_to(resource) method, which is used to broadcast to an ActionCable channel which makes use of the stream_for method. When using broadcast_to, we pass the channel constant instead of a String-based identifier.
class DocumentChannel < ApplicationCable::Channel
  def subscribed
    stream_for Document.find(params[:document_id])
  end
end

class Document< ApplicationRecord
  include CableReady::Broadcaster
  after_save do
    cable_ready[DocumentChannel].morph({selector: "#document", html: self.content})
    cable_ready.broadcast_to(self)
  end
end

A resource must be passed to the broadcast_to method. ActionCable builds a subscription handle based on the name of the channel and a unique hash of the resource.
When using CableReady, it is possible to have multiple identifiers "on the go" at one time, collecting operations to be run.
Both broadcast and broadcast_to methods to support an optional identifiers parameter splat (h/t: @StephenFiser). Default behaviour is unchanged from current behaviour, which is that all channels are broadcast one at a time. You can pass zero, one or several identifiers.
cable_ready.broadcast_to User.find(1), FooChannel, BarChannel

broadcast will ignore Channel constant identifiers. broadcast_to will ignore String identifiers. This means you can have String and Channel identifiers "on the go" at the same time, and they will not conflict.
cable_ready['test1'].morph()
cable_ready['test2'].morph()
cable_ready[FooChannel].morph()
cable_ready[BarChannel].morph()

cable_ready.broadcast # broadcasts to test1 and test2
cable_ready.broadcast('test1') # broadcasts to test1
cable_ready.broadcast(FooChannel) # ignored

cable_ready.broadcast_to(Foo.find(1)) # broadcasts to foo:Z2lkOi8vdGhlLWRlcGFubmV1ci9Qcm9wb3NhbC8x AND bar:Z2lkOi8vdGhlLWRlcGFubmV1ci9Qcm9wb3NhbC8x which is valid but probably not what you want
cable_ready.broadcast_to(Foo.find(1), FooChannel) # broadcasts to foo:Z2lkOi8vdGhlLWRlcGFubmV1ci9Qcm9wb3NhbC8x
cable_ready.broadcast_to(Foo.find(1)) # broadcasts to foo:Z2lkOi8vdGhlLWRlcGFubmV1ci9Qcm9wb3NhbC8x AND bar:Z2lkOi8vdGhlLWRlcGFubmV1ci9Qcm9wb3NhbC8x which is valid but probably not what you want
cable_ready.broadcast_to(Foo.find(1), FooChannel, BarChannel) # broadcasts to foo:Z2lkOi8vdGhlLWRlcGFubmV1ci9Qcm9wb3NhbC8x AND bar:Z2lkOi8vdGhlLWRlcGFubmV1ci9Qcm9wb3NhbC8x which is valid but probably not what you want
cable_ready.broadcast_to('test1') # ignored

Finally, both broadcast and broadcast_to now have an optional named last parameter called clear, which defaults to true. Setting it to false prevents CableReady from clearing the channel operations data after a broadcast, allowing it to potentially be sent multiple times or to multiple resources.
cable_ready[ImageChannel].morph()
cable_ready.broadcast_to(image1, clear: false)
cable_ready.broadcast_to(image2) # broadcasts the same morph to image2

This is pretty great. I'm looking forward to testing it.

@leastbad Will you add some usage examples to the comments on this PR so we have a paper trail for everyone to follow? It will also help shape the thinking around how to best document this new feature.

I spent a lot of time thinking about what additional information and ultimately, examples would be most helpful. Working with @stephenfiser on #42 helped me clarify what's missing or non-obvious.
In the same way that a lot of people were inspired seeing how you use CableReady broadcasts inside Reflex actions, I believe that there are similarly conceptual blocks keeping people from imagining new patterns and approaches with AC and CR that are a good example away from a lot of people suddenly being really excited about a new way to design these sorts of interfaces.

Assuming that all of your channel subscriptions are all importing the same memoized consumer.js there appears to be little if any overhead on the client or server to being subscribed to an arbitrary number of streams. Today, most folks using AC are thinking of channels and subscriptions as 1:1. The power of 1:N is still a "locked achievement" in gaming terms. I want to show how clever use of Stimulus' MutationObserver-powered connect() works really well with people subscribing to multiple instances of a channel; broadcast_to is key to unlocking that thinking.

The broadcast_to method that is inherited by every Channel class is really just the entry-point for a useful abstraction. Technically, you could call the same non-public resource serializer method that broadcast_to calls and create a subscription channel with the same identifier token using just stream_from. Ultimately it's the same data being sent the same way. However the AC framework has useful tooling (even down to the logging) which differentiates between generalized broadcast and resource-specific broadcast_to. It makes good sense to continue the same abstraction pattern started by AC by providing a broadcast_to in CR that expects a model.

I'm going to offer two examples which are really use cases for this approach.
Always up-to-date article
If you've read documentation hosted on GitBook, you've probably noticed that the document you're reading will change in real-time as the author makes changes to it. While in this case it's after-commit, there's no particular reason that this approach couldn't stream the current state of the document while the author is typing into it, similar to the real-only mode on Google Docs.
I usually recommend that people don't call CableReady in ActiveRecord callbacks directly, prefering instead of execute ActionJobs. However, I want to keep this light and tight, so please use your imagination.
class DocumentChannel < ApplicationCable::Channel
  def subscribed
    stream_for Document.find(params[:document_id])
  end
end

class Document< ApplicationRecord
  include CableReady::Broadcaster
  after_save do
    cable_ready['document'].morph({selector: "#content", html: self.content})
    cable_ready.broadcast_to(self)
  end
end

import { Controller } from 'stimulus'
import consumer from './consumer'

export default class extends Controller {
  connect () {
    consumer.subscriptions.create(
      {
        channel: 'DocumentChannel',
        document_id: this.element.dataset.documentId
      },
      {
        received (data) {
          if (data.cableReady) CableReady.perform(data.operations)
        }
      }
    )
  }
}

<div data-controller="document" id="document" data-document-id="<%= @document.id %>">
  <%= @document.content %>

More to follow

One of my go-to techniques is to eschew the entire app/javascript/channels folder entirely, opting to move consumer.js to a neutral location and create Stimulus controllers that are also ActionCable consumers. This allows me to have access to the Stimulus ecosystem, including not just the controller instance but a direct path to this.application and of course, the opportunity to initiate StimulusReflex activity.
When combined with the CableReady broadcast_to method, we can move past the familiar 1:1 associations we see in a resource view such as Article id#1 and start to imagine UIs composed of many instances of a combined Stimulus+consumer class - including elements that have been injected into the page dynamically.
In other words, once you have a connection established, it's now easily possible to present a vast collection of live components where each is subscribed to its own identifier but all data is still moving via the same channel, which itself could be one of several channels sharing the same consumer (and single WS connection to the server). This is incredibly powerful, as it cuts to the core value proposition that React was created to solve: how do you deliver a newsfeed where every element subscribes to its own feed of likes and comments? Well, with Stimulus and CableReady#broadcast_to, it's a few dozen lines of code.
Collection of images which display real-time voting

  <% @images.each do |image| %>
  <div id="image_<%= image.id %>" data-controller="image" data-image-id="<%= image.id">
    <img src="<%= image.path %>">
    ImageReflex#like"><%= images.likes %></span

  <% end %>

import { Controller } from 'stimulus'
import consumer from './consumer'

export default class extends Controller {
  connect () {
    consumer.subscriptions.create(
      {
        channel: 'ImageChannel',
        image_id: this.element.dataset.imageId
      },
      {
        received (data) {
          if (data.cableReady) CableReady.perform(data.operations)
        }
      }
    )
  }
}

class ImageChannel < ApplicationCable::Channel
  def subscribed
    stream_for Image.find(params[:image_id])
  end
end

class ImageReflex < StimulusReflex::Reflex
  include CableReady::Broadcaster
  def like
    image = Image.find(element.id)
    image.likes.increment
    image.save
    cable_ready['document'].morph({selector: "#image_#{image.id}", html: ApplicationController.renderer.render(partial: "images/image", locals: {image: image}})
    cable_ready.broadcast_to(image)
  end
end

I'm doing this without the benefit of a test environment, so assume that this is pseudocode... but hopefully it gets the idea across. You could have an arbitrary number of these images on your page, updating counters in real-time, with basically zero client code much less client state. So the whole premise of needing React and Redux to make this viable is absurd.
Please let me know if any of this needs "more".

Just as a heads-up, I woke up today thinking I might have a slight refactor in mind. Stay tuned!

Alright, I made a significant change to the way this PR works and it is far better than yesterday. Ready for prime time!

@leastbad what's the reason for calling @channels[identifier].send(:broadcast_to, model) as opposed to just @channels[identifier].broadcast_to(model)?

Also - would it make sense to do this?
def broadcast_to(model, identifiers = [])
  receiving(identifiers).reject { |identifier| identifier.is_a?(String) }.each do |identifier|
    @channels[identifier].broadcast_to(model)
  end
end

def receiving(identifiers)
   identifiers.any? ? identifiers : channel_identifiers
end

def channel_identifiers
  @channels.values.map(&:identifier)
end

This would allow you to broadcast to a one identifier or a subset of your identifiers as opposed to just broadcasting to either one or all. It also guards against strings accidentally being passed in via the identifiers array.

@StephenFiser you are absolutely right about not needing send. That was just a vestigal holdover from a long-gone iteration.
I'm probably too sleepy to properly parse what you're going for in your second comment. It's not clear if you imagine this being just for broadcast_to or broadcast as well, for example.
If you consult what's currently on master - which is largely unchanged since the project launched - you'll see that I'm actually giving folks some flexibility that doesn't currently exist in the library. I'm not going to be changing the API of the existing broadcast method, which is already in production on many apps. Adding an optional parameter is as radical as I can be. Given that I want to keep the API for broadcast_to as similar to broadcast as possible, it's not likely that this new mechanism will diverge far from where it started.
It's also important to say that I've never actually seen a scenario where more than one identifier is in play at once in the wild, ever, period. If I was designing the library from scratch today, I wouldn't have allowed it. So while I like the idea of giving people limitless flexibility, this PR already dramatically expands the flexibility of the library far beyond what's possible today without breaking compatibility or frankly, giving people another thing that they feel they have to learn or think about.
So yeah, I'm not yet convinced on the second point. Still listening, though. I just suspect most folks will be calling with 0 or 1 parameter (for resource_to) 95+ of the time.

@leastbad I just noticed a simple way to make the feature "mathematically complete" in some sense where you can broadcast to one, some, or all channels. Right now in both broadcast and broadcast_to it looks like it's either one or all. So yes, you'd probably need to do the same for broadcast. Just a thought.

I really like this. Let me see what I can do.

I just ran with two of @StephenFiser's good ideas, and I think the codebase is better for it now.
The whole process flows much more naturally now. Having an optional splat of identifiers allows for flexibility and automatically filters out params of the wrong type.

I just made a commit that adds a trailing named argument called clear, defaulting to true, to both broadcast and broadcast_to.
Setting clear: false has the effect of not clearing out the channel operations data after a broadcast. This will conserve resources when sending the same broadcast to multiple resources.
I will add information on this to the PR description.

@leastbad at first glance this looks really great. Hopefully going to have some time to play with it later tonight!

I ran through a quick demo app build out with this branch. I have to say it works amazingly well. I assume at least part of that feeling is that I'm new to cableready, so there's still a bit of wow factor with that alone. That said, here's what I did in a nutshell:

Set up a basic projects scaffold with name and text
Added two channels, one for an individual project and one for a "dashboard" (projects index)
Installed cable ready as prescribed in the docs and switched to your branch (@leastbad)
Iteratively added this to project.rb (started with just one channel, then went to both)

class Project < ApplicationRecord
  include CableReady::Broadcaster
  after_save do
    # for dashboard
    cable_ready[DashboardChannel].morph({
      selector: ".project-#{id}-name",
      html: "#{name}	"
    })

    # for project show
    cable_ready[ProjectChannel].morph({
      selector: ".project-name",
      html: "#{name}"
    })

    cable_ready.broadcast_to(self, DashboardChannel, ProjectChannel)
  end
end

Quick (unlisted) video showing it in action: https://youtu.be/GToUDbabdtw

That’s rad! I’m thrilled to see this working and jealous that I rarely get to actually play with the tools. 🍿

Huzzah!


Mapping UI Usability and Guidance Issues

From Nate Argo.
How would you map properties where the data is not in the immediate class, but a few layers deep?
For example, the value for this property ("Electronic Mail AddressType") would be in Organization > address > Place > targetContactPoint > ContactPoint > contactType.
It looks like "Transformed" is the best fit, but I didn't realize the comment functionality mentioned in the documentation existed until later (see next note).

In general yes, it would be as a transformation.


Pay for Pull - wiki updates

page: https://github.com/BCDevExchange/rapid-adoption/wiki

"the experiment" content should be moved to the top as it is a backgrounder to the process.
the content about the evaluation and selection should moved to the "What is Pay for Pull" section as it's directly related to payment
the terms section should have a link to the actual full Pay for Pull terms.

thanks for the updates Todd!


[NIFI-3360] Modularize Canvas

Most of this PR simply adds UMD (Universal Module Definition) boilerplate around NIFI's existing modules and re-orders the loading of the scripts on the page to match dependencies. However some refactoring was required.
Changes to be aware of:

Created new method nf.Graph.reload() to be called from nf.CanvasUtils.reloadConnectionSourceAndDestination since nf.Graph module has references to the canvas graph component controllers
moved nf.CanvasUtils.enterGroup to nf.ProcessGroup.enterGroup
nf.CanvasUtils.editable(...) now has nf.Connectable and nf.Draggable passed as args since the canvas graph components now have private references to the nf.Draggable and nf.Connectable controllers and can pass them along (rather than injecting nf.Connectable and nf.Draggable into nf.CanvasUtils)
nf-canvas-bootstrap.js leverages the Inversion Of Control design pattern to remove circular dependencies and allow the Nifi js modules to be unit tested using mocks of the injected controllers
nf.CanvasUtils now has a reference to the nf.Canvas (see nf-canvas-bootstrap.js) and now acts as a proxy for calls to the nf.Canvas
nf.CanvasUtils now has a reference to the nf.Graph (see nf-canvas-bootstrap.js) and a new method nf.CanvasUtils.getComponentByType() that allows dependent modules to get canvas graph component controllers

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Will review...

Thanks @scottyaslan. This has been merged to master!


Option to process data channels

An OpenColorIO ColorSpace has a property called isdata that is intended to be set to "true" for non-RGB image data such as alpha channels, normals, Z-depth, position, object IDs, etc.  OCIO gives these channels special treatment by not applying most color transforms.  For example, if the source or destination of a ColorSpaceTransform is a ColorSpace whose isdata is set to true, no processing is applied -- it is effectively a no-op.  This is usually a very sensible thing to do since it would usually be undesirable to apply color processing to data channels.
However, as we consider how to display data channels on a monitor, the situation is less clear cut.  Indeed, there are already some exceptions along these lines in OCIO.  For example, the DisplayTransform in some cases will apply certain operations such as the LinearCC and DisplayCC.  Even in these cases though, DisplayTransform does not apply the actual display/view transform.
In most applications, data channels are typically displayed simply by sending the values to the display unmodified, effectively considering them as being in the monitor color space.  For alpha channels in particular, this generally works quite well.  For other data such as normals (which typically have negative values) and Z-depth (which may span a large range much above 1) it could be argued that some processing would make the data easier to visualize.
The real issue however is the arrival of HDR displays.  Many of these displays are calibrated to have an ST-2084 (PQ) response.  This means that sending a normalized value of 1 to the display will be a request to display that pixel at 10,000 nits.  Sending an alpha channel without color management to such a display results in a very bad user experience!
In situations such as this, it would be very useful for OCIO to allow applications the ability to request that data color spaces are processed.
The proposal is to add methods setDataBypass and getDataBypass to the ColorSpaceTransform and DisplayTransform classes.  The default would be true, which would correspond to the current behavior -- data color spaces do not get processed.  However, applications would be able to set it to false for situations as described above where it is actually necessary to process data color spaces.

Good point. One tricky aspect to this is that even if you wanted to display a data channel through the PQ, you would only want to apply the device encoding portion of the display transform, but bypass any creative "look". You would essentially want to preserve the relative appearance of the data on the display. Perhaps it should still bypass the view transform, but not the display colorspace? Or at least have the option to do so?

Michael, my thinking was that the config would define a view specifically for a type of data (such as an alpha channel).  For example, it could make the alpha channel appear as it would have on an sRGB monitor without color management.  So the entire display/view processing could be applied in that case.

I think the view for data is a much needed result, and look forward to this implementation.
In your proposal Doug, if I am reading you correctly, I am guessing that a data’s view transform would honour the display colourimetry, thereby rendering the data in a somewhat “display agnostic” fashion?
Terrific stuff folks.

Ok, thanks for the explanation Doug. That makes sense.

Troy, yes, it would be far more independent of the display than bypassing color management.

Closing as this has now been implemented.


Update README.md

A copy paste of the line is causing an error. So, the change is for lazy people like me. :)

Good catch! The right option here though is ~> 1.0-pre, not >= 1.0.0-pre. Updated as such and will be merged directly!
Thanks for the PR!


Externally deleted favicons files are not fetched again.

Actual behaviour
If the favicons files are deleted externally, they are not fetched again as the database entry persists.
Expected behaviour
If a favicon file is absent it should be fetched again.
An alternate solution would be to allow forced refresh of favicons.
Note: this problem is very unlikely to happen during normal use. I forgot to copy the favicons over after changing server which is when I noticed this behaviour. Easily fixed in my case, but I think a way to refresh favicons without re-adding the feeds would be handy regardless.
Steps to reproduce
Delete favicon file.
Configuration

Miniflux version: master

This issue have been fixed in the current development branch.
Thanks


Linux and Xbox controller mappings are all -1 in controller Mapping class.

Issue details
The source code for the class Xbox.java only has correct mappings for the Mac environment. All other platforms have all button mappings listed as -1.
Reproduction steps/code
Attempt to poll a controller for input on Windows using the constants defined in Xbox.java.
Version of LibGDX and/or relevant dependencies
1.9.2
Platforms

Windows
Linux

Relevant Code
 if (SharedLibraryLoader.isWindows) {
            A = -1;
            B = -1;
            X = -1;
            Y = -1;
            GUIDE = -1;
            L_BUMPER = -1;
            R_BUMPER = -1;
            BACK = -1;
            START = -1;
            DPAD_UP = -1;
            DPAD_DOWN = -1;
            DPAD_LEFT = -1;
            DPAD_RIGHT = -1;
            L_TRIGGER = -1;
            R_TRIGGER = -1;
            L_STICK_VERTICAL_AXIS = -1;
            L_STICK_HORIZONTAL_AXIS = -1;
            R_STICK_VERTICAL_AXIS = -1;
            R_STICK_HORIZONTAL_AXIS = -1;
        } else if (SharedLibraryLoader.isLinux) {
            A = -1;
            B = -1;
            X = -1;
            Y = -1;
            GUIDE = -1;
            L_BUMPER = -1;
            R_BUMPER = -1;
            BACK = -1;
            START = -1;
            DPAD_UP = -1;
            DPAD_DOWN = -1;
            DPAD_LEFT = -1;
            DPAD_RIGHT = -1;
            L_TRIGGER = -1;
            R_TRIGGER = -1;
            L_STICK_VERTICAL_AXIS = -1;
            L_STICK_HORIZONTAL_AXIS = -1;
            R_STICK_VERTICAL_AXIS = -1;
            R_STICK_HORIZONTAL_AXIS = -1;
}

Bit of a tricky one.  I think @badlogic may have pushed that xbox binding for mac for the lwjgl3 controllers, which will probably give different results to the desktop controllers backend.  Not sure though, and I don't have a mac to test with.
I've had a quick look at the binding for windows, and the dpad is reported as PovDirections, so that wouldn't be compatible with the current state in the bindings.  The lwjgl3 controllers extension behaves a bit better, and each direction has its own int, however not all buttons work, left axis push in doesn't seem to register.  Also interestingly, with desktop controllers the native windows game controller doo-da kicks in with the 360 home button, but doesn't with lwjgl3.
Anyway, not sure of the direction we'd want to take with this.  We could have introduce a "is lwjgl3" into the binding, or just add a new binding for the desktop version of controllers (rather than the lwjgl3), but that still leaves us with the issue of PovDirection not being compatible with the current state of bindings.  Also on desktop controllers on windows you would have to query Pov rather than query button for dpad, on mac/linux this may even be different.

Would it make more sense for the controller mapping class to be in the controllers-lwjgl3 package?

That would solve one of the issues, there is still the issue with the Pov vs individual buttons.  Perhaps bindings could be a little more involved, but I guess this is only an issue with the desktop controllers so I'm not sure if it's even worth it. I'd be happy with just pushing lwjgl3 going forwards, in which case please feel free to create a PR for the bindings for windows/linux for the xbox controller!

LWJGL3 specific mappings for this were already accepted ( #5090 ), and the current workflow can be summarized as if your dpad mappings come in as -1 then use POV instead, which while not terribly convenient is how the controller is presented across different platforms. Given that I think this issue can probably be closed, unless I've missed something.

Hi, please take a look at gdx-controllers v2 prerelease snapshot. It should solve your issue.


Add scoped sessions to TestHelpers module

Hello!
I've added support for scoped sessions to the TestHelper module. Many applications use the user_session or admin_session, and the code in this branch should allow people to cover more of that functionality inside of their tests.
I've included several basic tests in test_helpers_tests.rb to demonstrate usage and guarantee scoped sessions can only be used in tests once a sign_in has occurred.
Thanks for the great Gem! Hope my additions are helpful.

Thank you but I would push for those to be accessed rather from the controller instead of extending the test API. :) Glad you are enjoying Devise!

Hi Jose,
What you describe is exactly my situation, maybe my explanation for adding scoped sessions to tests was insufficient.
I am currently using user_session in several of my controllers and would not only like to assert that functionality inside of my functional tests, I cannot get my test cases to pass (as planned) without adding data to user_session inside of my test cases.
At present, user_session is not available from ActiveController::TestCase even when adding the module via include Devise::TestHelpers. Maybe this test case below will make more sense.
demo_controller_test.rb
class DemoControllerTest < ActionController::TestCase
    include Devise::TestHelpers

    test "will fail because user_session raises NoMethodError" do

         # without my branch, this call to user_session is not possible, NoMethodError is raised.
        user_session['required_property'] = true

        # index requires user_sessions['required_property'] or else the action will respond with failure
        get :index
        assert_response :success
    end

end

demo_controller.rb
class DemoController < ActionController::Base

    def index
        if user_session['required_property'].nil?
            redirect_to error_path and return
        end
    end

end

Is there another way to populate user_session from a test case without having to basically implement integration testing to flow through my usual login and session initialization process? That seems very cumbersome when I can just set the user_session manually to provide accurate test coverage for my controllers.
Thanks again!
Jason

I misunderstood @josevalim's response above and was fortunate enough to have @lucasmazza point out that scoped sessions can be easily accessed via the controller instance being tested. Simply referring to @controller.user_session in your Rails / MiniTest TestCase permits the session handling required to get full test coverage. Thanks again to both of you guys.


Webpage inconsistent with repository version

The homepage still carries the very unfortunate

This project is currently migrating.

message, which was already fixed in the repository version. Probably you would somehow need to automate releasing the homepage.

Done. I also updated the y-* dependencies

:clap:


Poor performance of public_address() and public_view_key()

I'm using your project (good work by the way!) to generate hundreds/thousands of addresses and get their public address and private keys. But their generation is, unexpectedly, very slow. Some basic tests in python using timeit on Intel Core i7-3820 @ 3.8GHz:
In [1]: from monero import seed

In [2]: s = seed.Seed()

In [3]: %timeit s.public_address()
1.08 s ± 9.99 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [4]: %timeit s.public_view_key()
544 ms ± 9.18 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [5]: %timeit s.public_spend_key()
540 ms ± 3.53 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Any reason for this poor performance and any way/fix to make it faster? Generation of thousands addresses and keys on regular basis with that speed (basically one  every 1 to 2 seconds) is slow.

Little updated.
Ended up using the address generator in multiprocessing.Pool to speed it up and also wrapped it in asyncio as all my code was asyncio. With these I could efficiently work with address generation.
As a side note. The issue originated from development of short  asyncio program for stress-testing openmonero. The script utilizes monero-python and it is here is someone is interested in it:
https://github.com/moneroexamples/openmonero/blob/devel/scripts/batch_import_test.py

I managed to go from seed->public address in 0.091552 seconds using wallet_api2 directly and making sure major:minor lookahead was set to 1:1.
This is offtopic for monero-python however.

With 9aee6564e74862f041fa8baa1b7702f4d9cb3c70 I have changed the ed25519 implementation to one that cut the test suite running time by two orders of magnitude.
Integrating low-level C/C++ code from monero or libsodium could potentially cut the times further, however we still stay in pure Python world, so I stopped looking for other solutions ATM.


Gitleaks: ignore test slack tokens

Proposed changes:

Configure gitleaks to ignore test slack tokens

Status (please check what you already did):

[ ] added some tests for the functionality
[ ] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[ ] reformat files using black (please check Readme for instructions)

@tczekajlo i've updated the config on this repo too, ready for review 🌴


Response too large due to client request

With some queries, the sdmx server responds with the following message ...
Message code="413"
Response too large due to client request
Query size exceeds maximum limit (1000000 entries).
-- Code to replicate error --
estat = client('Eurostat', 'eurodb.db')
db = estat.get_dataflows()
gdp_table = db.execute('SELECT * FROM ESTAT_dataflows WHERE title LIKE "%GDP%"')
gdp_list = gdp_table.fetchall()
bop = gdp_list[1]
df, md = estat.get_data(bop, '', concat = True) # Request is too large and this call fails silently.
In theory, the filters parameter in the get_data function should help reduce the size of the request.

For those that are looking at using filters in request, the following code shows an example.
from pandasdmx import client
estat = client("Eurostat")
estat.get_data("namq_10_gdp",'.CLV10_MEUR.SWDA.B1GQ.DE', concat = False)
First argument is the flowref, second argument is the "key". This is where you add filters to your request.
A useful website for those looking to learn more about Eurostat's web services can be found here:
http://ec.europa.eu/eurostat/web/sdmx-web-services/data-struct-def
and
http://ec.europa.eu/eurostat/web/sdmx-web-services/a-few-useful-points

Thanks. You are right: This error is not caught although it should be.
And it will be in v0.2 (see below).
I have embarked on a complete rewrite. v0.1.2 was mainly a proof of
concept. I have since liaised with the creators of SDMX who gave me some
advice. End November I decided to rewrite the whole thing. I have looked
into configuration and type validation through IPython.utils.traitlets
and IPython.config.configurable, application-wide logging etc.
I have dropped the SQLite stuff. It will be replaced by xpath queries on
the XML document. At the heart of the new architecture there will be a
pythonic representation of the SDMX information model (see Part 2 of the
SDMX2.1 Standard and the user guide on www.sdmx.org). The information
model is separate from the code relating to XML, JSON etc. in the reader
package, and pandas, xray, Excel, SQLAlchemy, smoke signals etc. in the
writer package. The information model classes call reader methods which
execute xpath queries on the xml document. These method calls are
wrapped in properties.
The code is evolving but far from stable. There is an incredible amount
of work remaining. But I believe the goal is worth the effort. Some one
has to do it, not necessarily myself.
On v0.2dev: The information model in model.py is heavily under
construction. There is a Message class representing an SDMX message. It
will allow error detection after downloading.
Please let me know any thoughts you might have.
Leo
Am 02.01.2015 um 02:09 schrieb geoffwright240:

With some queries, the sdmx server responds with the following message ...
footer:FooterResponse too large due to client request/common:TextQuery
size exceeds maximum limit (1000000
entries)./common:Text/footer:Message/footer:Footer
Generic setup
estat = client('Eurostat', 'eurodb.db')
db = estat.get_dataflows()
gdp_table = db.execute('SELECT * FROM ESTAT_dataflows WHERE title LIKE
"%GDP%"')
gdp_list = gdp_table.fetchall()
bop = gdp_list[1]
df, md = estat.get_data(bop, '', concat = True) # Request is too large
and this call fails silently.
In theory, the filters parameter in the get_data function should help
reduce the size of the request.
—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4.

Thanks. Very useful.
The forthcoming v0.2 will support filter creation pythoniccally rather
than through a bare function argument. Any help is much appreciated. To
get a first impression check out the dev versions regularly. But don't
expect anything to work as expected. Currently I am working on the
pythonic representation of structures such as codelists and
conceptschemes. Datastructure-Definitions will follow, only thereafter
datasets.
Am 02.01.2015 um 14:27 schrieb geoffwright240:

For those that are looking at using filters in request, the following
code shows an example.
from pandasdmx import client
estat = client("Eurostat")
estat.get_data("namq_10_gdp",'.CLV10_MEUR.SWDA.B1GQ.DE', concat = False)
First argument is the flowref, second argument is the "key". This is
where you add filters to your request.
A useful website for those looking to learn more about Eurostat's web
services can be found here:
http://ec.europa.eu/eurostat/web/sdmx-web-services/data-struct-def
and
http://ec.europa.eu/eurostat/web/sdmx-web-services/a-few-useful-points
—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68526378.

0.2dev looks very promising … although I don’t see model.py in the repo yet.
I was extremely happy to see this though as I was just in the process of trying to figure out how to solve this issue. I almost wonder if this bit of code should be in pandas as opposed to pandaSDMX.  I also wonder what the best way to handle this is … the time period we are talking about is actually a quarter so not totally correct to convert to first month. (I wonder how other languages treat this. I recall from many years ago that FAME had a quarter time period defined.)
Time span conversions not recognised by pandas:
time_spans = {
'Q1' : '01-01',
'Q2' : '04-01',
'Q3' : '07-01',
'Q4' : '10-01',
'S1' : '01-01',
'S2' : '07-01'
}
Geoff

On Jan 2, 2015, at 4:22 PM, dr-leo<EMAIL_ADDRESS>wrote:
Thanks. Very useful.
The forthcoming v0.2 will support filter creation pythoniccally rather
than through a bare function argument. Any help is much appreciated. To
get a first impression check out the dev versions regularly. But don't
expect anything to work as expected. Currently I am working on the
pythonic representation of structures such as codelists and
conceptschemes. Datastructure-Definitions will follow, only thereafter
datasets.
Am 02.01.2015 um 14:27 schrieb geoffwright240:

For those that are looking at using filters in request, the following
code shows an example.
from pandasdmx import client
estat = client("Eurostat")
estat.get_data("namq_10_gdp",'.CLV10_MEUR.SWDA.B1GQ.DE', concat = False)
First argument is the flowref, second argument is the "key". This is
where you add filters to your request.
A useful website for those looking to learn more about Eurostat's web
services can be found here:
http://ec.europa.eu/eurostat/web/sdmx-web-services/data-struct-def
and
http://ec.europa.eu/eurostat/web/sdmx-web-services/a-few-useful-points
—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68526378.

—
Reply to this email directly or view it on GitHub https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68563571.

I double-checked that model.py is tracked. It should be on github for
quite a while. Have you updated to tip? the v0.2dev branch?
On pandas time spans: Agreed. I wanted to open an issue on pydata/pandas
but I wasn't entirely sure that I hadn't overlooked anything. There is
so much debate on time spans on the pydata mailing list... I take your
comment as a nudge to raise this question on the list.
Leo
Am 03.01.2015 um 01:22 schrieb geoffwright240:

0.2dev looks very promising … although I don’t see model.py in the
repo yet.
I was extremely happy to see this though as I was just in the process
of trying to figure out how to solve this issue. I almost wonder if
this bit of code should be in pandas as opposed to pandaSDMX. I also
wonder what the best way to handle this is … the time period we are
talking about is actually a quarter so not totally correct to convert
to first month. (I wonder how other languages treat this. I recall
from many years ago that FAME had a quarter time period defined.)
Time span conversions not recognised by pandas:
time_spans = {
'Q1' : '01-01',
'Q2' : '04-01',
'Q3' : '07-01',
'Q4' : '10-01',
'S1' : '01-01',
'S2' : '07-01'
}
Geoff

On Jan 2, 2015, at 4:22 PM, dr-leo<EMAIL_ADDRESS>wrote:
Thanks. Very useful.
The forthcoming v0.2 will support filter creation pythoniccally rather
than through a bare function argument. Any help is much appreciated. To
get a first impression check out the dev versions regularly. But don't
expect anything to work as expected. Currently I am working on the
pythonic representation of structures such as codelists and
conceptschemes. Datastructure-Definitions will follow, only thereafter
datasets.
Am 02.01.2015 um 14:27 schrieb geoffwright240:

For those that are looking at using filters in request, the following
code shows an example.
from pandasdmx import client
estat = client("Eurostat")
estat.get_data("namq_10_gdp",'.CLV10_MEUR.SWDA.B1GQ.DE', concat =
False)
First argument is the flowref, second argument is the "key". This is
where you add filters to your request.
A useful website for those looking to learn more about Eurostat's web
services can be found here:
http://ec.europa.eu/eurostat/web/sdmx-web-services/data-struct-def
and

http://ec.europa.eu/eurostat/web/sdmx-web-services/a-few-useful-points

—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68526378.

—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68563571.

—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68575639.

Leo,
Sorry. My mistake — yes, model.py is in v0.2dev branch. Also, don’t let me slow you down as there is a ton of learning I need to do before being able to contribute in a meaningful way.
I read http://pandas.pydata.org/pandas-docs/stable/timeseries.html more carefully and think that using a Period is the way to go for handling quarterly time series that are returned by Eurostat. (There could be something more but this looks like a decent enough solution for now.)
p = Period('2012Q1', freq='Q-DEC')
What would be the best way for me to help contribute ? Once v0.2dev is close, I could help with the documentation. Also, would it make sense for me to write a couple of unit tests ? Not too sure exactly what this entails but I could dig into it if that makes the most sense.
If you are interested in my motivation for the pandaSDMX project, I am looking at this is to integrate links to SDMX web services into a little tool that I am building for Excel. I am not a huge fan of Excel but it is so ubiquitous (and what I use at my job that pays the mortgage) so there is a huge user community there. The tool is open source and I would like to eventually port to Openoffice and gnumeric. My project is very rudimentary at this stage but it generally works as hoped. I have not put it up on Github yet as the code is way too premature to have in the public.
Once complete, a user will only have to put the following function into a MS Excel cell and it will pull a time series into your spreadsheet.
=DataReader(series:‘GDP’,db:’fred’,startdate:’2000-01-01’,enddate:’2014-01-01’,showdates:’True’,...)
This is similar to functionality you can get through Bloomberg’s BDH, SunGard’s FAMEPopulator and S&P’s CapitalIQ but you are not restricted to their proprietary databases (which are in large part just aggregators for public data anyways.)
My approach only works on Mac currently as the implementation of xlwings is not async and therefore gets trapped by limitations on Microsofts com restrictions. The guys behind xlwings have a fix for this that they are currently implementing.
Anyways, not to distract you from your projects at hand, but if you are interested here is a little bundle of code and a video to give you an idea of what I am trying to do. (It’s probably just easiest to watch the video but happy to take any comments you have on the code.)
http://wrightville.org/rel/xlwings_dl.zip
All the best Leo !
Geoff
Le 2015-01-03 à 03:11, dr-leo<EMAIL_ADDRESS>a écrit :

I double-checked that model.py is tracked. It should be on github for
quite a while. Have you updated to tip? the v0.2dev branch?
On pandas time spans: Agreed. I wanted to open an issue on pydata/pandas
but I wasn't entirely sure that I hadn't overlooked anything. There is
so much debate on time spans on the pydata mailing list... I take your
comment as a nudge to raise this question on the list.
Leo
Am 03.01.2015 um 01:22 schrieb geoffwright240:

0.2dev looks very promising … although I don’t see model.py in the
repo yet.
I was extremely happy to see this though as I was just in the process
of trying to figure out how to solve this issue. I almost wonder if
this bit of code should be in pandas as opposed to pandaSDMX. I also
wonder what the best way to handle this is … the time period we are
talking about is actually a quarter so not totally correct to convert
to first month. (I wonder how other languages treat this. I recall
from many years ago that FAME had a quarter time period defined.)
Time span conversions not recognised by pandas:
time_spans = {
'Q1' : '01-01',
'Q2' : '04-01',
'Q3' : '07-01',
'Q4' : '10-01',
'S1' : '01-01',
'S2' : '07-01'
}
Geoff

On Jan 2, 2015, at 4:22 PM, dr-leo<EMAIL_ADDRESS>wrote:
Thanks. Very useful.
The forthcoming v0.2 will support filter creation pythoniccally rather
than through a bare function argument. Any help is much appreciated. To
get a first impression check out the dev versions regularly. But don't
expect anything to work as expected. Currently I am working on the
pythonic representation of structures such as codelists and
conceptschemes. Datastructure-Definitions will follow, only thereafter
datasets.
Am 02.01.2015 um 14:27 schrieb geoffwright240:

For those that are looking at using filters in request, the following
code shows an example.
from pandasdmx import client
estat = client("Eurostat")
estat.get_data("namq_10_gdp",'.CLV10_MEUR.SWDA.B1GQ.DE', concat =
False)
First argument is the flowref, second argument is the "key". This is
where you add filters to your request.
A useful website for those looking to learn more about Eurostat's web
services can be found here:
http://ec.europa.eu/eurostat/web/sdmx-web-services/data-struct-def
and

http://ec.europa.eu/eurostat/web/sdmx-web-services/a-few-useful-points

—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68526378.

—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68563571.

—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68575639.

—
Reply to this email directly or view it on GitHub.

Dear Jeff,
I am very happy about your interest in pandaSDMX. Apparently Excel users
including myself could greatly benefit from your connector. Once
pandaSDMX has matured, you may want to focus on a writer for Excel if
you feel the detour through pandas has too many pitfalls.

Excel
I hadn't heard about xlwings before, and I know little about Excel
except I have to use it on the job as well, but hardly for data
analysis. A limitation on Mac would seem to be something you want to
overcome in your clients' interest. There are a couple of Excel-related
libs for Python. You will have looked at OpenPyXL that ships with
Anaconda. And there is this nice weekly Python newsletter that talks a
lot about blog posts and books on data analysis including with Excel and
pandas. I'll forward you the latest one with an announcement on Yve
Hilpisch's most recent title.

Contributing
I am sure you will identify areas where you can best contribute. You are
already of such tremendous help. I face a steep learning curve, but
Python makes it easy to enter new territory. And I have spent some time
reading through the SDMX standard. When I decided to release pandaSDMX
0.1 I was aware of many deficiencies. But I still wanted to release it
to have something to show to the SDMX community. And it has paid off: I
am in contact with a very nice guy from the ECB who has given some great
advice. The leading company on SDMX software is UK-based Metadata
Technologies. But their free product, see www.sdmxsource.org, seems
untractable. I will never learn Java or .NET. Interestingly, they have
developed browser-based SDMX software written in JavaScript. I
understand they can also export to Excel as central bankers use Excel
for macro-economic analysis.

On time spans:
Yes, Pandas supports time spans (periods). But the key question is how
to generate a period index from a list of strings gleened from an XML
document. pandas.to_datetime() fails. So I have asked on the pydata
mailing list. Let's see what the experts say. - At the end of the day,
i.e. once pandaSDMX 0.2 is ready, this will be no big deal as SDMX
DataStructureDefinitions contain an abstract description of the datetime
index which will then be generated exactly as you set out below. But
when relying on the generic dataset format, each observation is related
to a time representation such as "2007-Q3". So you have to digest long
lists of these strings. If pandas cannot parse them, you have to work
around this gap. Hence the little dictionary you compared to FAME. By
the way, it is very valuable that you seem to have some familiarity with
Bloomberg, S&P etc. So you'll have good ideas on usability. I work in
financial supervision, but as a lawyer, not statisticion, so I have no
experience with Bloomberg, Thomson-Reuters etc.

In order to get a good understanding of XML within the SDMX context, you
want to make sure to be able to read XML documents. Eclipse seems like a
good environment for this, but there are many IDE's out there.
Anyway, time spans  are details for later. the next goal is to get the
Message class in the model right. I'll have to read through the SDMX-ML
specification (= the XML representation for SDMX) to find out about the
different message types, attributes and children and create a pythonic
API for that. This is no rocket science. But it requires good intuition
for usability and the most concise and elegant way of expressing the
structures. I recommend looking at the pandasdmx.agency.Agency class as
well as the utils.HasItems class (the latter being under construction).
I am working on it. I hope to allow indexing and dict-like attribute
access on XML attributes and nodes, all through properties. Don't know
if this is too fancy... I'll commit the latest ideas to github now. I
think I have a decent implementation for ItemSchemes such as Codelists
and ConceptSchemes. Extending it to CategorySchemes and Dataflows will
be very easy. But the DataStructureDefinition itself with all those
datatypes (facets) will be a bit of a challenge I think. And then
constraints...
So much for now. Please ignore from the above whatever bores you or
looks too messy.
Again, I am very glad you are interested in my little pet project and I
am hopeful we can both benefit from each other's skills and efforts.
Leo
Am 03.01.2015 um 15:11 schrieb geoffwright240:

Leo,
Sorry. My mistake — yes, model.py is in v0.2dev branch. Also, don’t
let me slow you down as there is a ton of learning I need to do before
being able to contribute in a meaningful way.
I read http://pandas.pydata.org/pandas-docs/stable/timeseries.html
more carefully and think that using a Period is the way to go for
handling quarterly time series that are returned by Eurostat. (There
could be something more but this looks like a decent enough solution
for now.)
p = Period('2012Q1', freq='Q-DEC')
What would be the best way for me to help contribute ? Once v0.2dev is
close, I could help with the documentation. Also, would it make sense
for me to write a couple of unit tests ? Not too sure exactly what
this entails but I could dig into it if that makes the most sense.
If you are interested in my motivation for the pandaSDMX project, I am
looking at this is to integrate links to SDMX web services into a
little tool that I am building for Excel. I am not a huge fan of Excel
but it is so ubiquitous (and what I use at my job that pays the
mortgage) so there is a huge user community there. The tool is open
source and I would like to eventually port to Openoffice and gnumeric.
My project is very rudimentary at this stage but it generally works as
hoped. I have not put it up on Github yet as the code is way too
premature to have in the public.
Once complete, a user will only have to put the following function
into a MS Excel cell and it will pull a time series into your
spreadsheet.
=DataReader(series:‘GDP’,db:’fred’,startdate:’2000-01-01’,enddate:’2014-01-01’,showdates:’True’,...)
This is similar to functionality you can get through Bloomberg’s BDH,
SunGard’s FAMEPopulator and S&P’s CapitalIQ but you are not restricted
to their proprietary databases (which are in large part just
aggregators for public data anyways.)
My approach only works on Mac currently as the implementation of
xlwings is not async and therefore gets trapped by limitations on
Microsofts com restrictions. The guys behind xlwings have a fix for
this that they are currently implementing.
Anyways, not to distract you from your projects at hand, but if you
are interested here is a little bundle of code and a video to give you
an idea of what I am trying to do. (It’s probably just easiest to
watch the video but happy to take any comments you have on the code.)
http://wrightville.org/rel/xlwings_dl.zip
All the best Leo !
Geoff
Le 2015-01-03 à 03:11, dr-leo<EMAIL_ADDRESS>a écrit :

I double-checked that model.py is tracked. It should be on github for
quite a while. Have you updated to tip? the v0.2dev branch?
On pandas time spans: Agreed. I wanted to open an issue on
pydata/pandas
but I wasn't entirely sure that I hadn't overlooked anything. There is
so much debate on time spans on the pydata mailing list... I take your
comment as a nudge to raise this question on the list.
Leo
Am 03.01.2015 um 01:22 schrieb geoffwright240:

0.2dev looks very promising … although I don’t see model.py in the
repo yet.
I was extremely happy to see this though as I was just in the process
of trying to figure out how to solve this issue. I almost wonder if
this bit of code should be in pandas as opposed to pandaSDMX. I also
wonder what the best way to handle this is … the time period we are
talking about is actually a quarter so not totally correct to convert
to first month. (I wonder how other languages treat this. I recall
from many years ago that FAME had a quarter time period defined.)
Time span conversions not recognised by pandas:
time_spans = {
'Q1' : '01-01',
'Q2' : '04-01',
'Q3' : '07-01',
'Q4' : '10-01',
'S1' : '01-01',
'S2' : '07-01'
}
Geoff

On Jan 2, 2015, at 4:22 PM, dr-leo<EMAIL_ADDRESS>wrote:
Thanks. Very useful.
The forthcoming v0.2 will support filter creation pythoniccally
rather
than through a bare function argument. Any help is much
appreciated. To
get a first impression check out the dev versions regularly. But
don't
expect anything to work as expected. Currently I am working on the
pythonic representation of structures such as codelists and
conceptschemes. Datastructure-Definitions will follow, only
thereafter
datasets.
Am 02.01.2015 um 14:27 schrieb geoffwright240:

For those that are looking at using filters in request, the
following
code shows an example.
from pandasdmx import client
estat = client("Eurostat")
estat.get_data("namq_10_gdp",'.CLV10_MEUR.SWDA.B1GQ.DE', concat =
False)
First argument is the flowref, second argument is the "key".
This is
where you add filters to your request.
A useful website for those looking to learn more about
Eurostat's web
services can be found here:

http://ec.europa.eu/eurostat/web/sdmx-web-services/data-struct-def

and

http://ec.europa.eu/eurostat/web/sdmx-web-services/a-few-useful-points

—
Reply to this email directly or view it on GitHub

https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68526378.

—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68563571.

—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68575639.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/issues/4#issuecomment-68596035.

Fixed in v0.2.0. v0.2.1 will add native support to handle the zip files returned by Eurostat.


should settings parsing ignore invalid values?

should settings parsing ignore invalid values?

It would definitely be better if we were printing a warning in case of invalid entries.
We could also have another setting (disabled by default for now) to make it fail/stop test execution.

[like]  Jakub Jares reacted to your message:

From: Amaury Levé @.>
Sent: Wednesday, May 29, 2024 7:31:54 AM
To: microsoft/testfx @.>
Cc: Subscribed @.***>
Subject: Re: [microsoft/testfx] should settings parsing ignore invalid values? (Issue #3020)
It would definitely be better if we were printing a warning in case of invalid entries.
We could also have another setting (disabled by default for now) to make it fail/stop test execution.
—
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/testfx/issues/3020#issuecomment-2136727690 or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABLYLYLVCUBGF23ODCXDZNDZEV76XBFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVE2TKNBTGYYTAOJRURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2UNBSHA3TENZSG4ZTJJDOMFWWLKLIMFZV63DBMJSWZLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOKIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVA3DKMBYGA3TENMCUR2HS4DFUVUXG43VMWSXMYLMOVS2UMRTGIZDGMRQGE4TNAVEOR4XAZNFNRQWEZLMUV3GC3DVMWUTKNJUGM3DCMBZGGBKI5DZOBS2K3DBMJSWZJLWMFWHKZNKGQZDQNZSG4ZDOMZUU52HE2LHM5SXFJTDOJSWC5DF.
You are receiving this email because you are subscribed to this thread.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


[Event Request] codeunit 444 "Purchase-Post Prepayments" - OnBeforePostVendorEntry

Please extend the event OnBeforePostVendorEntry with additional parameter:
        if GLSetup."Journal Templ. Name Mandatory" then
            "Journal Template Name" := GenJournalTemplate.Name;

        OnBeforePostVendorEntry(GenJnlLine, TotalPrepmtInvLineBuffer, TotalPrepmtInvLineBufferLCY, SuppressCommit, PurchHeader);
        GenJnlPostLine.RunWithCheck(GenJnlLine);
        OnAfterPostVendorEntry(GenJnlLine, TotalPrepmtInvLineBuffer, TotalPrepmtInvLineBufferLCY, SuppressCommit);
    end;
end;

[IntegrationEvent(false, false)]
local procedure OnBeforePostVendorEntry(var GenJnlLine: Record "Gen. Journal Line"; TotalPrepmtInvLineBuffer: Record "Prepayment Inv. Line Buffer"; TotalPrepmtInvLineBufferLCY: Record "Prepayment Inv. Line Buffer"; CommitIsSupressed: Boolean; PurchaseHeader: Record "Purchase Header")
begin
end;

Can't find any of this code in PurchaseHeader, please double check which version/country you are using.

Sorry, wrong title. I have corrected the title. The change should happen in the codeunit 444 "Purchase-Post Prepayments" analog to the event OnBeforePostCustomerEntry in codeunit 442 "Sales-Post Prepayments"

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


Added the possibility to check if any objects are still busy in BaseViewModel

This allows the view to check whether there are still any running processes in the ViewModel. For further description red #64

Doooooope! I shall merge and push this right now.

@timcreatedit pushed it's in 1.5.8

Great, happy to help! And thanks for the amazing work you keep doing for the community