validation TRE and CT data preprocessing

Thanks a lot for this nice work!
I have 3 questions:

I want to evaluate the landmark TRE in the validation step during training and also use it as an evaluation metric in the inference step.
How can I transform the n 3d landmarks (array of [n,3]) with the estimated transformation from a forward pass to the trained model?

Do you have any recommendations for the validation step? Currently, I run multiple forward passes over a predefined range of lambdas and then take the best result

I am a little confused about the preprocessing of lung CT data. In file:///home/nnrthmr/Downloads/Learn2Reg2021_Tony_oral-1.pdf it says that you performed windowing of [100, 1518]. Are these the ranges of the pixel values after clipping or are they the window level and window width? Also, do I need to perform the clipping first and then normalize to [0,1] or the other way around?

Thanks a lot!

Hi @annareithmeir,

I first transform the F_X_Y to unnormalized displacement field and applied it to the landmarks using map_coordinates  function.

def compute_tre(x, y, spacing=(1, 1, 1)):
    return np.linalg.norm((x - y) * spacing, axis=1)

    F_X_Y_xyz = torch.zeros(F_X_Y.shape, dtype=F_X_Y.dtype, device=F_X_Y.device)
    _, _, x, y, z = F_X_Y.shape
    F_X_Y_xyz[0, 0] = F_X_Y[0, 2] * (x - 1) / 2
    F_X_Y_xyz[0, 1] = F_X_Y[0, 1] * (y - 1) / 2
    F_X_Y_xyz[0, 2] = F_X_Y[0, 0] * (z - 1) / 2

    F_X_Y_xyz_cpu = F_X_Y_xyz.data.cpu().numpy()[0]

    moving_keypoint = keypoints[:, 3:]
    fixed_keypoint = keypoints[:, :3]

    fixed_disp_x = map_coordinates(F_X_Y_xyz_cpu[0], fixed_keypoint.transpose())
    fixed_disp_y = map_coordinates(F_X_Y_xyz_cpu[1], fixed_keypoint.transpose())
    fixed_disp_z = map_coordinates(F_X_Y_xyz_cpu[2], fixed_keypoint.transpose())
    lms_fixed_disp = np.array((fixed_disp_x, fixed_disp_y, fixed_disp_z)).transpose()

    warped_fixed_keypoint = fixed_keypoint + lms_fixed_disp
    tre_score = compute_tre(warped_fixed_keypoint, moving_keypoint, spacing=(1.5, 1.5, 1.5)).mean()
    tre_total.append(tre_score)

I didn't run multiple forward passes during validation. Instead, I will just set a fixed hyperparameter for the validation and run one forward pass. After training the model, I will select the model with the best TRE in the validation set, and then search for the best hyperparameter using the well-trained model for the validation set.

windowing of [100, 1518] = np.clip(fixed_img, a_min=100, a_max=1518). But this is not the best approach. You should use a lung lobe mask to mask the similarity and use a wider window, i.e.,  np.clip(fixed_img, a_min=-1100, a_max=1518). And yes,  you need to perform the clipping first and then normalize to [0,1]. In this year's MICCAI, I will present an improved LapIRN for this task.

Hope the above comments help.
Regards,
Tony

Thanks a lot! Looking forward to the improved LapIRN!

Just for clarification: By map_coordinates you mean the scipy.ndimage.map_coordinates function?

Yes.


Remove ROUTER_SELECTION_JITTER setting for ED/SED in cert scripts.

Current coverage is 71.99% (diff: 100%)

Merging #792 into master will decrease coverage by 0.60%

@@             master       #792   diff @@
==========================================
  Files           107        106     -1
  Lines         15994      15985     -9
  Methods        2181       2176     -5
  Messages          0          0
  Branches       1901       1901
==========================================
- Hits          11612      11509   -103
- Misses         3761       3853    +92
- Partials        621        623     +2

Powered by Codecov. Last update 0754ac7...c294cd0


Conversational Form error, the formEl needs to be defined

Hi,
First of all congratulations!!
Is a very good idea.
But, I'm trying to test it and I get this error:
Uncaught Error: Conversational Form error, the formEl needs to be defined.
I instantiate converstaional form this way (but I had test it with jQuery too):
if(!window.ConversationalForm){
      window.ConversationalForm = new cf.ConversationalForm({
        formEl: document.getElementById("contact-form-1234"),
        context: document.getElementById("wpcf7-f455-p450-o1")
    	});
    }

Any idea?

hi @aireeria thanks :)
When creating the instance you need to make sure that the formEl isnt null/undefined.
So from your code it would seem that
document.getElementById("contact-form-1234")

is null.
One would normal include the Conversational Form script and custom code below the rest of your code-base.
Can you confirm?

hi @felixnielsen
thanks for your work!
It's loaded.
document.getElementById("contact-form-1234") in console throws the form element but it keeps throwing
the Uncaught Error: Conversational Form error, the formEl needs to be defined.

I'm dumb!
Sorry!
You can delete all the thread!

It works like a charm!!!


ROX-17963: CI: Fix central channel handling

Description
#6590 changed the test<->central channel state to use ThreadLocal variables. The intent being that each concurrent test thread would not have to be concerned with other tests altering its central connection. However a side effect of this change is that when threads complete and the ThreadLocal vars are GC'd the gRPC package produces a nasty error message:
    Jun 30, 2023 9:29:07 AM io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
    SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=16, target=localhost:8000} was not shutdown properly!!! ~*~*~*
        Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
    java.lang.RuntimeException: ManagedChannel allocation site
    	at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:94)
    	at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:52)
... snip very long stack trace ...

It also turns out that because of the use of org.junit.rules.Timeout, each spock test feature is executed in a separate thread. And so along with #6590, a new channel is now created for each test feature.
This PR reverts most of #6590 and switches back to using global state to manage the test<->central channel. Changes to the channel are protected with synchronized blocks in anticipation of future concurrent test work. Note: with this change, tests that test the central connection (e.g. AuthServiceTest, ClientCertAuthTest) will not be parallel-able.
ref: https://redhat-internal.slack.com/archives/CLUNQEEMA/p1688074475205819
cc @rukletsov
Checklist

[ ] Investigated and inspected CI test results

Testing Performed
CI is sufficient.

/test gke-upgrade-tests

/test gke-upgrade-tests

/fyi @janisz @tommartensen a cleanup relating to #6590


tpm2-tss: fix build

Channel blocker https://hydra.nixos.org/build/109901320

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Just going to ping gnome maintainers because it was relevant @hedning @jtojnar.


fix: vtls add select ts version

Adds a keymap to select typescript version

Also noticed we deleted treesitter config on this file, was that intentional.

typescript and tsx already included in the default treesitter spec.

oh nice.

Thanks!


"Invariant Violation: Expecting a parsed GraphQL document" Error when using query option

I am using the query option to make the stool query to run if only view_stool is true. here's the code:
apollo: {
stool: {
    query () {
        // Here you can access the component instance with 'this'
        if (this.view_stool === true) {
            return stoolQuery
        }
    },

    variables () {
        return {
            id: this.stool_id
        }
    },

    update (data) {
        this.stool.name = data.stool.name
        this.stool.stool_owner = data.stool.stool_owners.name
        this.stool.stool_type = data.stool.stool_type
        this.stool.date_of_ownership = data.stool.date_of_ownership
        this.stool.rental_per_month = data.stool.rental_per_month
        this.stool.remarks = data.stool.remarks
        this.stool.address = data.stool.address
        this.stool_payments = data.stool.stool_payments
        this.view_stool_dialog = true
        this.view_stool = false
    },

    fetchPolicy: 'network-only'
}
}

Everytime it runs it shows the error but the query is working fine. it just shows the error. Can someone tell me if I have made a mistake?
NOTE: the stoolQuery is a gql file.

if this.view_stool is false then your query is undefined, which causes the error.
You can use the skip option instead:
apollo: {
  stool: {
    skip() {
      return this.view_stool;
    },

    query: stoolQuery,

    variables () {
        return {
            id: this.stool_id
        }
    },


pull request for e2e test at 12/22/2016 05:45:13

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Document and test onDelete option

As discussed in https://github.com/overlookmotel/sequelize-hierarchy/issues/48

Also could add to the docs the procedure for deleting a node and all its children.

Some work is done in this PR: https://github.com/overlookmotel/sequelize-hierarchy/pull/131

Done in v2.0.4.


Gap v3.0.0 breaks under Flutter 3.12.0-8.0.pre.21

In attempting to compile https://github.com/flutter/samples/tree/main/next_gen_ui_demo (and by extension https://github.com/flutter/codelabs/tree/main/next-gen-ui) under Flutter master channel, I'm seeing the following failures.
$ flutter run
Connected devices:
macOS (desktop) • macos  • darwin-arm64   • macOS 13.4 22F66 darwin-arm64
Chrome (web)    • chrome • web-javascript • Google Chrome 114.0.5735.133

No wireless devices were found.

[1]: macOS (macos)
[2]: Chrome (chrome)
Please choose one (or "q" to quit): 1
Launching lib/main.dart on macOS in debug mode...
2023-06-22 12:25:01.777 xcodebuild[68777:17538613] DVTCoreDeviceEnabledState: DVTCoreDeviceEnabledState_Disabled set via user default (DVTEnableCoreDevice=disabled)
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:arm64, id:00006000-001250C00C38801E }
{ platform:macOS, arch:x86_64, id:00006000-001250C00C38801E }
../../../../.pub-cache/hosted/pub.dev/gap-3.0.0/lib/src/rendering/gap.dart:43:38: Error: A value of type 'RenderObject?' can't be assigned to a variable of type 'AbstractNode?'.
 - 'RenderObject' is from 'package:flutter/src/rendering/object.dart' ('../../../../flutter/packages/flutter/lib/src/rendering/object.dart').
 - 'AbstractNode' is from 'package:flutter/src/foundation/node.dart' ('../../../../flutter/packages/flutter/lib/src/foundation/node.dart').
    final AbstractNode? parentNode = parent;
                                     ^
../../../../.pub-cache/hosted/pub.dev/gap-3.0.0/lib/src/rendering/gap.dart:45:25: Error: The getter 'direction' isn't defined for the class 'AbstractNode?'.
 - 'AbstractNode' is from 'package:flutter/src/foundation/node.dart' ('../../../../flutter/packages/flutter/lib/src/foundation/node.dart').
Try correcting the name to the name of an existing getter, or defining a getter or field named 'direction'.
      return parentNode.direction;
                        ^^^^^^^^^
Target kernel_snapshot failed: Exception

Command PhaseScriptExecution failed with a nonzero exit code
warning: Run script build phase 'Run Script' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'Flutter Assemble' from project 'Runner')
** BUILD FAILED **

Building macOS application...
Exception: Build process failed

I expect this version of Flutter to show up on the beta channel in the coming weeks, and show up in stable in August-ish.
For context, here's flutter doctor output:
$ flutter doctor -v
[✓] Flutter (Channel master, 3.12.0-8.0.pre.21, on macOS 13.4 22F66 darwin-arm64, locale en)
    • Flutter version 3.12.0-8.0.pre.21 on channel master at /Users/brettmorgan/flutter
    • Upstream repository https://github.com/flutter/flutter
    • Framework revision a2739c09a7 (31 minutes ago), 2023-06-22 06:01:18 -0400
    • Engine revision 8cc6d6d5ef
    • Dart version 3.1.0 (build 3.1.0-236.0.dev)
    • DevTools version 2.24.0

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
    • Android SDK at /Users/brettmorgan/Library/Android/sdk
    • Platform android-33, build-tools 33.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E300c
    • CocoaPods version 1.12.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] IntelliJ IDEA Community Edition (version 2022.2.2)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 70.0.5
    • Dart plugin version 222.4167.21

[✓] VS Code (version 1.79.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.66.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-arm64   • macOS 13.4 22F66 darwin-arm64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 114.0.5735.133

[✓] Network resources
    • All expected network resources are available.

• No issues found!

Thanks, a fix has been published in 3.0.1

Fpr older versions use
final AbstractNode? parentNode = parent; - >  final RenderObject? parentNode = parent;


fixed the handling of startTick = 0

Congrats on 1.7.0!  I found a little bug with notes where startTick = 0.  When they are created, the startTick is set to null, so they are treated by the track as being sequential instead of explicit.

Hey thanks @jpriebe!
I'm thinking it might be clearer to change the way we set these default values overall.  Maybe could use Object.assign():
// Set defaults
fields = Object.assign({
    repeat: 1,
    sequential: false,
    startTick: null,
    velocity: 50,
    wait: 0,
}, fields);

What do you think?

That sounds like a good idea.
I was thinking you might also run into similar problems with things like velocity=0, where the current code would see velocity as unspecified and default it to 50.  You might ask why would somebody ever send velocity=0, but I think that if you were generating the notes programmatically, you might have an algorithm that gradually reduces the velocity, and it takes it all the way to zero.   You wouldn't want that last note to jump back to velocity=50.
So I think your proposed solution would set reasonable defaults, but still allow for zero-value properties.

Any chance you can get a 1.7.1 official release into npm?

Hey Jason,
I think 1.7.1 is currently the latest version  available in npm, no?
https://www.npmjs.com/package/midi-writer-js
-Garrett

You're right.  I set up my project on a new machine, and I assumed that when I ran npm install, I would get the latest version of all the libraries, including midi-writer-js.  But my package-lock.json had me stuck on 1.6.0.  All is good.  Thanks!


Add ability to auto populate adding KUBERNETES_SERVICE_HOST to noProxy

What this PR does / why we need it:
Auto populates KUBERNETES_SERVICE_HOST by adding it to noProxy list of kapp-controller Secret config.
Which issue(s) this PR fixes:
Fixes #391
Additional Notes for your reviewer:
Additional documentation e.g., Proposal, usage docs, etc.:
Will add small note to carvel.dev once UX is solidified.

lgtm, fwiw :)


AS3 does not allow 2 VIPs in different partitions even if ports are different

          I also have this issue but I believe it was closed incorrectly. If I add a VIP to a partition and then add another VIP to a different partition with the same IP address but a different port, the GUI and TMSH allows me to do this. However, with AS3, the API errors.

Example:

I have added 3 VIPS - one to the Common partition, and 1 each to "test1" and "test2" partitions. The IP address is the same, but the port is unique - this is added with no problems and tests fine.
However, if I try and run an AS3 declaration with the same config, I get the following error:
 Error: posting as3 config failed for tenants:(dccdm-eit) with error: Tenant Creation failed with Response: [
       {
               "code": 422,
               "host": "localhost",
               "response": "0107176c:3: Invalid Virtual Address, the IP address <IP_ADDRESS> already exists.",
               "runTime": 22895,
               "tenant": "test1"
       }
 ]

The logic here should be that a VIP cannot be added if another partition has the same IP+Port, but if either is different, it should be allowed.
Originally posted by @thepowercoders in https://github.com/F5Networks/f5-appsvcs-extension/issues/186#issuecomment-1641126575

AS3 team - any traction on this?  We are running into the same issue.

Attention: @mdditt2000, @sunitharonan

@ghalevy - I've been doing some testing here and actually there is a way of getting this to work in AS3. The issue is similar to the way AS3 works with pools and nodes. There is no actual AS3 class for a node - when you add a pool you define the node addresses in the pool class and an option "shareNodes" allows you to reuse the same node in different partitions (basically by sticking the node into /Common). I was thinking - when you add a virtual server, you also add a virtual address resource (which again doesn't have a specific class in AS3) - so I guessed maybe it works the same way.... and it does!
There is an AS3 property in the Service class "shareAddresses" which does the same thing... It indicates whether virtualAddresses should be added to or checked for in /Common instead of the tenant.
Example AS3 below adding a VIP into 2 different partitions/tenants using the same IP:
{
    "class": "AS3",
    "action": "deploy",
    "persist": true,
    "declaration": {
        "class": "ADC",
        "schemaVersion": "3.45.0",
        "test1": {
            "class": "Tenant",
            "test1": {
                "class": "Application",
				"vip1": {
					"class": "Service_HTTP",
					"virtualAddresses": [
						"<IP_ADDRESS>"
						],
					"virtualPort": 80,
                    "shareAddresses": true
                }
            }
        },
        "test2": {
            "class": "Tenant",
            "test2": {
                "class": "Application",
				"vip2": {
					"class": "Service_HTTP",
					"virtualAddresses": [
						"<IP_ADDRESS>"
						],
					"virtualPort": 443,
                    "shareAddresses": true
                }
            }
        }
    }
}

This worked with the result:

Looks promising, good find @thepowercoders - I will try this out tomorrow and report back 👍

The shareNodes: true workaround is working for Gilad and myself.
@thepowercoders and @RobCupples Could we reopen this and request a documentation update for this behavior and/or prevent this from happening in AS3?


ElastAlert docker image cannot run successfully

Hi,
I'm now considering about using latestelastalert server docker image ( which could be found in the docker hub bitsensor/elastalert, tag:latest), but the image are not be able to run successfully and docker logs command shows below 2 error messages:

ERROR elastalert-server:

    ProcessController:  Traceback (most recent call last):
      File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
        "__main__", fname, loader, pkg_name)
      File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
        exec code in run_globals
      File "/opt/elastalert/elastalert/create_index.py", line 275, in <module>
        main()
      File "/opt/elastalert/elastalert/create_index.py", line 252, in main
        es.indices.put_mapping(index=index, doc_type='elastalert', body=es_mapping)
      File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
        return func(*args, params=params, **kwargs)
      File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/indices.py", line 339, in put_mapping
        "PUT", _make_path(index, "_mapping", doc_type), params=params, body=body
      File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request
        timeout=timeout,
      File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 143, in perform_request
        self._raise_error(response.status_code, raw_data)
      File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/base.py", line 168, in _raise_error
        status_code, error_message, additional_info
    elasticsearch.exceptions.RequestError: RequestError(400, u'illegal_argument_exception', u'Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true.')

ERROR elastalert-server:

    ProcessController:      "__main__", fname, loader, pkg_name)
      File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
        exec code in run_globals
      File "/opt/elastalert/elastalert/elastalert.py", line 1929, in <module>
        sys.exit(main(sys.argv[1:]))
      File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main

09:49:06.039Z ERROR elastalert-server:
    ProcessController:      client.start()
      File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start

09:49:06.039Z ERROR elastalert-server:
    ProcessController:      self.run_all_rules()
      File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules

09:49:06.040Z ERROR elastalert-server:
    ProcessController:      self.send_pending_alerts()
      File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts

09:49:06.040Z ERROR elastalert-server:
    ProcessController:      pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
      File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts

09:49:06.040Z ERROR elastalert-server:
    ProcessController:      size=1000)
      File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
        return func(*args, params=params, **kwargs)
    TypeError:
09:49:06.041Z ERROR elastalert-server:
    ProcessController:  search() got an unexpected keyword argument 'doc_type'

I also tried some other versions like tag: 3.0.0-beta.0 and it seems like different image occurs different error but won't be running. due to some internet access limitation I can not build the docker image locally on that server, can anyone give any advices on this and thanks in advance. (I'm using ES 7.0.1).

I'm seeing the same thing here.

I'm seeing the same thing here.

hi cmeisinger,
Since there is no reply at all and after trying several different image versions and I found this image (tag: 3.0.0-beta.0) could work through but only occurs a  permission exception ( mounted file cannot be read due to the user in the container is node but it's not outside in the host machine in my case). in order to solve this, I have to grant them 777 access permission ( could be lower but it doesn't matter) in the host machine. I think you could give it a shoot. 🙂
BYW, I still want to hear from alert team  ...

It seems Elastalert has the latest elasticsearch python package as dependency. But Elasticsearch fastly progressing with versions, so it downloads now the package made for Elasticsearch7.
If you change "elasticsearch" to "elasticsearch6" in setup.py, it will be able to build it. (other method is to add >5,<7 version limit to elasticsearch.)
Is 3.0.0-beta.0 from Bitsensor's repo? That is a REST API extension for Yelp's elastalert and that  downloads elastalert.zip from here or from a specific custom url - you can add with that you modded version of elastalert.

@azbesthu thanks for your input i am using ESS 7.3.3  still ima getting following error i am using https://github.com/Yelp/elastalert/archive/v0.2.1.zip
`10:47:29.843Z  INFO elastalert-server: Server:  Server started
10:47:31.134Z ERROR elastalert-server:
ProcessController:  Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/elastalert.py", line 1929, in
10:47:31.134Z ERROR elastalert-server:
ProcessController:      sys.exit(main(sys.argv[1:]))
File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main
client.start()
File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start
self.run_all_rules()
File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules
self.send_pending_alerts()
File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts
pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts
10:47:31.136Z ERROR elastalert-server:
ProcessController:      size=1000)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'
`
any input would be great and please let me know is there any way I can contribute as well

@here i have managed to fix the issue to upgrade to latest version 0.2.1 and working fine

I'm having the same issue. my elasticsearch version is 7.5. ElastAlert does not start:
14:34:02.081Z ERROR elastalert-server:
ProcessController:  Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
14:34:02.081Z ERROR elastalert-server:
ProcessController:      exec code in run_globals
File "/opt/elastalert/elastalert/elastalert.py", line 1929, in
sys.exit(main(sys.argv[1:]))
File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main
14:34:02.081Z ERROR elastalert-server:
ProcessController:      client.start()
File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start
14:34:02.082Z ERROR elastalert-server:
ProcessController:      self.run_all_rules()
14:34:02.082Z ERROR elastalert-server:
ProcessController:    File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules
14:34:02.082Z ERROR elastalert-server:
ProcessController:      self.send_pending_alerts()
14:34:02.082Z ERROR elastalert-server:
ProcessController:    File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts
14:34:02.082Z ERROR elastalert-server:
ProcessController:      pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
14:34:02.083Z ERROR elastalert-server:
ProcessController:    File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts
14:34:02.083Z ERROR elastalert-server:
ProcessController:      size=1000)
14:34:02.083Z ERROR elastalert-server:
ProcessController:    File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'
14:34:02.107Z ERROR elastalert-server: ProcessController:  ElastAlert exited with code 1
14:34:02.107Z  INFO elastalert-server: Server:  Stopping server
14:34:02.108Z  INFO elastalert-server: ProcessController:  ElastAlert is not running
14:34:02.108Z  INFO elastalert-server: Server:  Server stopped. Bye!

@gabricar
Which DockerImage are you using? bitsensor/elastalert:3.0.0-beta.1?

@daichi703n I'm running the latest. Let me try this beta tag

@daichi703n  thx it looks better with the beta release.

docker run -d -p 3030:3030
-v pwd/config/elastalert.yaml:/opt/elastalert/config.yaml
-v pwd/config/config.json:/opt/elastalert-server/config/config.json
-v pwd/rules:/opt/elastalert/rules
-v pwd/rule_templates:/opt/elastalert/rule_templates
--net="host"
--name elastalert bitsensor/elastalert:3.0.0-beta.1


Can't seem to get app to start on Heroku

The Supervisor configuration doesn't seem to play nice with Heroku.

I don't use Heroku.  Exit status 1
2015-03-08T15:45:19.994677+00:00 app[web.1]: npm ERR!
2015-03-08T15:45:19.994821+00:00 app[web.1]: npm ERR! Failed at the<EMAIL_ADDRESS>start script 'eval 'supervisor dist/server >tmp/supervisor.log 2>&1 & SUPERVISOR_PID=$!';     sleep 1;     kill $SUPERVISOR_PID;'.
2015-03-08T15:45:19.994974+00:00 app[web.1]: npm ERR! This is most likely a problem with the bitwatcher package,
2015-03-08T15:45:19.995099+00:00 app[web.1]: npm ERR! not with npm itself.
2015-03-08T15:45:19.995229+00:00 app[web.1]: npm ERR! Tell the author that this fails on your system:
2015-03-08T15:45:19.995374+00:00 app[web.1]: npm ERR!     eval 'supervisor dist/server >tmp/supervisor.log 2>&1 & SUPERVISOR_PID=$!';     sleep 1;     kill $SUPERVISOR_PID;
2015-03-08T15:45:19.995488+00:00 app[web.1]: npm ERR! You can get their info via:
2015-03-08T15:45:19.995618+00:00 app[web.1]: npm ERR!     npm owner ls bitwatcher
2015-03-08T15:45:19.995744+00:00 app[web.1]: npm ERR! There is likely additional logging output above.
2015-03-08T15:45:19.998152+00:00 app[web.1]:
2015-03-08T15:45:20.643048+00:00 heroku[web.1]: Process exited with status 1
2015-03-08T15:45:20.665030+00:00 heroku[web.1]: State changed from starting to crashed
2015-03-08T15:45:21.719448+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=bitwatcher.herokuapp.com request_id=4be4ef58-f83c-42a7-8f04-49338495d5fe fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes=
2015-03-08T15:45:22.129301+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=bitwatcher.herokuapp.com request_id=76341116-379b-48c6-9586-6f5c8dc5e61e fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes=
2015-03-08T15:45:22.673869+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=bitwatcher.herokuapp.com request_id=42612294-57c1-4367-ab6f-1a759a49404a fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes=

See: https://github.com/RickWong/react-isomorphic-starterkit/commit/cb0a4c61a049340d18207454f7c4f16af79edcb7
Instead of the command-line eval + PID-magic the starterkit now uses concurrent(ly). Does it run on Heroku now?

Heroku will infer that this is a node app from the package.json file and try to run npm start by default. If you add a Procfile you can override this to run something else. There will be an environment variable, PORT that your app is expected to bind shortly after starting.
It's problematic that server.js hardcodes 8000 instead of env.PORT || 8000 or similar.
It's also problematic that npm start runs npm run localhost, which seems relevant only to local development. This could be fixed with a Procfile that did the right thing. Maybe that's best left to the client project as they need it? It probably depends on how you see this project getting deployed in a production setting.

Still getting this error:
2015-03-12T12:17:46.160436+00:00 heroku[web.1]: State changed from crashed to starting
2015-03-12T12:17:49.510297+00:00 heroku[web.1]: Starting process with command `npm start`
2015-03-12T12:17:51.006574+00:00 app[web.1]: Detected 512 MB available memory, 512 MB limit per process (WEB_MEMORY)
2015-03-12T12:17:51.006601+00:00 app[web.1]: Recommending WEB_CONCURRENCY=1
2015-03-12T12:17:51.698939+00:00 app[web.1]:
2015-03-12T12:17:51.698946+00:00 app[web.1]: ><EMAIL_ADDRESS>start /app
2015-03-12T12:17:51.698949+00:00 app[web.1]: > sleep 6; concurrent 'supervisor dist/server' 'npm run localhost'
2015-03-12T12:17:51.698950+00:00 app[web.1]:
2015-03-12T12:17:57.730160+00:00 app[web.1]: sh: 1: concurrent: not found
2015-03-12T12:17:57.743679+00:00 app[web.1]: npm ERR!  Exit status 127
2015-03-12T12:17:57.745923+00:00 app[web.1]: npm ERR!
2015-03-12T12:17:57.746086+00:00 app[web.1]: npm ERR! Failed at the<EMAIL_ADDRESS>start script 'sleep 6; concurrent 'supervisor dist/server' 'npm run localhost''.
2015-03-12T12:17:57.746373+00:00 app[web.1]: npm ERR! This is most likely a problem with the bitwatcher package,
2015-03-12T12:17:57.746520+00:00 app[web.1]: npm ERR! not with npm itself.
2015-03-12T12:17:57.746673+00:00 app[web.1]: npm ERR! Tell the author that this fails on your system:
2015-03-12T12:17:57.746812+00:00 app[web.1]: npm ERR!     sleep 6; concurrent 'supervisor dist/server' 'npm run localhost'
2015-03-12T12:17:57.746969+00:00 app[web.1]: npm ERR! You can get their info via:
2015-03-12T12:17:57.747119+00:00 app[web.1]: npm ERR!     npm owner ls bitwatcher
2015-03-12T12:17:57.747451+00:00 app[web.1]: npm ERR! There is likely additional logging output above.
2015-03-12T12:17:57.750472+00:00 app[web.1]: npm ERR! Please include the following file with any support request:
2015-03-12T12:17:57.750606+00:00 app[web.1]: npm ERR!     /app/npm-debug.log
2015-03-12T12:17:58.553089+00:00 heroku[web.1]: State changed from starting to crashed
2015-03-12T12:17:58.542021+00:00 heroku[web.1]: Process exited with status 1
2015-03-12T12:17:59.775297+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=bitwatcher.herokuapp.com request_id=1f3eb99c-cd83-4bab-8be7-ab188dff3f89 fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes=
2015-03-12T12:18:00.362100+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=bitwatcher.herokuapp.com request_id=28189cc9-d21a-4de2-9be9-bb07e70d1602 fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes=
2015-03-12T12:18:01.269384+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=bitwatcher.herokuapp.com request_id=bbc56db3-5059-41a5-8b30-a6c89e3a34f5 fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes=

@lonnen I've got the proper env port set up on my config.
This is the repo i'm trying to deploy if you're interested: https://github.com/michaelryancaputo/bitwatcher

Recently added concurrently as a global npm dependency. Try install it first with: npm install -g concurrently

I've done that as per the commit you referenced above

That's weird then. It shouldn't say sh: 1: concurrent: not found if concurrently is installed. Or does Heroku not allow it?

Was anyone able to successfully deploy it on Heroku?
I'm getting the following with the latest version:
npm ERR! Linux 3.13.0-77-generic
npm ERR! argv "/app/.heroku/node/bin/node" "/app/.heroku/node/bin/npm" "run" "build"
npm ERR! node v5.7.1
npm ERR! npm  v3.6.0
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR<EMAIL_ADDRESS>build: `concurrently "npm run build-server" "npm run build-client"`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>build script 'concurrently "npm run build-server" "npm run build-client"'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the react-isomorphic-starterkit package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     concurrently "npm run build-server" "npm run build-client"
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs react-isomorphic-starterkit
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls react-isomorphic-starterkit
npm ERR! There is likely additional logging output above.

Some help would be very much appreciated :)

concurrently is listed in devDependencies. Heroku doesn't install those by default. You can alter this behavior with an env var that NPM will pick up. Try setting:
heroku config:set NPM_CONFIG_PRODUCTION=false

Hi @lonnen
Thanks for your reply, I got the right dependencies now. But I ran in another issue. Heroku gives me the following:
==> 🌎  Go to http://localhost:8000
Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
Stopping process with SIGKILL
Process exited with status 137
State changed from starting to crashed

http://localhost:8000 is obviously not correct while deploying on Heroku.
I added a console.log of process.env.PORT which results as undefined
Any idea what could cause this?
I have the following env vars set:
NPM_CONFIG_LOGLEVEL=error
NPM_CONFIG_PRODUCTION=true
NODE_ENV=production
NODE_MODULES_CACHE=true
Thanks

You're hitting the same issue I did earlier. Heroku exposes a $PORT value to you (arbitrary and generated at runtime, so you can't hard code it). You need to modify the app to use env.PORT || 8000 or similar.

I have this line
const port = process.env.PORT || 8000;

But its value is undefined
Kinda looks like Heroku fails to inject his environment variables...
Could you share your config?

I'm not running this anymore so I have nothing to share. You can find a bunch of node services doing the same -- see hubot's use of  process.env.PORT on line https://github.com/github/hubot/blob/master/src/robot.coffee#L416
I'd be very surprised if the problem was Heroku not injecting the env var. You could try to log the following:
for (var key in process.env) {
 console.log(process.env[key]);
}

Tried this already, but it only returns the NODE_ENV var (which contains production).
I must be doing something wrong somewhere else.
Thanks for your help anyway!

@littletower I'm curious -- what is the process name you're using in your Procfile? Only the web process will have a port exposed to it.

I have this
web: npm run build; npm run start

I believe you want it to be a single command.
web: npm run build && npm run start

On second thought, you probably don't want to run build here unless you have to do it. You want to keep time to startup the process short. You probably want npm run build to be postinstall instead. See -- https://devcenter.heroku.com/articles/nodejs-support

thanks, so now I have web: npm run start. I also added a postinstall with the npm run build command, it now builds on commit which is nice, but that does not fix the problem with the env vars... :(


[Feature] rencrypt all files (only if the decrypted file changed)

I use this workflow (i use a command from my Makefile (see bellow) )

git checkout/pull project
unlock all files with make secret-unlock
work on my project
lock all files with make secret-lock secret-check
commit all my works

But If the decrypted files not changed and if i use  make secret-lock the content of encrypted files has changed, is it possible to store de SHA1 in the encrypted file (or tempory hidden file during decryption step) and reencrypt only if SHA1 changed and restore encrypted file git state if SHA1 is identical
secret-lock: requirements-check ## lock all files from repository
    agebox reencrypt

secret-unlock: requirements-check ## unlock all files from repository
    agebox decrypt --all

secret-check: requirements-check ## Verify all secrets is in agebox vault
    agebox validate --no-decrypt

@badele Heya! I have exactly this problem as well :-)
The root cause of this is, I believe, because agebox uses an encryption mechanism (or at least, a mode of operating that mechanism) that doesn't produce repeatable, deterministic encryption. This is generally a good thing, as it defeats some known-plaintext attacks. But it has the downside that every time a file's contents is re-encrypted, the cypher text changes - and because git sees the file as binary, it treats it as 100% modified.
The git-crypt tool notes that it works around this problem, but in the absence of similar crypto nuance at the code level (which is well beyond my skills!), I think there's a possible fix we could implement, here:
I figure that because @slok has given us #143, it's now viable to write a script that:

reads the decryption passphrase from the user, once;
set up an envvar containing the passphrase (PPHRASE)
loops over the output of agebox validate (or /perhaps/ the contents of the agebox config file?):

for each Invalid secret: \"foo\" decrypted secret exists, grab the secret-id= value
we know the plaintext is present at path $SECRET_ID
the cypher text is deleted from disk, and git knows that it was at $SECRET_ID.agebox
create some temp file .agebox: tmp123456.agebox
use some git magic to cat the encrypted, in-git-storage version of the secret into the temp file:

git cat-file -p $(git ls-files --stage $SECRET_ID.agebox | awk '{print $2}') > tmp123456.agebox

compare the current plaintext on disk with the result of de-crypting the temp file with agebox cat:

diff -q $SECRET_ID <(agebox cat -i keys --passphrase-env PPHRASE temp.file.agebox) >/dev/null

if the diff exits with code 0, there were no differences: git restore $SECRET_ID.agebox and delete $SECRET_ID
if the diff exits with code >=1, there were differences; leave the plaintext file alone

Now, after the loop ends, the only plaintexts on disk are the modified ones: it's now "safe" to run agebox reencrypt

Ok, this is a bit hacky! It sticks a bunch of things together, and could probably use some refinement :-)
But I think that it's at least viable, and by writing it we'll probably come up with some feature requests for @slok, if there're ways to make the Developer Experience nicer :-)

Thanks @jpluscplusm for your response, i think another solution can be developped
When i use agebox decrypt --all it store the sha1 files in .ageboxreg.yml file (section file_ids) and when i use agebox reencrypt it re-encrypt only file changed (verified with previous sha1).
I think this feature can be develop easyly ? (sorry, i not a go developer)

Maybe agebox should not delete the original encrypted .agebox files and compare saved hashes of files with the unencrypted file.
So if a file is modified the hash will indicate it and it can be reencrypted. If another file is not modified the hashes will be the same and we just delete the unencrypted file and the file.agebox will be untouched.
the hashes can be saved in our .ageboxreg.yml and we only need to commit changed files into git

@Threnklyn I think there are a couple of nuances to work through, conceptually:

If an unencrypted hash of the plain text is ever stored in a file that is intended to be stored in version control, then some real thought has to go into potential data leakage scenarios. For example, think about very short plaintexts ("true" vs "false") in files that are /named/ in a way which, when combined with the contents of the file, leaks too much to an attacker. Having a deterministic hash visible in plaintext changes the threat model that either has to be mitigated, or the user has to be warned not to use the tool in a vulnerable way.

Is the stored hash of the plain text or the cypher text? If the former, then the comparison operation needs access to the private key. If the latter, then the sequence of operations after noticing a difference needs to be very carefully planned, and special attention given to the fact that agebox has a stated aim of being VCS-agnostic.

Just came across this discussion before trying out the tool. The non-deterministic nature of age encryption is the main reason I started looking for tools that can solve this issue conveniently in order to commit encrypted files in repos, but it looks like this is not tackled here.


Remove react-error-overlay in dev mode and field error messages

Removed react-error-overlay which was making using the playground in dev pretty unusable (this doesn't happen in production). It's a bit of a hack, but haven't figured out a proper way to turn it off just yet.
Have also updated some form fields error messages to make it easier to understand what's causing an issue with these components. It's pretty bare atm but keen on your thoughts of overall error handling of components @moroshko.

@moroshko I've updated all as we discussed if you want to take another look 👍

Great work @samuelclements, let's merge this!


Typo in README.md

When copypasting commands from the readme the second line has an extra
echo "
which may confuse someone as the terminal will expect end of the quotes...

I see now that the
echo "
probably belongs to the beginning of the last line.

Yes, that is intended. By creating the run.sh script first and then executing it, I avoid one issue: If you would pass the commands directly to the terminal, any required input (e.g. password) would be filled with the subsequent commands.


[Question]: Retrieval with LlamaIndex

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question
I mentioned that I wanted to do this in the linket issue(https://github.com/run-llama/llama_index/issues/8661  but closed) and @logan-markewich  was very helpful.
Thinking back, I thought I could do it this way:

I send my file.
In the data ingestion section, I divide it into chunks and save it to a vector db. Here I should try chroma, deeplake, qdrant and see their performance.
I perform similarity checks on embeddings consisting of files. I will send the answer to the input given according to the similarity checking results.

I would be very pleased if there is someone who has knowledge about this task that I need to do and can provide code and explanation.

@4entertainment the process you described is exactly what a VectorStoreIndex does. The bot actually gave a pretty good example.
You can see details for all vector stores here (links at the bottom)
https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_stores.html

Thanks @4entertainment. Based on the responses from both @logan-markewich and dosubot, I'm closing this issue. If you feel it needs to be re-opened please let us know.


Fix #1517 even more

Description
Fixes width and vertical alignment of the blockName input element as per https://github.com/payloadcms/payload/issues/1517#issuecomment-1693452854

[x] I have read and understand the CONTRIBUTING.md document in this repository.

Type of change

[x] Bug fix (non-breaking change which fixes an issue)

Checklist:

[ ] I have added tests that prove my fix is effective or that my feature works
[x] Existing test suite passes locally with my changes
[ ] I have made corresponding changes to the documentation

Can you provide a before/after image or animation of the fix?

I've reverted vertical alignment, since it requires more testing


zephyr bugfix: mcuboot doesn't link into code-partition

Select the code partition as the target for link into.
This patch together with
https://github.com/zephyrproject-rtos/zephyr/pull/14620
fixes the issue which was imposed while zephyr moves
DTC before Kconfig processing in its build process.
Signed-off-by: Andrzej Puzdrowski<EMAIL_ADDRESS>

cc @SebastianBoe @carlescufi @galak @lemrey @rgundi


parser not working, resulting in excalamation mark / error rather than checked

I generated a consolidated report from CAS - CAMS + KFintech at https://www.camsonline.com/Investors/Statements/Consolidated-Account-Statement
Executing the casparser cli utility does not return successfully.
Is this expected ?
Please note the Error and the Excalamation marks in the image below.

Command executed,
$ casparser <filename>.pdf -p '<$password$>'
File Type details,

File Type : FileType.CAMS
CAS Type : CASFileType.DETAILED

also, is up-to-date,
(.venv_py310) iceman@pop-os ~/D/M/Statements> casparser-isin --update
2023-08-31 00:26:23,325 - INFO - Fetching remote isin db metadata
2023-08-31 00:26:24,283 - INFO - Local db version  : 2023.8.18
2023-08-31 00:26:24,283 - INFO - Remote db version : 2023.8.18
2023-08-31 00:26:24,283 - INFO - casparser-isin database is already upto date

Thanks for the report.
Looks like the CAS format has changed recently. Shall push an updated version in a couple of days.

cool thanks, plz consider me as a reviewer.
We could have a test with the new file/format validating the changes if possible?

cool thanks, plz consider me as a reviewer.
We could have a test with the new file/format validating the changes if possible?

I've created PR #79 with support for new files. Can you please give it a try?


[SPARK-40448][CONNECT] Spark Connect build as Driver Plugin with Shaded Dependencies

What changes were proposed in this pull request?
This is a draft change of the current state of the Spark Connect prototype implemented as a driver plugin to separate the classpaths and shaded dependent libraries. The benefit of a driver plugin is that the plugin has access to everything inside Spark but can itself be a leaf-build. This allows us to shade it's own minimal set of dependencies without brining new dependencies into Spark.
How is Spark Connect Integrated?
Spark Connect is implemented as a driver plugin for Apache Spark. This means that it can be optionally loaded by specifying the following option during startup.
--conf spark.plugins=org.apache.spark.sql.sparkconnect.service.SparkConnectPlugin

The way that the plugin is integrated into the build system does not require any additional changes and the properly configured Jar files are produced as part of the build.
How are GRPC and Guava dependencies handled?
Spark Connect relies both on GRPC and Guava. Since the version of Guava is incompatible to the version used in Spark, the plugin shades this dependency directly as part of the build. In the case of Maven this is very easy in the case of SBT it's a bit trickier. For SBT we hook into the copyDeps stage of the build and instead of copying the artifact dependency we copy the shaded dependency instead.
Why are the changes needed?
https://issues.apache.org/jira/browse/SPARK-39375
Does this PR introduce any user-facing change?
Experimental API for Spark Connect
How was this patch tested?
This patch was added by adding two new test surface areas. On the driver plugin side this patch adds rudimentary test infrastructure that makes sure that Proto plans can be translated into Spark logical plans. Secondly, there are Python integration tests that test the DataFrame to Proto conversion and test the end-to-end execution of Spark Connect queries.
To avoid issues with other parts of the build, only tests that require Spark Connect will be started with the driver plugin.
Note:

The pyspark tests are only support for regular CPython and disabled for pypy
mypy was initially disabled for this patch to reduce friction. Will be enabled again.

Caveats

mypy is disabled on the Python prototype until follow-up patches fix the warnings.
The unit-test coverage in Scala is not good. Follow up tests will increases coverage, but to keep this test reasonable I did not add more code.

New Dependencies
This patch adds GRPC, Protobuf, and an updated Guava to the leaf-build of Spark Connect. These dependencies are not available to the top-level Spark build and are not available to other Spark projects. To avoid dependency issues, the Maven and SBT build will shad GRPC, Protobuf and Guava explicitly.
Follow Ups
Due to the size of the PR, smaller clean-up changes will be submitted as follow up PRs.

Can one of the admins verify this patch?

You can retrigger the ci, the issue already fix by github. https://www.githubstatus.com/incidents/d181frs643d4

To speed the base image build time, you could move new added python depends in the end of dev/infra/dockerfile when your PR still WIP.

@Yikun the Docker build works fine again, but I'm still facing the issue with the Doc build that causes my build to fail. It happens in a plain SBT build sbt clean package -Phive
https://github.com/grundprinzip/spark/runs/8226589025?check_suite_focus=true#step:21:6379
Any ideas?

not having a deep look but you didn't change any code in [1][2], so this might relate to some change in pom.xml, SparkBuild.scala.
[1] https://github.com/grundprinzip/spark/runs/8226589025?check_suite_focus=true#step:21:483
[2] https://github.com/grundprinzip/spark/runs/8226589025?check_suite_focus=true#step:21:6379

I did more investigations on the build failure. It seems that the doc build for Python (and R) is the only place where we actually call the build/sbt package target. Most of the other targets execute Test/package

My high level concern about this init PR is that it introduces a lot of code which is beyond what is required for a minimal valued product, and those code are not well tested.
IMO, a MVP is just one that demonstrates the client-server architecture, client can do a single select from a table, and server side execute the plan and return the data. This PR adds a lot more on the API surface, like join, aggregate, etc., which is beyond the MVP requirement. And because those surface are not well tested, I would expect many improvements on those surface to happen.
The direct consequence is we will need to file commits afterwards to update the dumped code. Many will fall into cleanup/bug fix/missing tests, etc. But with a MVP, this could be minimized. The following could be an example that I was thinking of:
With this init PR, we could have some commits in the future:
[SPARK-xxxxx][CONNECT] fix outer join implementation due to xxx
[SPARK-xxxxx][CONNECT] add test for non-equality join condition
[SPARK-xxxxx][CONNECT] fix issues for nested join due to xxx
[SPARK-xxxxx][CONNECT] fix issues when join condition contains a function
With MVP which does not have join on the surface at the beginning, the commits we bring in could be
[SPARK-xxxxx][CONNECT] init inner equality join support with tests. Throw exception for non-inner joins
[SPARK-xxxxx][CONNECT][FOLLOWUP] improve inner equality join and more tests
[SPARK-xxxxx][CONNECT][FOLLOWUP] bug fix for inner equality join on xxx
[SPARK-xxxxx][CONNECT] extend join types to outer joins, with tests for situations a, b, c, d...
[SPARK-xxxxx][CONNECT][FOLLOWUP] more tests for outer joins types on uncovered situations f, k, h..
For maintaining the quality of the commit history purpose, the MVP path might be better.
The longer term impact is for the first release we want for the connect project. When we plan to release the first version of it, generally speaking we won't want to release some API that are not well tested or even functional incorrect.  Eventually we either make sure each API is well implemented and tested, or we choose to block those API from being used, which already lead to that we need a version that is implemented good first, then incrementally append high quality changes. Then why don't we target to achieve it at very beginning?
My proposal is really to only keep minimal required code that demonstrates the client-server architecture and client can issue a simple operation.
This is not blocking this PR and I am open to this concern. As long as Spark community is ok for the big dump, it's good to go.  Just want to make sure some perspectives are covered and we can make good decisions on valid perspectives.

This is ready for a look now.
Since the whole feature and codes would be very large, we (explicitly I, @martin-g, @amaliujia, and @cloud-fan) discussed offline, and decided to propose to split this. This PR is basically the minimal working version  note that most of code lines here were generated from the protobuf.
SPARK-39375 is a parent JIRA, and we described the current action items at this moment.
More JIRAs will be filed accordingly to the plan below:
High-level plan and design:

High-Level Design Doc for Spark Connect
Spark Connect API Testing Plan

Low-level plan:
Short-term

Extend test coverage for SparkConnectPlanner (right now at 76% line coverage)
Extend test coverage for Spark Connect Python client
Type annotations for Spark Connect Python client to re-enable mypy
Clean-up documentation in PySpark code for Spark Connect
Documentation for PySpark in README and doctests
Proto validation in server and/or client
Validation:

Syntactic -> Parsing
Semantic -> Analysis

Alternatively only return error class to clients upon failures.
Initial DSL framework for protobuf testing
Restructure the build structure to match with other components

Maven
SBT

Long-term

Testing with custom DSL
LocalRelation
Better error handling for semantic failures
Spark and Session configurations
Scala Client
SBT incremental build and testing environment
DataSources
UDFs
Packaging / Releasing

I am thinking about merging it without making major changes in this PR if there aren't major issues found, and I myself will take a look for important/urgent items very soon according to the plan described above.
I would like to be transparent here. The frank reasons that I think as above are as follows:

Multiple people will intensively cowork together for this component but individual works in a different timezone which makes it difficult to work within this @martin-g's branch.
Difficult to manage the credibility. The whole size of work would be very huge, and I would like to avoid sharing the same credit with all the coauthors. Different person will sign off and be the author for individual change.
I would like to speed up by fully leveraging individual fork's GitHub Actions resources. Currently, @martin-g's GitHub resource here is a bottleneck.

Hope this plan and thought make sense to other committers too. Are you guys okay with this? @dongjoon-hyun @viirya @mridulm @srowen @wangyum @sunchao @huaxingao (derived from SPIP voting)

It wasn't me! :-)
You meant Martin Grund (@grundprinzip)

I will start reviewing closely from late tonight or early tomorrow assuming that we're fine with the plan shared above. I created a new component for CONNECT, see also https://github.com/apache/spark/pull/37925.

I would be ok with merging a minimal working version as long as it doesn't impact many other components and destabilize the builds and other developers activities.  If it doesn't fit this, I think it should either be changed to be decoupled or wait til we think it is.  Based on how its a plugin I would think it would be fairly decoupled but I haven't looked at the code yet.
I think we should ideally give people a few more days to give input to make sure they agree.
.
Unfortunately just seeing this so will take a high level look in the next day or so.

Thanks for your feedback. Yes, it's pretty much decoupled, and I believe this doesn't affect anything to other components. Sure, I will leave it out for more days.

There is a testing plan (Spark Connect API Testing Plan) that I and @amaliujia will work on it right away after this PR. I promise that I will make sure on the testing coverage - it's a bit difficult to work together in one branch together because of several issues (https://github.com/apache/spark/pull/37710#issuecomment-1249269885).
FWIW, there is more detailed (High-Level Design Doc for Spark Connect) here.

For clarification, I am fine with reverting the whole component if the plan isn't followed in the future.

@tgravescs I will follow up on the testing plan doc to address your comments. Please feel free to bring up anything in the doc or here.

@pan3793 thanks for the thorough review. I will address the comments shortly.

Merged to master.
I will follow up and actively work on cleaning up and followup tasks from tomorrow.

Ack, I will regenerate the protos and update.

There's an outstanding comment: https://github.com/apache/spark/pull/37710#discussion_r978291019. I am working on this.

FYI, I made a PR https://github.com/apache/spark/pull/38109 to address https://github.com/apache/spark/pull/37710#discussion_r978291019


High overhead calling compiled darkroom function.

These stats were generated using the identity function and the canny edge detection function from the examples directory being run 600 times over some image data.
identity function:
256x256:   31.0 MPix/s
1080x1920: 230.0 MPix/s
canny edge detection:
256x256:   2.6 MPix/s
1080x1920: 7.6 MPix/s
There seems to be quite some overhead involved into the call to the compiled darkroom function. As far as I can tell this is due to the compiled code performing several memory allocations, and the mmap calls aren't always cheap.
There doesn't seem to be anything in the docs about amortizing this cost. What methodology was used to get the performance specified in the paper?

Hi. It probably wouldn't be that difficult to lift those allocations outside of the function, but no, there isn't a built-in way to do that currently. When I benchmarked this for the paper, I convinced myself that this didn't change the performance by more than a percent or two (on our particular platform, example programs etc). Obviously the current interface is a lot simpler as a result. There is also thread launch overhead btw.
IMO I still don't think these fixed overheads are significant. But I suppose it's possible there was a regression. Feel free to contact me<EMAIL_ADDRESS>directly with more info about what you're trying to do and I can try to help you out.


ngHide/ng-show directive don't  animate in ng 1.4.4

when i update to angular 1.4.4, there is a strange thing -- some animation i define for the element with ng-hide directive doesn't seem to work. Then i check the api docs. Ok, [the 1.4.4 version docs of ng-show] (https://code.angularjs.org/1.4.4/docs/api/ng/directive/ngShow)'s example plunkr seems to have no animation too.
Does Angular have enough tests before the new version is about to release?

Can you please provide a demo of the problem? Otherwise, we cannot do anything to fix the problem.

Just look how work https://code.angularjs.org/1.4.4/docs/api/ng/directive/ngShow#example or http://plnkr.co/edit/y2JZH3xTB9MI6VXZ0o4b?p=preview
when check labels angular don't add intermediates classes:
.ng-hide-add
.ng-hide-add-active,
.ng-hide-remove
.ng-hide-remove-active

I think that proble related with #12612, #12583

Ok, I see now. @matsko can you take a look at this? It seems pretty bad if even the "official" ng-hide animations aren't working.

Turns out that the API docs example code is incorrect. Transitions always need to be applied to the non active CSS class:
http://plnkr.co/edit/JzI0NnxtPdySR2sMcGeW?p=preview
Or on the class that is being added or removed:
http://plnkr.co/edit/EdhLQkurZRRYjhmSUrrX?p=preview
Or on a shared CSS class that is always there:
http://plnkr.co/edit/HX9BQwtwJ5WvylbWVOa3?p=preview
Placing it on the active CSS classes may have worked before, and I will look into making the fix properly for 1.4.5. Here is an issue to keep note of that:
https://github.com/angular/angular.js/issues/12638
In the meantime here is a commit that will clean up the example in the docs a bit so nobody else comes across this issue:
https://github.com/angular/angular.js/pull/12637

OK 1.4.5 will fix this entirely: http://plnkr.co/edit/y2JZH3xTB9MI6VXZ0o4b?p=preview

Sorry I provided the wrong link earlier. Here is a patched version of angular.js that fixes the issue: http://plnkr.co/edit/YLTe1SObEAaQn99iOiNv?p=preview
This patch will debut in 1.4.5

thx! =)


Rules for picking unsafe pointer types

*const vs *mut and interior mutability, Rc::into_raw
std::ptr::Shared may return the wrong one

Sufficiently few crates use raw pointers in public API that I don't think this would be worthwhile as an API guideline. I imagine an eventual unsafe code guidelines would cover this in depth.


joint_state_broadcaster/JointStateBroadcaster re-load appends states

Hi,
upon re-loading the JointStateBroadcaster via service call, as indicated here https://discourse.ros.org/t/ros2-control-controller-restart/24662/2?u=mhubii, it appears that the published states are appended to the already existing ones. For example, if one publishes the states for joint_0, re-loading will cause states to be published twice for joint_0. Re-loading once more, will cause the states to be published 3 times and so on.
I would guess this happens because joint_names_ in JointStateBroadcaster are pushed-back, but never cleared in the JointStateBroadcaster::on_activate method, e.g. https://github.com/ros-controls/ros2_controllers/blob/76bf6ae70c432565c8767fba14c9adc75028292e/joint_state_broadcaster/src/joint_state_broadcaster.cpp#L260
I am running the foxy distributed binaries. Please feel free to close if I am wrong. Thanks for the help.

You are probably right. Can you provide t up a PR against master please? I can backport to foxy and galactic and put out releases sasap


feat: witness generation

The actual data needed for "WitnessedOperations" is up in the air, so consider this a dummy.
In any case, we can do proving based on this alone.

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#60  👈
#57
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @rphmeier and the rest of your teammates on  Graphite

Merge activity

Apr 4, 3:42 AM EDT: @pepyakin started a stack merge that includes this pull request via Graphite.


[SPARK-14929][SQL] Disable vectorized map for wide schemas & high-precision decimals

What changes were proposed in this pull request?
While the vectorized hash map in TungstenAggregate is currently supported for all primitive data types during partial aggregation, this patch only enables the hash map for a subset of cases that've been verified to show performance improvements on our benchmarks subject to an internal conf that sets an upper limit on the maximum length of the aggregate key/value schema. This list of supported use-cases should be expanded over time.
How was this patch tested?
This is no new change in functionality so existing tests should suffice. Performance tests were done on TPCDS benchmarks.

cc @davies

Test build #57024 has started for PR 12710 at commit f48eba1.

Test build #57026 has started for PR 12710 at commit 3c0ff54.

Test build #57024 has finished for PR 12710 at commit f48eba1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57024/
Test PASSed.

LGTM
Merging this into master, thanks!

Test build #57026 has finished for PR 12710 at commit 3c0ff54.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57026/
Test PASSed.


Confusing inferred names of anonymous classes

🔎 Search Terms
inferred name anonymous class expando
🕗 Version & Regression Information

This is the behavior in every version I tried

⏯ Playground Link
https://www.typescriptlang.org/play?ts=5.5.0-dev.20240304#code/MYewdgzgLgBAhgJwQRhgXhgbQLoCheIqYAM26MwANnBBDAN4C++hyuA9OzAHoD8+uUJFhxyyYviHQYIAEYArcvVwx4ALgrVaDZs0HhpCAKYQArpVgY58gHRwOXVT35A
💻 Code
const arr1 = []

arr1[0] = class {}

arr1
// ^? const arr1: (typeof 0)[]

const a = 10

const obj = {
  a: class {}
}

const result = obj.a
//    ^? const result: typeof a

🙁 Actual behavior
Symbols of anonymous classes are given inferred names based on the properties to which they are assigned to and that inferred name is used by typeof operator to refer to them
🙂 Expected behavior
I think that in both of those situations, the typeof inferredName display is somewhat confusing. However, the inferred name algorithm comes from JS spec so perhaps it's just "it is what it is" kind of situation.
This display doesn't convey any information about the scope that owns that name so the instinct is to look in the scope within which the declaration is. That name can exist there as a local binding and thus the information is a little bit confusing.
I get that this is... well, a convoluted situation :P I'm primarily reporting this because it's just something interesting that I've noticed here while investigating something entirely different. I don't necessarily care about the fix or this behavior - I'm just looking for some "working as intended"/"bug" confirmation here.
That said, I think that at the very least the inferred name should not come from an array index when dealing with auto-array types. Inferred names don't come from there in JS:
var arr1 = [class {}]
arr1[0].name // ''

var arr2 = []
arr2[0] = class {}
arr2[0].name // ''

Additional information about the issue
No response

(typeof 0)[] does not make any sense

Do u classify the other case (presented by obj) as working as intended?

Acknowledging that it's inconsistent, I think it'd be better to say (typeof obj)['a'], but at least typeof a is likely to be sensical in cases where a is a real identifier

Sensical, but misleading.  In the example given, typeof a at that position is number.


Cleanup error reporting providers

Error reporting in providers/ is changed to make better use of the newer ERR building blocks, and to drop the old function codes everywhere.

Not quite sure I understand the separation..
Aren't quite a few of the files in crypto (e.g crypto/rsa) include into the provider (and these contain RSAErr?)

A thought, should all of the *_F_* macros be deprecated?

Not in this PR.  util/mkerr.pl adds them automatically, and I believe that we're already treating that problem elsewhere

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set.  Please review the updates and set the label manually.

Merged
6debc6ab7499745baf7123e69936a9997dbbe6cb Convert all {NAME}err() in providers/ to their corresponding ERR_raise() call


Add Avana Wallet

Hi @jordansexton , this PR adds Avana Wallet. Our adapter includes 458f108 recent import/type fixes changes from yesterday. We have reviewed #384, given it has been referenced in most new wallet PR discussions.
We have implemented this adapter on our website - you can view a live working integration at https://www.avanawallet.com/.
We are available to answer questions.
Cheers,
Patrick

Can you run prettier on this before I review it? yarn fmt from the project root should do the trick.

Can you run prettier on this before I review it? yarn fmt from the project root should do the trick.

Sure - just updated it for yarn fmt.

Will be released today.


Inline emojis are not rendered properly on Safari 10.1.1

This is browser specific.
on Safari

on Firefox

@brockwhittaker can you reproduce this?  Would be great to resolve this soon.

Since this only impacts chat.zulip.org (it's related to #3923 which we've deployed for now), downgrading the priority.

@HarshitOnGitHub did we end up fixing this issue on your branch?

@timabbott I am using Ubuntu, hence can't tell whether it has been fixed in #3923.

OK, let's just close this for now.  We can address it again when we merge #3923 if it's there; it's pretty obvious :)

Yup! this is definitely fixed 👍 🏅


fixed changes

Draft

Sent

Replied

right now the UX and BA are not 100% on what state they actually want/need i am keeping them as this reference all of COH possible states which could be relevant in the near future


Use new headless  mode

https://developer.chrome.com/articles/new-headless/
 Puppeteer old Headless deprecation warning:
    In the near feature `headless: true` will default to the new Headless mode
    for Chrome instead of the old Headless implementation. For more
    information, please see https://developer.chrome.com/articles/new-headless/.
    Consider opting in early by passing `headless: "new"` to `puppeteer.launch()`
    If you encounter any bugs, please report them to https://github.com/puppeteer/puppeteer/issues/new/ch.

Default value is new.
# new(default)
html-export-pdf-cli index.html --headless new -o result.pdf
# true
html-export-pdf-cli index.html --headless true -o result.pdf
# false
html-export-pdf-cli index.html --headless true -o result.pdf


Add current balance to MakeAPayment.

Maps api.account.balance to props of the MakeAPayment component.
Added CSS to line up with existing form.

Closes #2704

Coverage decreased (-0.03%) to 68.17% when pulling d3a0ae0c6f71a7e8cdc36f6261f3a2168a5c61c5 on rjstires:feature-2704 into 2ddedf583d5fe309a2dade0b95e3e22a8aec6423 on linode:develop.

Could you throw a screenshot in here of it?

Dependency: 2723

Coverage decreased (-0.03%) to 68.327% when pulling 860bd2cc58a55d44f0bc70c873c39485041d2bf9 on rjstires:feature-2704 into 219a84ad47e6a716fc29fb64b204b8b0fb4faa43 on linode:develop.


Imports using providesModule broken in the latest Jest

After originally using 0.9.x with no problems, I moved to update to the latest today 0.12.x, and find that my test suite has problems resolving any of my modules.
My React Native code base makes heavy usage of @providesModules MyComponentName in order to also use import MyComponentName from 'MyComponentName.
This worked fine in 0.9.x, but in the latest I'm receiving many variations on this error:
Runtime Error
  - Error: Cannot find module 'SKZBracket' from 'SKZBracket-spec.js'
        at Resolver.resolveModule (node_modules/jest-resolve/src/index.js:117:17)
        at Runtime._resolveModule (node_modules/jest-cli/src/Runtime/Runtime.js:214:32)
        at Object.<anonymous> (app/ObjectModel/__tests__/SKZBracket-spec.js:9:1)
        at jasmine2 (node_modules/jest-jasmine2/src/index.js:252:16)

After changing all imports in SKZBracket-spec.js to be relative, the problem then moves to complaining about the imports in SKZBracket itself. ie:
Runtime Error
  - Error: Cannot find module 'I18n' from 'SKZBracket.js'
        at Resolver.resolveModule (node_modules/jest-resolve/src/index.js:117:17)
        at Runtime._resolveModule (node_modules/jest-cli/src/Runtime/Runtime.js:214:32)
        at Object.<anonymous> (app/ObjectModel/SKZBracket.js:9:1909)
        at Object.<anonymous> (app/ObjectModel/__tests__/SKZBracket-spec.js:12:39)

In case it's possible that this is just an issue with my setup, here is the current Jest configuration I'm using in my package.json:
  "jest": {
    "globals": {
      "__DEV__": true
    },
    "setupTestFrameworkScriptFile": "<rootDir>/jestSupport/setup-jasmine-env.js",
    "testPathIgnorePatterns": [
      "/node_modules/",
      "packager/react-packager/src/Activity/"
    ],
    "collectCoverage": true,
    "testFileExtensions": [
      "js"
    ],
    "unmockedModulePathPatterns": [
      "promise",
      "source-map",
      "react"
    ]
  }

Here is my .bablerc:
{
  "presets" : [
    "jest",
    "react-native"
  ]
}

Finally here is the setup-jasmine-env.js file referenced above:
var reporters = require('jasmine-reporters');
var junitReporter = new reporters.JUnitXmlReporter({
    savePath: 'testoutput',
    consolidateAll: false,
    filePrefix: 'xmloutput',
});

jasmine.getEnv().addReporter(junitReporter);
jasmine.VERBOSE = true;

(Note the setup-jasmine-env.js file is currently not outputting reports, and my efforts to upgrade to the latest Jest are so that I can continue working towards getting jasmine-reporters to work with the latest Jest)

Here's a repository that shows the issue. Switch devDependency between 0.9.x and 12.x to see the issue in action: https://github.com/MattFoley/jest-resolve-issue

Hey @MattFoley,
sorry that I made you wait so long. We made a change to how we detect "@providesModule" to be closer to how we use it at Facebook. You'll need to use a real docblock:
/**
 * @providesModule Car
 */

Note that the first line is /** instead of /*. Sorry that I'm putting you through the trouble to update all your files but thank you for providing a repro that enabled me to figure this out :)
Also, you can install jest-repl now which allows you to have a repl where you can require modules that have @providesModule annotations :)

Hi I'm having the same problem with jest version  13.2.3. I also made sure that the format has two stars like below.
/**

@providesModule Start
*/

I also installed jest-repl. Do I need to have do a special configuration to use jest-repl or will it automatically get picked up when I run the test using "npm test"?
Any help would be greatly appreciated.

@ranjitpandit can you provide a repo that shows this problem just like @MattFoley did? That would be very helpful :)

in the process of creating a sample repo to show the problem I stumbled upon a solution that worked.
I had a license and author disclaimer at the top and then the providedModule annotation such as below.
/**
 * Created  on 7/18/16.
 * all rights reserved
 */

/**
 * @providesModule Types
 */

If I put the providedModule section at the top then Jest is able to pick it up. Subtle but hard to figure out. It would be good if the order doesn't matter as it's easy to get the order wrong.
/**
 * @providesModule Types
 */

/**
 * Created  on 7/18/16.
 * all rights reserved
 */

Same experience here. I had a 'use strict' at the top. I just moved it below the comments and everything works


When searching with _security:not , deleted resources are included in the search results.

Describe the bug
When the _security:not search parameter is specified in a search operation, IDs for soft-deleted resources are included in the search results. I've reproduced the behaviour with a fresh install of version 4.0.160 , but it appears at least as far back as in version 3.4.124 .
FHIR Version?
R4
Data provider?
SQL Server
To Reproduce
Steps to reproduce the behavior:

POST a Patient resource to the fhir-server instance. The contents are not important, for repro purposes even an empty Patient resource is enough.
DELETE the resource that was created in step 1.
SEARCH for resources using the _security:not parameter, for instance /Patient?_security:not=V . It needs to be a generic search, if the ID for the resource is specified with _id=... the behaviour does not appear, but adding _lastUpdated=... to limit the results if testing on a server with lots of existing data works.

Expected behavior
The deleted resource does not appear in the list of results.
Actual behavior
The deleted resource appears in the list as an empty resource (containing only the id and meta.lastUpdated fields).

Thank you for the bug report.  I have also been able to repro the issue and confirmed it.  We'll get it on the list to fix.

@feordin - can this be closed?

Yes, this is resolved.


Feat/raae 194/standard session search index

This PR replaces the list implementation of Standard Session Manager to use a hash implementation.

@justin-cechmanek just need to resolve merge conflicts and get tests working, otherwise go ahead an merge away!


Features missing in UE 5.0.3

Hi, I'm using the plugin in UE 5.0.3, but I only see the option of refreshing all the nodes from all the blueprints:

I can't do it in single BP or folders. The following is the result of right-clicking on a folder:

I also can't find the option of automatically compile after refreshing in "Projects settings":

Am I doing something wrong? In case it is helpful here I describe what I did for installing the plugin

I've downloaded the .tar.gz file from here
Extracted and moved the result to the Plugins folder
Downloaded the 3 files UnrealEditor-RefreshAllNodes.dll, UnrealEditor-RefreshAllNodes.pdb, UnrealEditor.modules, from the same link.
Moved those files to the Plugins\RefreshAllNodes-1.2-UE5EA2\Binaries\Win64. Also trying to make it work I've removed the previous UE4 files that were there, but I didn't solve the issue.

Did I missed something?

You did nothing wrong. Those features were added in version 1.3, and I hadn't released the binaries for that version yet. I just built the new binaries and released them. So it should work now :)
I can't do it in single BP or folders

Version 1.3 lets you refresh the blueprints in a folder via the context menu in the content browser, but I will note that you cannot do it to single blueprints-- if you need to do it to just one blueprint then you can just open the blueprint and run "Refresh All nodes" from under the File menu at the top (that's built into UE by default). :)
Hope it helps :)

I think the .zip and .tar files from the release 1.3 were not build. At least the Binaries folder doesn't appear in my case.

I'm very for the inconvenience-- I had built that last release incorrectly.
Please try the latest release, it should be easier and work cleanly :)

Great. now it is just straightforward! Super simple, thank you :)


Responsive

Proposed changes
Describe the big picture of your changes here to communicate to the
maintainers why we should accept this pull request.
If it fixes a bug or resolves a feature request, be sure to link to
that issue.
Types of changes
What types of changes does your code introduce to the project: Put
an x in the boxes that apply

[x] Bugfix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Documentation (improve documentation)
[ ] Continuous Integration (functionality that improve automation or testing)
[ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)

Checklist
Put an x in the boxes that apply. You can also fill these out after
creating the PR. If you're unsure about any of them, don't hesitate to
ask. We're here to help! This is simply a reminder of what we are going
to look for before merging your code.

[ ] I have read the CONTRIBUTING doc
[ ] I have signed the Contributor License Agreement
[ ] I have updated the RELEASE NOTES
[ ] I have added tests that prove my fix is effective or that my
feature works
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged and published in
downstream modules

Further comments
If this is a relatively large or complex change, kick off the discussion
by explaining why you chose the solution you did and what alternatives
you considered, etc...

not sure if I am doing something wrong, but this is what is see using this branch and Iphone 12 size

one mobile phones ui is still not usable
what part of the UI?

not sure if I am doing something wrong, but this is what is see using this branch and Iphone 12 size

have you cleaned up everything before testing?.
looks like to me the branche before the changes:

have you cleaned up everything before testing?. looks like to me the branche before the changes:

indeed, after cleaning the build, the view is now as in your screenshot, we should still:

reduce the title size (is really too much in mobile screen)

reduce space when filters go on multiple lines

reduce the title size (is really too much in mobile screen)

maybe this is a good solution: https://mui.com/material-ui/customization/theming/#responsivefontsizes-theme-options-theme


https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard contains broken links

URL(s) with the issue:
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard
Description of issue (what needs changing):
The doc for the embeddings_metadata argument contains a broken link. It says "See the details about metadata files format.", and the link redirects to https://www.tensorflow.org/how_tos/embedding_viz/#metadata_optional, which is broken.

@atn832 ,
Can you please share us the exact link where embeddings_metadata was located, so that we can fix it? Thanks!

@rmothukuru maybe I'll look on this. Actually what @atn832 is trying to say that the in the TensorBoard docs, the embeddings_metadata has an link for reference but seems it's broken. So it must be either replaced or removed

So, I'd like to take this... may I know what it needs to be done? Look at the site and suggest me

Hmm, I'll make a PR on this so that it'll be clear


SLIM-3190 Identification number

Successful nightlies: https://ci.opensmartgridplatform.org/view/Nightly/job/OSGP_open-smart-grid-platform_nightly/1396/


Input isn't blurred on clear

Hello again 😺
The Input isn't blurred when you clear the selection, it should go back to it's "empty" visual state with the floating label as a placeholder.
Here is a repro :

Hello 😄
Do you think it makes more sense to blur the input on clear (like you mentioned)...

or re-focus the input (allowing for new entry) but also allowing the user to click away to blur.

Currently there is a bug (that you demonstrated) where you can not unblur the input, so one of these should be put in place.  I was kind of leaning towards the second, but there is benefit to both.  With the "X" representing both "clear input" and "clear selection" it's kind of a tradeoff for one or the other.
Oddly, to implement the first (blur the input), I had to first focus the input and then blur (document.activeElement showed <body> so I wasn't sure what to blur otherwise).  Probably has to due with the clear button disappearing from the dom, but input gets in limbo somehow.  Material-UI appears to track the focused state internal with a lot of workarounds, which is likely the culprit.  While digging around, I also saw their <Select> would ignoreNextBlur (which I'm not using, but another case of some tricking blur behavior).  It's always been one of React's weaker areas.
Anyways, both options are implemented, just let me know which sounds good to you.

Also, I changed the clear icon from material-ui-icons/Cancel to material-ui-icons/Clear.  I swear this one didn't exist when I looked for it on https://material.io/icons/.

I went with option 2 (based on your bullet 4 in #10 and personal preference).  I pushed this as part of 1.0.0-7 which you can grab with yarn install mui-downshift@next.  Also take a look at the storybook.


Bump gitlab-ci-pipelines-exporter chart version

To fix release job

thanks @Nevoff89, could you update the patch version instead please? (0.2.9)

thanks @Nevoff89, could you update the patch version instead please? (0.2.9)

Done

cheres 🙇 !


Add an API compatibility guide for pandas and PySpark users

Codecov Report

Merging #901 into master will decrease coverage by 2.15%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #901      +/-   ##
==========================================
- Coverage    94.3%   92.14%   -2.16%
==========================================
  Files          34       34
  Lines        6213     6213
==========================================
- Hits         5859     5725     -134
- Misses        354      488     +134

Impacted Files
Coverage Δ

databricks/koalas/usage_logging/__init__.py
24.54% <0%> (-72.73%)
:arrow_down:

databricks/koalas/usage_logging/usage_logger.py
50% <0%> (-50%)
:arrow_down:

databricks/koalas/__init__.py
77.5% <0%> (-7.5%)
:arrow_down:

databricks/conftest.py
91.48% <0%> (-6.39%)
:arrow_down:

databricks/koalas/frame.py
94.38% <0%> (-1.58%)
:arrow_down:

databricks/koalas/namespace.py
85.01% <0%> (-1.31%)
:arrow_down:

databricks/koalas/plot.py
93.33% <0%> (-0.96%)
:arrow_down:

databricks/koalas/utils.py
97.24% <0%> (-0.69%)
:arrow_down:

databricks/koalas/testing/utils.py
77.35% <0%> (-0.63%)
:arrow_down:

databricks/koalas/series.py
95.41% <0%> (-0.16%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e38e54...15ad4d7. Read the comment docs.

Had an offline discussion. Seems it's not good to lose index column in this example. Let me take a look tomorrow.

Looks good. Thanks.

Let me update this PR soon per #906

Let me merge this one.


Art: re-organize Google Drive

Right now all of the GDrive contents are decently organized (by artist) but they are often not particularly well labeled.
We need to:

clean up the naming to describe the purpose and features of the model that distringuish it, rather than who made it/when
make sure the model loads by default to perform its main function. For example, if it's intended to copy paste like a weapon then the render mode/layout should be ready to do that without reconfiguration needed
upload a screenshot of every model, named the same as the model, so previews are fast

Additionally, I've been running into several cases where I could not find the model in the GDrive at all (such as the silk blob .blend file).

That's an older proof of concept file early from #1338, which was marked as complete, but it's not clear which is the correct file.
I'm going through and filling out descriptor boxes on my own assets since that's fast, but a screenshot pass is definitely a good idea.
Generally speaking I've been filing things as follows, since it seems to make the most sense going from Github issue to direct file:
Assignee (creator) > category > item > folder with most recent iteration, and archive of older files.

Assignee (creator) > category > item > folder with most recent iteration, and archive of older files.

Yes, that seems like the best approach. I think it's the responsibility of each assignee to organize their folders.

Re: storage, we can re-own the directories/folders to Webaverse org so it's not a burden on anyone's account.

Been working on arranging, adding descriptions and screenshots to my drive.
I've been uploading using the image links from github directly - makes the most sense in terms of grabbing the most recent screenshot IMO.

Been working on arranging, adding descriptions and screenshots to my drive.

Thank you!

I've been uploading using the image links from github directly - makes the most sense in terms of grabbing the most recent screenshot IMO.

Yeah, that makes sense, as long as the file hasn't really changed since then.

This was not finished: https://github.com/webaverse/app/issues/1892


[DOC] User paths in tutorial notebooks (tool suggestion)

Describe the issue linked to the documentation
I noticed there are some user paths in the tutorial notebooks. Example: https://www.sktime.org/en/latest/examples/feature_extraction_with_tsfresh.html
/Users/mloning/Documents/Research/software/sktime/sktime/sktime/transformations/panel/tsfresh.py:164: UserWarning: tsfresh requires a unique index, but found non-unique. To avoid this warning, please make sure the index of X contains only unique values.
  "tsfresh requires a unique index, but found "
Feature Extraction: 100%|██████████| 5/5 [00:11<00:00,  2.32s/it]

Suggest a potential alternative/fix
I recently created a (very beta) tool that strips user paths from Jupyter notebook output cells. It can be used with a pre-commit hook. https://github.com/bdice/nb-strip-paths
Here is an example of the behavior:
# When executing from a repository "my_awesome_code", this text in a Jupyter cell output...
/home/bdice/my_awesome_code/some_directory/file.txt
# becomes...
some_directory/file.txt

Here is an example pre-commit hook configuration:
https://github.com/glotzerlab/signac-examples/blob/5478bb68517789fdd5b2831059059f062e46153a/.pre-commit-config.yaml#L51-L54
I don't have time to contribute a PR at the moment but I hope this is useful.

@bdice good idea! @mloning can I add it to the pre-commit yaml file?

Martin Walter<EMAIL_ADDRESS>Mercedes-Benz AG on behalf of Daimler TSS GmbH.
https://github.com/Daimler/daimler-foss/blob/master/LEGAL_IMPRINT.md

@bdice Thanks for reaching out, sounds like a great idea! What does "very beta" mean? How many other repos are using this already?
Happy to try it out if you want to implement it @aiwalter, but may have to remove it again if we run into bugs.

@mloning @aiwalter Good questions -- it's less than a month old. I made nb-strip-paths for example notebooks for the signac data management framework but I think there are quite a few projects in the SciPy ecosystem with embedded notebooks that might find it handy. I'm using the hook with projects developed by my research lab but there aren't any other users yet.
nb-strip-paths is meant to be a very small package with a singular purpose, and I think it is stable enough to use in your pre-commit hooks (especially since pre-commit hooks are usually pinned to a tag).
To strip the user paths, you may need to re-render the notebooks locally so that the path to the repository matches your path. For example, paths containing /Users/mloning/.../sktime/abc would not be stripped to sktime/abc if the current repository path is /home/aiwalter/.../sktime/abc. This is why it's helpful to have as a pre-commit hook, because it ensures that the user making a commit will strip their own paths from the outputs.
Thanks for the consideration, I'd be happy to hear your feedback!

@bdice I tried your package in  a separate test repo and produced two test cases which were however not detected. I created some error messages in a notebook and got two paths like...

C:\Users\MYUSERID\bla\bla\bla\lib\site-packages\sktime\forecasting\compose\__init__.py
~\AppData\Local\bla\bla\bla\lib\site-packages\pandas\io\common.py

In both cases I would ideally expect to strip the path, however after a commit nothing has been modified. Did I do sth wrong?
nb-strip-paths...........................................................Passed

@aiwalter Currently the only paths that are stripped are paths to the repository containing the notebooks. The user site packages directory is not stripped (since it is not a subdirectory of the repository containing the notebooks). I would be happy to add support for the two cases you’re mentioning — can you tell me what behavior you would expect in each case?

@bdice
Is there actually in general a way to strip ANY occurrence of my USERID in a notebook (and even other files)? In cells as well as error messages?
Regarding my two examples, I would like to propose the following solutions. We could add additional argument to nb-strip-paths in order to...

strip any path but only keep the file name only? (most secure)
Strip any path before site-packages e.g.? (less secure)
Just strip any USERID occurances in a notebook

For me its actually very important not to expose my USERID as I am using my company PC :) So I would prefer solution 1 for sktime. Or 2 in combination with 3. What do you think?

Martin Walter<EMAIL_ADDRESS>Mercedes-Benz AG on behalf of Daimler TSS GmbH.
https://github.com/Daimler/daimler-foss/blob/master/LEGAL_IMPRINT.md

@aiwalter @mloning Sorry for the slow response, I didn't see your reply until now! I like your suggestions and have some follow-up questions before I work on the implementation.

strip any path but only keep the file name only? (most secure)

This sounds prone to false positives but could be useful with some clarification. For example, strings like 2021/01/01 or his/hers or alpha/beta/gamma could be relative paths to a file. Similarly, regular expressions printed like /abc/g could appear to be absolute paths. This would also need to be a fairly high-priority rule (applied before the other rules that are more specific). Thus, I think you would want to detect only absolute paths for potential stripping/truncation. You could also use something like os.path.exists to verify if it's really a path on the current system before stripping/truncating. What do you think of that?

Strip any path before site-packages e.g.? (less secure)

I like this proposed rule! This is a narrower scope and thus easier to apply without worrying about false positives. Any path-like string that includes site-packages as a substring could have its prefix (the part before site-packages) replaced with an ellipsis. This would look like .../site-packages/more/stuff.py. That seems very safe and unlikely to cause problems for readers.
For consistency, we could also have an option to change the paths to be UNIX-like with forward slashes. That would make it more likely that the output of a notebook executed on Windows, Linux, or Mac would be the same.

Just strip any USERID occurances in a notebook

In this case, would you expect that the output of the cell below would be empty after stripping the username? Or would you suggest a replacement such as the literal string $USER?
import getpass
print(getpass.getuser())

@bdice sry as well, I was off a few days.

file-name-only (1 above)
strip-site-packages-prefix (2 above)
strip-repo-root-prefix (the current behavior)
strip-username (3 above)

This sounds very cool and would be nice to have all those four options as argument each and a short description, if possible.

In this case, would you expect that the output of the cell below would be empty after stripping the username? Or would you suggest a replacement such as the literal string $USER?

I would prefer replacement with USER

You could also use something like os.path.exists to verify if it's really a path on the current system before stripping/truncating. What do you think of that?

That would be very smart and cool to have!
Let me know once you have this implemented, so I can test it again and then add it to sktime :)

Martin Walter<EMAIL_ADDRESS>Mercedes-Benz AG on behalf of Daimler TSS GmbH.
https://github.com/Daimler/daimler-foss/blob/master/LEGAL_IMPRINT.md

@aiwalter Cool! I appreciate your input. I’ll follow up when I have a chance to add these features.


Popup version does not work.

Trying to add something like this in my own local test does not produce the popover as shown in the demo.

Even in the material demo page though, if you click cancel, the calendar wont pop up again anymore.

I have the same issue as well. Exactly what you have stated.

same here i cant make it work using Popup version

same issue here.

I think I've figured this out, hopefully this helps someone (although I'm sure all of you other commenters have picked something else a long time ago):
You need to copy the directives from the extra files loaded - so for Bootstrap for example the directives are in this file http://simeonc.github.io/sc-date-time/javascripts/bootstrap.js
These directives have a dependency to AngularStrap which is loaded from here in the demo files https://cdnjs.cloudflare.com/ajax/libs/angular-strap/2.3.3/angular-strap.min.js