Unable to resolve hostnames after deployment

General information

OS: Windows 10
Hypervisor: Hyper-v
Did you run crc setup before starting it (Yes/No)? yes
Running CRC on: Laptop

CRC version
# Put `crc version` output here
```crc version: 1.10.0+9025021 Openshift version 4.4.3 (embedded in binary)

## CRC status
```bash
# Put `crc status --log-level debug` output here

C:\Users\A04440\.crc>crc status --log-level debug
DEBU Running 'C:\Users\A04440\.crc\bin\oc.exe get co -ojson --kubeconfig C:\Users\A04440\.crc\machines\crc\kubeconfig --context admin --cluster crc'
DEBU Command failed: exit status 1
DEBU stdout:
DEBU stderr: error: CreateFile C:\Users\A04440\.crc\machines\crc\kubeconfig: The system cannot find the file specified.
DEBU error: CreateFile C:\Users\A04440\.crc\machines\crc\kubeconfig: The system cannot find the file specified.
ERRO Error getting config for native Go SSH: open C:\Users\A04440\.crc\machines\crc\id_rsa: The system cannot find the file specified.

## CRC config
```bash
# Put `crc config view` output here
``` there was nothing

## Host Operating System
```bash
# Put the output of `cat /etc/os-release` in case of Linux
# put the output of `sw_vers` in case of Mac
# Put the output of `systeminfo` in case of Windows
```Host Name:                 9RCFXT2
OS Name:                   Microsoft Windows 10 Enterprise
OS Version:                10.0.18363 N/A Build 18363
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Member Workstation
OS Build Type:             Multiprocessor Free
Registered Owner:
Registered Organization:
Product ID:                00329-00000-00003-AA802
Original Install Date:     2/20/2020, 12:47:04 PM
System Boot Time:          5/27/2020, 2:23:52 PM
System Manufacturer:       Dell Inc.
System Model:              Latitude 7490
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 142 Stepping 10 GenuineIntel ~1910 Mhz
BIOS Version:              Dell Inc. 1.13.1, 11/8/2019
Windows Directory:         C:\WINDOWS
System Directory:          C:\WINDOWS\system32
Boot Device:               \Device\HarddiskVolume1
System Locale:             en-us;English (United States)
Input Locale:              en-us;English (United States)
Time Zone:                 (UTC-05:00) Eastern Time (US & Canada)
Total Physical Memory:     32,627 MB
Available Physical Memory: 15,378 MB
Virtual Memory: Max Size:  37,491 MB
Virtual Memory: Available: 17,021 MB
Virtual Memory: In Use:    20,470 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    arrownao.corp.arrow.com
Logon Server:              \\USSDENDC016P
Hotfix(s):                 8 Hotfix(s) Installed.
                           [01]: KB4552931
                           [02]: KB4497727
                           [03]: KB4517245
                           [04]: KB4537759
                           [05]: KB4538674
                           [06]: KB4541338
                           [07]: KB4552152
                           [08]: KB4556799
Network Card(s):           5 NIC(s) Installed.
                           [01]: Intel(R) Dual Band Wireless-AC 8265
                                 Connection Name: Wi-Fi
                                 Status:          Media disconnected
                           [02]: Intel(R) Ethernet Connection (4) I219-LM
                                 Connection Name: Ethernet
                                 Status:          Media disconnected
                           [03]: PANGP Virtual Ethernet Adapter
                                 Connection Name: Ethernet 5
                                 Status:          Media disconnected
                           [04]: Realtek USB GbE Family Controller
                                 Connection Name: Ethernet 3
                                 DHCP Enabled:    Yes
                                 DHCP Server:     <IP_ADDRESS>
                                 IP address(es)
                                 [01]: <IP_ADDRESS>
                                 [02]: fe80::149e:13ab:efac:e21b
                                 [03]: 2600:1702:7e0:11f0:45c3:2118:501f:437d
                                 [04]: 2600:1702:7e0:11f0:149e:13ab:efac:e21b
                                 [05]: 2600:1702:7e0:11f0::50a
                           [05]: Hyper-V Virtual Ethernet Adapter
                                 Connection Name: vEthernet (Default Switch)
                                 DHCP Enabled:    No
                                 IP address(es)
                                 [01]: <IP_ADDRESS>
                                 [02]: fe80::6d2d:2e13:3065:3bc1
Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.

### Steps to reproduce

  1. crc setup
  2. crc start
  3.  get dns error Failed to query DNS from host: lookup api.crc.testing: no such host
  4.

### Expected
running cluster

### Actual
cannot get cluster to run and continue to use it

### Logs

Before gather the logs try following if that fix your issue
```bash
$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

got errors with crc delete -f access denied and crc cleanup error removing
Please consider posting the output of crc start --log-level debug  on http://gist.github.com/ and post the link in the issue.
crcstartoutput.txt

Not very conclusive from the errors. it seems DNS related, but is PANGP Virtual Ethernet Adapter perhaps a VPN adapter? If so, were you connected when you tried CRC?

@lcory It also looks like you have incomplete state from a previous run, as some files are missing.. Cleanup you env and start from a fresh env..
$ crc delete
$ crc cleanup
$ crc setup
$ crc start

I have tried both on VPN and off. Get DNS error when not on VPN also. ERRO Failed to query DNS from host: lookup api.crc.testing: no such host
I have tried the 4 crc steps multiple times also, with same results.
Any other ideas?

You can use command "crc ip" to get an ip and then add this two lines to windows hosts file.
       api.crc.testing
       console-openshift-console.apps-crc.testing

Please do not test DNS resolving ON VPN. Make sure you
disable/disconnect the VPN, perform the deployment (after a delete)
and try to resolve the address.
Make sure you also use 1.11.0 as there is an issue with certs with
1.10 that prevent a deployment.
Note that putuo is right about adding the records, but these would
only work for the basic functionality and not when services are
exposed.
On Thu, Jun 4, 2020 at 4:35 PM puttuo<EMAIL_ADDRESS>wrote:

You can use command "crc ip" to get an ip and then add this two lines to windows hosts file.
api.crc.testing
console-openshift-console.apps-crc.testing
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

--
Gerard Braad | http://gbraad.nl
STEM is the source, but BUILD is how you develop!
[ Better Understanding Involves Learning and Doing ]

I have verified v1.11.0 and started from beginning totally off of the VPN. I still get the same issue. Does it matter where I run crc.exe from? Is there a specific location needed to run it? I have the folder I have it stored at in the path, but if it has to be a specific location, I might not have it in the correct place.

It matters; the executable can onl,y be run from WINDRIVE if not mistaken.
This is a limitation/known issue in the used library. We would still like
to fix this, but had no time to investigate further
On Sat, Jun 6, 2020 at 4:19 AM lcory<EMAIL_ADDRESS>wrote:

I have verified v1.11.0 and started from beginning totally off of the VPN.
I still get the same issue. Does it matter where I run crc.exe from? Is
there a specific location needed to run it? I have the folder I have it
stored at in the path, but if it has to be a specific location, I might not
have it in the correct place.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/code-ready/crc/issues/1261#issuecomment-639778658,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAAOZQTU52NBSM4AEFV4YDRVFHMPANCNFSM4NNLAINQ
.

--
Gerard Braad | http://gbraad.nl
STEM is the source, but BUILD is how you develop!
[ Better Understanding Involves Learning and Doing ]

@gbraad What do you mean by WINDRIVE? the Windows c: drive? Is there a specific folder it must be in?

WINDRIVE is a variable that exists inside a Windows session that points to
the drive Windows is installed on, this can be C:, D:, or anything you
used.
This is specific to your setup. it does not have to be in a specific
folder. Your temp, download or user folder would be OK (if they are on the
same drive)
Corporate network shares are to be avoided. So, I usually create a
[C/D]:\Applications\ folder that contains downloaded applications
On Wed, Jun 10, 2020 at 12:56 AM lcory<EMAIL_ADDRESS>wrote:

@gbraad https://github.com/gbraad What do you mean by WINDRIVE? the
Windows c: drive? Is there a specific folder it must be in?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/code-ready/crc/issues/1261#issuecomment-641443736,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAAOZROTZB2CCBPIILJZQ3RVZSVBANCNFSM4NNLAINQ
.

--
Gerard Braad | http://gbraad.nl
STEM is the source, but BUILD is how you develop!
[ Better Understanding Involves Learning and Doing ]

I got pinged about this question. I thought this was handled as the question got diverged by the OP? Confused...
So to confirm. The error shows: ERRO Error getting config for native Go SSH: open C:\Users\A04440\.crc\machines\crc\id_rsa: The system cannot find the file specified. which indicates a possible issue in the way the commands get run or the setup.
Make sure you use PowerShell or the standard Command Terminal.
Run:
PS> crc delete -f
PS> crc cleanup
PS> crc setup
PS> crc start

and let me know if this is reproducible.

Closing this since reporter is not responding now.


services.IRequestInformation.user has no data

Morning.
I'm having issues with services.IRequestInformation.user.
When I try to call this, it appears that it's not storing any data. I'm simply trying to hide services based on the groups the user is assigned to.
For example, we have the standard 'Jenny' user and I try to simply log if this user is an Analyst as per the test below:
if (requestInformation.user.groups.includes('Analyst')) {
        console.log("I AM AN ANALYST!");
      } else {
        console.log("I SHOULD NOT PRINT!");
      }

Running the above, it logs I SHOULD NOT PRINT.
If I try to log what groups the user is in, it is blank.
Could you point me in the direction of getting this to work?

Hi @CharlesMarriott-VA
I suspect, as you say it is empty,  that the reason you are experiencing this is that the connector has not been set up to send the sensitive header information required for this.
Looking at our documentation I can see we don't seem to mention this anywhere.
I did find reference to it when searching our  knowledge base  as part of some Fixpack notes here
At the bottom of that it says:
Enabling user information to be sent to a connector in header messages
By default, user information is not sent to a connector in header messages. To enable this information being sent to the connector do the following:

Update topology.xml and add send-sensitive-headers="true" to all appropriate connector entries.
For example:
<connector base-url="[http://localhost:3700/"](http://localhost:3700/)" name="Example Connector" id="example-connector" send-sensitive-headers="true"/>
Redeploy and restart each Liberty server.

In the deployed Liberty/Gateway  you can see this gets written to connectors.json
We will get the documentation improved so that we include reference to this.
Cheers

Hi @CharlesMarriott-VA,
it sounds like you're trying to generate user specific configurations. You can find information and an example of how to do this here: https://i2group.github.io/analyze-connect/content/example-connectors/user-configuration.html#enable-user-specific-configurations
The connector needs to be configured to declare that it is going to provide user specific configurations, and the i2 Connect Gateway needs to be configured to send sensitive header information. Both of these are illustrated in the link above.

Hi @CharlesMarriott-VA
I am just checking to see if, given the responses above, you were able to resolve your issue?
Cheers

Closing this issue as the responses given should have solved this and there has been no reply.


forbidden: 11.2 -> 12.5

Diff: https://github.com/ivan-sincek/forbidden/compare/refs/tags/v11.2...v12.5
Changelog: https://github.com/ivan-sincek/forbidden/releases/tag/v12.5
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 344284 run on x86_64-linux 1

2 packages built:

forbidden
forbidden.dist


Synthesis failed for firestore

Hello! Autosynth couldn't regenerate firestore. :broken_heart:
Here's the output from running synth.py:
Cloning into 'working_repo'...
Switched to branch 'autosynth-firestore'
Running synthtool
['/tmpfs/src/git/autosynth/env/bin/python3', '-m', 'synthtool', 'synth.py', '--']
synthtool > Executing /tmpfs/src/git/autosynth/working_repo/Firestore/synth.py.
synthtool > Ensuring dependencies.
synthtool > Pulling artman image.
latest: Pulling from googleapis/artman
Digest: sha256:d3df563538225ac6caac45d8ad86499500211d1bcb2536955a6dbda15e1b368e
Status: Image is up to date for googleapis/artman:latest
synthtool > Cloning googleapis.
synthtool > Running generator for google/firestore/artman_firestore_v1.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/php/google-cloud-firestore-v1.
synthtool > Running generator for google/firestore/artman_firestore.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/php/google-cloud-firestore-v1beta1.
synthtool > Replaced 'Copyright \\d{4}' in src/V1beta1/Gapic/FirestoreGapicClient.php.
synthtool > Replaced 'Copyright \\d{4}' in src/V1beta1/FirestoreClient.php.
synthtool > Replaced 'Copyright \\d{4}' in tests/Unit/V1beta1/FirestoreClientTest.php.
synthtool > Replaced 'Copyright \\d{4}' in src/V1/Gapic/FirestoreGapicClient.php.
synthtool > Replaced 'Copyright \\d{4}' in src/Admin/V1/Gapic/FirestoreAdminGapicClient.php.
synthtool > Replaced 'Copyright \\d{4}' in src/V1/FirestoreClient.php.
synthtool > Replaced 'Copyright \\d{4}' in tests/Unit/V1/FirestoreClientTest.php.
synthtool > Replaced 'Copyright \\d{4}' in tests/Unit/Admin/V1/FirestoreAdminClientTest.php.
synthtool > Replaced 'Copyright \\d{4}' in tests/System/Admin/V1/FirestoreAdminClientSmokeTest.php.
synthtool > Running generator for google/firestore/admin/artman_firestore_v1.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/php/google-cloud-firestore-admin-v1.
synthtool > Replaced 'Copyright \\d{4}' in src/Admin/V1/Gapic/FirestoreAdminGapicClient.php.
synthtool > Replaced 'Copyright \\d{4}' in src/Admin/V1/FirestoreAdminGrpcClient.php.
synthtool > Replaced 'Copyright \\d{4}' in tests/Unit/Admin/V1/FirestoreAdminClientTest.php.
synthtool > Cleaned up 1 temporary directories.
synthtool > Wrote metadata to synth.metadata.

Changed files:
M Firestore/src/Admin/V1/resources/firestore_admin_client_config.json
 M Firestore/src/V1/resources/firestore_client_config.json
 M Firestore/src/V1beta1/resources/firestore_client_config.json
 M Firestore/synth.metadata
[autosynth-firestore 5e07579] [CHANGE ME] Re-generated Firestore to pick up changes in the API or client library generator.
 4 files changed, 6 insertions(+), 3 deletions(-)
To https://github.com/googleapis/google-cloud-php.git
 + 42f312d...5e07579 autosynth-firestore -> autosynth-firestore (forced update)
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/git/autosynth/autosynth/synth.py", line 223, in <module>
    main()
  File "/tmpfs/src/git/autosynth/autosynth/synth.py", line 219, in main
    gh.update_pull_labels(pr, add=[api_label])
  File "/tmpfs/src/git/autosynth/autosynth/github.py", line 181, in update_pull_labels
    labels=list(label_names),
  File "/tmpfs/src/git/autosynth/autosynth/github.py", line 166, in replace_issue_labels
    response.raise_for_status()
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.github.com/repos/googleapis/google-cloud-php/issues/1918/labels

Google internal developers can see the full log here.

Autosynth passed, closing! :green_heart:


fix bug of verifyMultiRowRootsToDataRootTupleRoot

The original _root parameter is not verified before being used in verifyMultiRowRootsToDataRootTupleRootProof, it will allow arbitrary _rowRoots to be proved valid.

Summary by CodeRabbit

Refactor

Updated the verification process in the application to enhance data integrity checks.

Thanks for reporting this 🙏
Can you provide test cases that fail without this fix?

I don't have a test case, I found this problem when code review.

@zhiqiangxu I confirm the bug, would you be able to add the tests? Also, this fix should also be done for: verifySharesToDataRootTupleRoot and verifyRowRootToDataRootTupleRoot. Would you be down to updating this PR to include fix + tests for those too?
If not, I'll gladly pick this PR up and add the necessary changes :D

@zhiqiangxu I confirm the bug, would you be able to add the tests? Also, this fix should also be done for: verifySharesToDataRootTupleRoot and verifyRowRootToDataRootTupleRoot. Would you be down to updating this PR to include fix + tests for those too?
If not, I'll gladly pick this PR up and add the necessary changes :D

The last commit should already fix them. And the existing tests are updated, basically removing the redundant root parameter.
Feel free to add more tests:)

@zhiqiangxu I confirm the bug, would you be able to add the tests? Also, this fix should also be done for: verifySharesToDataRootTupleRoot and verifyRowRootToDataRootTupleRoot. Would you be down to updating this PR to include fix + tests for those too?
If not, I'll gladly pick this PR up and add the necessary changes :D

I fixed the mentioned functions, please take a look, and feel free to add more tests:)


Incorrect road classification for driveways in Finland; overnoding

Hi Microsoft Open Maps Team,
It seems that you've added a large number of rural driveways in Finland like this one https://www.openstreetmap.org/way/1150093621 and incorrectly classified them as highway=track.
Referencing OpenStreetMap practices of road classification that its Wiki describes, the entry for highway=track states:

Some examples where highway=track is generally not appropriate:

A road that is minor but serves as a connection in the general-purpose road network. See highway=unclassified.
A minor road with permanent residences along it. See highway=residential.
A minor road providing primary access to one or several permanent residences (driveway). See highway=service.
A minor road providing primary access to a commercial or industrial facility. See highway=service.
A trail or path that is not wide enough for a typical four-wheeled motor vehicle. See highway=path, footway, cycleway, and bridleway.

If none of the above are true for a road, and its primary use is access to the surrounding land, highway=track is likely appropriate. Some examples:

A farm track used for access to crop fields, pastures, orchards or similar agricultural land.
A forest road used for outdoor recreation, logging, or other forestry purposes (aka, forest track, woods road, logging road)
A fire road used for fighting wildfires in grassland, scrubland, and forestland.
A minor road used for access to conservation land.
A dirt track used for access to the mountains, desert, or other remote area.

Also at Examples:

Please note that highway=track should be chosen based on function, not simply because a road looks like one.
However, driveways or even rural public roads can look like tracks as well. In those cases a different highway=* classification should be used.

In short, highway=track is supposed to be used only for roads that are primarily for agriculture or forestry, not residences.
The road in question – and other similar roads that are not part of the general public road network and only lead to residences – are supposed be classified as highway=service, and, where applicable, service=driveway, if that road ends at a residence.
In other words, roads that are basically dead ends and lead nowhere but residences, summer cottages or industry, are to be tagged highway=service, and service=driveway where applicable.
If road topology is such that roads form an enclosed loop and are in public use, they are tagged highway=unclassified or above.
At the bottom is an illustration of this.
Roads like these are tracks https://www.openstreetmap.org/way/327563497 https://www.openstreetmap.org/way/465443951, since they do not lead to buildings, and are only used for agriculture and forestry.
—
Another issue with the road I linked in the beginning is that it suffers from "overnoding"; it has a very large number of nodes that are not strictly functional.
When the Simplify-function in the JOSM editor is applied with a 2 meter accuracy, 35 of its 48 nodes can be deleted, so the road could have been drawn with 73% less data and the road would still have reasonable accuracy.
Overnoding puts unnecessary load on the database, so it is encouraged that roads are drawn with reasonable accuracy, not perfect. JOSM's Simplify does a very good job of cleaning up unnecessary nodes.
I have supplied another illustration of this below.  It is a bit messy and unclear but I hope it illustrates how unnecessary nodes can be safely culled while retaining accurate road geometry.
—
I request that MSFTOpenMaps

changes all the previously uploaded roads to the appropriate classifications,
changes their classification methods so that they follow the wider OpenStreetMap practices,
changes their importing/uploading methods so that roads are cleaned up from overnoding before uploading so that unnecessary data that burdens the database is no longer uploaded.

Thank you.

Dear ctphel thank you for reaching out. We have always been mindful in our mapping, trying to be respective towards community efforts and policies. With that in mind we will approach this seriously and carefully review our work as well as community policies and inputs and resources you've provided. We'll reach out soon with information on any changes and clarifications if these are necessary.


Sort Collections by Last Updated

Please Select if your Request is Either something new or an Enhancement

[x] Enhancement of an existing Feature.
[ ] Request of a new feature.

Please select the area your request applys to. (Multiple selections are Possible)

[ ] Onboard. Initial vault setup and import/export pods
[ ] Create. Note creation, lookup, snippets and templates
[ ] Retrieve. Backlinks, references, graph view
[ ] Structure. Refactoring, multi-vault and schemas
[x] Publish. Sharing your repo with the world
[ ] Misc (Choose this if your not sure)

Is your feature request related to a problem? Please describe
I would like to be able to list pages by last updated as my 'reviews' of games are not final, but growing and I feel a created-by sort list would give the wrong impression.
Describe the solution you'd like
Collections can sort by created already, so it seems like a small change could be made to permit a declarable attribute.
Describe alternatives you've considered
A proper query system would work, but that seems overkill for this use-case.
Additional context
I'm happy to try myself if I can figure it out! :)

Thank you for the feature request.
Currently our publishing features are moving from 11ty to nextjs, so this may have to come after nextjs based publishing is on par with 11ty's feature sets.
That being said, it seems like a small enough fix (and a useful one too) so I'll leave this as a good first task for the community 🙂

We can do the following to make this happen:

introduce the sort_by attribute as part of a note, with values being created|updated

created is default behavior
updated sorts by updated

This should be a oneline change:

updating toCollection to sort by created if _.get(note, "custom.sort_by") === ''created'

@BiosElement if your still interesting in making this change, you can find our dev docs in our contribution guide


Added my name to people.yml

Your name's already on the list from a previous commit. You're good, no merging needs to be done!


Rsa based commitments

@mancabizjak I think I applied all suggestions, except the last one - it is somehow difficult to name it reasonably.

Closing in favor of #31

Closing in favor of #31.


Use API Facade instead of request url

In my project, I use API Facades to fetch data and saga looks like
export const handleGetItem: Saga = function*({
  payload,
}: PayloadAction<typeof ActionEnum.API_GET_REQUEST, Payload>): IterableIterator<any> {
  const { id }: Payload = payload;
  const facade: Facade = yield getContext(SagaContextEnum.facade);

  try {
    const storeItem: IItem = yield call(() => facade.getItem(id));
    yield put(actions.successGetItem(storeItem));
  } catch (err) {
    yield put(actions.errorGetItem(err));
  }
};

How can I pass facade.getItem(id) to the redux-saga-request action creator??
Is it possible to implement this feature in this library??

So you mean you dont want to interact with your server with REST?
If yes, probably you will need to write a facade driver. https://github.com/klis87/redux-saga-requests/blob/master/packages/redux-saga-requests-axios/src/axios-driver.js is how it is done for axios, you can also find fetch and graphql drivers here

@klis87, My facade uses axios
import { AbstractFacade, IFacadeConfig } from 'my-lib';
import { AxiosResponse } from 'axios';
import { ItemModel } from 'items';
import { ItemsServerType } from './types';
export class Facade extends AbstractFacade {
  constructor(config: IFacadeConfig) {
    super(config);
  }

  async getItems(id: string): Promise<ItemsServerType> {
    const response: AxiosResponse<ItemsServerType> = await this.client.get(`/${id}/list`);
    return response.data;
  }
}

How should I create the driver?
I initiate facade and access it using sagaContext.
I also need to pass the Facade method to the action creator.
How hard is to implement these features above you library (if I fork it)!

@strdr4605 the question is why do you need it? in my opinion this is just overabstraction. This library removes any need to create api modules like you did, instead of getItems method of your class, you can easily do:
const getItems = id => ({
  type: 'GET_ITEMS',
  request: {
    url: `/${id}/list`
  }
});

Alternatively you could write promise middleware like recommended here: https://github.com/klis87/redux-saga-requests/issues/244
But promise middleware imho makes sense with some external APIs like firebase which provides you some functions/classes to interact with API. But for in-house API I am sure my suggestion will simplify your codebase a lot. I know that class can keep some common state for you api methods, but everything could be solved without class, if you have a use case which you think requires facade, let me know and I will show you how this can be handled with this library

@klis87, The backend provides the facades that frontend uses. If API URL is changed the Facade may remain and no changes in frontend required.

@strdr4605 ok, so I recommend to write promise driver like in mentioned topic. In the future I might add official promise driver, I will add it to my todo list

@klis87 is it possible to use redux-saga selectors with this library?
Lets say I have the id in my store. And I don't what to pass it as payload for action, but use select from redux-saga/effects and get the id and put in in url?

@strdr4605 of course, if you dispatch request action like yield put(getItems(id)), then id can be from yield select
personally though, If I had some id which is part of many request actions, I would just use redux thunk, which makes your sagas clean, or works if you dispatch actions from react too:
function getItems() {
  return (dispatch, getState) => {
    const id = idSelector(getState());

    return dispatch({
      type: 'GET_ITEMS',
      request: {
        url: `/${id}/list`
      }
    };
  };
}

In my apps I sometimes do also another thing, kind of magical but I find it very convenient:
import { mapRequest } from 'redux-saga-requests/lib/helpers';

const interpolatePlaceholders = (config, { user }) => {
  if (config.url.includes('{user}')) {
    config = { ...config, url: config.url.replace('{user}', user) };
  }

  return config;
};

function* onRequest(request) {
  const user = yield select(userIdSelector);

  return mapRequest(request, config =>
    interpolatePlaceholders(config, { user }),
  );
}

By using request interceptor, I can then define actions like:
const getItems = () => ({
  type: 'GET_ITEMS',
  request: {
    url: '/{user}/list'
  }
});

@klis87, Interesting approach, but if I use https://github.com/klis87/redux-saga-requests/issues/244#issuecomment-487880194 I can't modify promise with interceptor :smile: . I will discuss with the team about using this package without facades (not sure if the team will agree).
Maybe we can create our implementation for our needs.

@strdr4605 yeah, for facade you would need to use either thunk approach, or adjust my placeholder solution to allow passing callback to request, like:
const getItems = () => ({
type: 'GET_ITEMS',
request: ({ idFromSelectorInonRequest }) => myFacade.getSth(idFromSelectorInonRequest)
});
The only limitation is your imagination, adjusting my example could also be:
const getItems = () => ({
type: 'GET_ITEMS',
request: {
url: ({ user }) => /${user}/list
}
});
You are free to do any transformation of request action in onRequest before request is fired

@klis87, Sorry for bothering so much :relaxed: but I implemented
export function createRequestsActions(env: EnvInterface): any {
  const placePickerFacade: PlacePickerFacade = new PlacePickerFacade({
    axiosConfig: {
      baseURL: env.PLACE_PICKER_URL,
    },
  });

  const requestGet: any = (merchantId: string, sessionId: string): object => ({
    type: LocationsActionEnum.API_LOCATIONS_GET_REQUEST,
    payload: {
      request: placePickerFacade.getStoreLocations(merchantId, sessionId),
    },
  });

  return {
    requestGet,
  };
}

export const createDriver: () => Driver = (): Driver => ({
  requestInstance: null,
  getAbortSource(): null {
    return null;
  },
  abortRequest(): void {},
  async sendRequest(promise: Promise<StoreLocationServerResponse>): Promise<StoreLocationServerResponse> {
    return promise;
  },
  getSuccessPayload(response: StoreLocationServerResponse): StoreLocationServerResponse {
    return response;
  },
  getErrorPayload(error: Error): Error {
    return error;
  },
});

export const locationsReducerNew: Reducer = requestsReducer({ actionType: LocationsActionEnum.API_LOCATIONS_GET_REQUEST });

and the action is dispatched but no success or error action triggered

Can be a problem that I did not implement the driver correctly?

I can see 2 issues:
export const createDriver: () => Driver = (): Driver => ({ which is duplicated return
async sendRequest(promise: Promise): Promise {
return promise;
},
u return promise in async function, so actually you return promise of promise, remove async

@klis87, () => Driver is the type for typescript of the createDriver, so there is no dublication.
async sendRequest(promise: Promise<StoreLocationServerResponse>): Promise<StoreLocationServerResponse> {
    return promise;
  },

Typescript requires the function to be async if it returns a promise. From what I understood the driver should handle it correctly.
Anyway, thank you for helping! You can label this issue as a question and let it open or close!

@strdr4605 I am surprised as
const x = () => {
  return new Promise(resolve => resolve(1));
};

does not show any TS errors.
Anyway, I am closing this issue in favour of #244


Update ingress-controllers.md

I think what is meant is "to provide a default ingress CONTROLLER"

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Remember to delete this note before submitting your pull request.
For pull requests on 1.15 Features: set Milestone to 1.15 and Base Branch to dev-1.15
For pull requests on Chinese localization, set Base Branch to release-1.14
For pull requests on Korean Localization: set Base Branch to dev-1.14-ko.<latest team milestone>
If you need Help on editing and submitting pull requests, visit:
https://kubernetes.io/docs/contribute/start/#improve-existing-content.
If you need Help on choosing which branch to use, visit:
https://kubernetes.io/docs/contribute/start#choose-which-git-branch-to-use.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

/assign @bradtopol

@serdarosmanonur Please consider signing the CLA without which we cannot approve this change.

@tengqm I think it is done. Could you check, please?

/lgtm

/approve


Replaced VTune report parser with a more capable one that doesn't get confused with C++ overloads/demangled names

Also includes payload to list line numbers where functions were defined from pdbs.

@rgesteve, What's the update on this pull request? Is this a feature you still want to integrate into PTVS?

Yes, this is to integrate into PTVS for VTune support.  @huguesv had asked for unit tests and this PR update includes some.  More are forthcoming.

Sorry, read the wrong end of this, tests are in PR #4906, the current one can be closed, only left open to reference earlier comments in the code.

@rgesteve Are you still working on this PR?
I wonder whether I can discard this PR.

The other PR has all the changes from this one. I'll close it and hopefully we can still see the review comments, which I hadn't repeated in the PR.


add a way to use a containerview to add a tabstrip in there

its easier than trying to configure the uicollectionview correctly

closing. seems like it won't be merged.


[SMALLFIX] Ignore hadoop-1 for set mode test.

The set mode semantics in hadoop-1 is different. Ignoring the load metadata mode test for hadoop-1 to fix master build with -Phadoop-1 -PhdfsTest

@aaudiber

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/12266/
Test PASSed.

LGTM


Correction for raw affiliation Centre Atlantique de Philosophie, 7463, EA, Nantes, France

Correction needed for raw affiliation Centre Atlantique de Philosophie, 7463, EA, Nantes, France
raw_affiliation_name: Centre Atlantique de Philosophie, 7463, EA, Nantes, France
new_rors: 059ynac98
previous_rors:
works_examples: W4306181632
contact: 5d669f725376bc012d8793633a919d52:dbdc5f42bdb18fad125e163c938d34 @ univ-nantes.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-10. The new affiliations should be visible within the next 7 days.


K8's 1.31 support for Prometheus-federator

Add k8's 1.31 support for Prometheus-Federator

Pass : Verified in rancher v2.10.0-alpha3 on local and downstream cluster. Below are the test cases.
Local Cluster

Provisioned Rancher with local k8s v1.30.1+rke2r1 cluster.
Installed prometheus-federator:105.0.0-rc.1+up0.4.3-rc.1
Create a new project, namespace with at least one workload on it so you can track pod metrics.
Under the cluster view hit monitoring > project monitors and click the project you created.
Verify that metrics are flowing and graph are displaying correct data.
Upgraded Local cluster to k8s v1.31.1+rke2r1
Verified configured project still projecting logs correct metrics and data.
Delete Prometheus Chart

Downstream Cluster

Provisioned downstream k8s v1.30.1+rke2r1 cluster.
Installed prometheus-federator: 105.0.0-rc.1+up0.4.3-rc.1
Create a new project, namespace with at least one workload on it so you can track pod metrics.
Under the cluster view hit monitoring > project monitors and click the project you created.
Verify that metrics are flowing and graph are displaying correct data.
Upgraded Local cluster to k8s v1.31.1
Verified configured project still projecting logs correct metrics and data.
Delete Prometheus Chart

above issue was intermittent and validated again. Test passed as it is not reproducible.
Pass : Verified in rancher v2.10.0-alpha7 on local and downstream cluster. Below are the test cases.
Local Cluster

Provisioned Rancher with local k8s v1.30.1+rke2r1 cluster.
Installed prometheus-federator:105.0.0-rc.1+up0.4.3-rc.1
Create a new project, namespace with at least one workload on it so you can track pod metrics.
Under the cluster view hit monitoring > project monitors and click the project you created.
Verify that metrics are flowing and graph are displaying correct data.
Upgraded Local cluster to k8s v1.31.1+rke2r1
Verified configured project still projecting logs correct metrics and data.
Delete Prometheus Chart .
Install fresh prometheus federator chat and repeat above validation steps on local cluster.

Downstream Cluster

Provisioned downstream k8s v1.30.1+rke2r1 cluster.
Installed prometheus-federator: 105.0.0-rc.1+up0.4.3-rc.1
Create a new project, namespace with at least one workload on it so you can track pod metrics.
Under the cluster view hit monitoring > project monitors and click the project you created.
Verify that metrics are flowing and graph are displaying correct data.
Upgraded Local cluster to k8s v1.31.1
Verified configured project still projecting logs correct metrics and data.
Delete Prometheus Chart.
Install fresh prometheus federator chat and repeat above validation steps on downstream cluster.


Fix issue #8 with IP address for custom network

HI,
First of all thanks for your work, I fixed for my needs the issue #8 , for custom network .
Comment are welcom I hope it will help !

LGTM
Thanks a lot.


[SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion

What changes were proposed in this pull request?
OrcFilters.createBuilder has exponential complexity in the height of the filter tree due to the way the check-and-build pattern is implemented. We've hit this in production by passing a Column filter to Spark directly, with a job taking multiple hours for a simple set of ~30 filters. This PR changes the checking logic so that the conversion has linear complexity in the size of the tree instead of exponential in its height.
Right now, due to the way ORC SearchArgument works, the code is forced to do two separate phases when converting a given Spark filter to an ORC filter:

Check if the filter is convertible.
Only if the check in 1. succeeds, perform the actual conversion into the resulting ORC filter.

However, there's one detail which is the culprit in the exponential complexity: phases 1. and 2. are both done using the exact same method. The resulting exponential complexity is easiest to see in the NOT case - consider the following code:
val f1 = col("id") === lit(5)
val f2 = !f1
val f3 = !f2
val f4 = !f3
val f5 = !f4

Now, when we run createBuilder on f5, we get the following behaviour:

call createBuilder(f4) to check if the child f4 is convertible
call createBuilder(f4) to actually convert it

This seems fine when looking at a single level, but what actually ends up happening is:

createBuilder(f3) will then recursively be called 4 times - 2 times in step 1., and two times in step 2.
createBuilder(f2) will be called 8 times - 4 times in each top-level step, 2 times in each sub-step.
createBuilder(f1) will be called 16 times.

As a result, having a tree of height > 30 leads to billions of calls to createBuilder, heap allocations, and so on and can take multiple hours.
The way this PR solves this problem is by separating the check and convert functionalities into separate functions. This way, the call to createBuilder on f5 above would look like this:

call isConvertible(f4) to check if the child f4 is convertible - amortized constant complexity
call createBuilder(f4) to actually convert it - linear complexity in the size of the subtree.

This way, we get an overall complexity that's linear in the size of the filter tree, allowing us to convert tree with 10s of thousands of nodes in milliseconds.
The reason this split (check and build) is possible is that the checking never actually depends on the actual building of the filter. The check part of createBuilder depends mainly on:

isSearchableType for leaf nodes, and
check-ing the child filters for composite nodes like NOT, AND and OR.
Situations like the SearchArgumentBuilder throwing an exception while building the resulting ORC filter are not handled right now - they just get thrown out of the class, and this change preserves this behaviour.

This PR extracts this part of the code to a separate class which allows the conversion to make very efficient checks to confirm that a given child is convertible before actually converting it.
Results:
Before:

converting a skewed tree with a height of ~35 took about 6-7 hours.
converting a skewed tree with hundreds or thousands of nodes would be completely impossible.

Now:

filtering against a skewed tree with a height of 1500 in the benchmark suite finishes in less than 10 seconds.

How was this patch tested?
There are no changes in behaviour, and the existing tests pass. Added new benchmarks that expose the problematic behaviour and they finish quickly with the changes applied.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

cc @dongjoon-hyun @cloud-fan - it'd be great if you get a chance to review this! I know it's a bit of a large change, but I tried to make it as isolated as possible and explain it well - let me know if I can be of additional help. (Also, a large part of the diff delta comes from the regenerated benchmark results AFAICT.)

ok to test

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8845/
Test PASSed.

Test build #103425 has started for PR 24068 at commit 4eae044.

Just remembered that I should probably apply these changes to the hive module as well. Might be more efficient to do so after we settle on whether and how this is going in so that we don’t have to apply the potential code review changes twice.

Test build #103425 has finished for PR 24068 at commit 4eae044.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103425/
Test PASSed.

@cloud-fan @dongjoon-hyun ping - let me know if someone else would be better suited to review this, or if there are things I can do to make the review easier!

I checked the implementation of orc filter builder, it creates ExpressionTree internally. ExpressionTree is a public class, can we use it directly at Spark side?

ExpressionTree is a public class, but it doesn’t expose any public constructors so we can’t instantiate it without reflection or the like.
The way I understand this is:

some parts of the hive codebase work with the ExpressionTree directly so the class is public;
however, the builder performs a bunch of optimizations on the tree, which appears to be the reason one can’t just directly create an ExpressionTree (so that the SearchArgument always has the builder transformations).

@cloud-fan I pushed a new version that has the same complexity and behaviour but doesn't need the FilterWithConjunctPushdown class. This should help eliminate some of the complexity.
While I agree that building an ExpressionTree directly might result in easier to follow code, I also think it's a much riskier change than this one. A couple of reasons:

Without being completely familiar with the Hive codebase, I can't be entirely confident that the transformations performed by the SearchArgumentBuilder are strictly for optimization purposes. There might be other parts of the codebase that depend on the fact that a SearchArgument will have certain properties - which are achieved by creating it from the builder. Using reflection to create the underlying representation directly is obviously not the intended purpose of this code and I'm not confident it will be correct.
This PR as it is now doesn't change any external behaviour at all. The resulting ORC SearchArguments will be exactly the same as they were before the change. Using ExpressionTree with reflection instead will be a full rewrite, and especially combined with 1. above, it can generate SearchArguments that are different from the current ones.

So, even if the changes introduced because of reasons 1. and 2. above are both correct, they could still end up producing ORC predicates that are less efficient than the current ones because they haven't been transformed / optimized in the proper way by the builder, and thus introduce performance regressions.
I'm okay taking a stab at the ExpressionTree change, but I'm curious to hear more about which parts of the current solution you find complicated. The way I think about it, this change contains the following at a high level:

Split the check and build phases of creating a SearchArgument. Even this change on its own is enough to fix the complexity, taking it from exponential to quadratic. And while it introduces some code duplication, I think it doesn't introduce additional complexity - I actually find it easier to understand the behaviour this way, with the checking and the building separated, and without non-obvious effects resulting in exponential complexity.

Memoize the results from check. The results from check don't change, so they don't need to be checked multiple times per node.

After getting rid of the FilterWithConjunctPushdown class in the latest version, this is pretty much all that is left here. We can get rid of the memoization as well, but I don't think that's worth it - it's < 10 lines of obvious code, and it makes the complexity easier to analyze.
Let me know what you think about the newest version and my thoughts above, and thanks for taking a look!

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9193/
Test PASSed.

Test build #103826 has started for PR 24068 at commit eb3c158.

I agree with you that building ExpressionTree directly is not a good idea, but there is another idea I'd like to try.
The problem we have is, the orc filter builder has no way back, so we have to do an extra check before calling the builder methods.
The idea I have is, create a tree-like orc filter representations by our own (similar to ExpressionTree), and convert the Spark filters to this tree representations. After the tree is built, call the real orc filter builder to create the SearchArgument.

@cloud-fan interesting! A couple of questions to make sure we're on the same page:

If I understand your suggestion correctly, on a high-level it would roughly result in trimming the Spark Filter to only the parts of the tree that are convertible to SearchArguments, and then converting these directly without the need for checking. Is that an accurate representation of your idea, or do you have something else in mind? If it's on the same page, I can think about it some more and take a stab at implementing.

What's your optimization function between the different ideas here? Are you thinking about ways to decrease code length, code duplication, complexity / maintainability? Want to make sure we're on the same page about this as well so we can evaluate the ideas similarly.

ah yes, that's better! We don't need a new tree-like representation, just reuse the Spark filter tree, with unconvertible part trimmed.

The idea is to traverse the tree only twice: one for trimming unconvertible filters, one for building orc filters.

Test build #103826 has finished for PR 24068 at commit eb3c158.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103826/
Test PASSed.

To clarify with regards to 2. - this is exactly the behavior that this PR introduces. There’s effectively a single pass throughout the tree to check for convertibility, and then a second pass to build the ORC filter. While the build phase calls isConvertible for each node, this doesn't cause additional passes across the tree since the results have already been computed in the initial checking pass.
I’ll think about the details of implementing the trimming and take a stab at an implementation if it seems like it can be made cleaner.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10530/
Test PASSed.

Test build #105249 has started for PR 24068 at commit 25b8ce1.

@cloud-fan after some noticeable delay, I managed to find the time to implement the tree trimming idea yesterday. I think the result looks pretty nice - thanks for the suggestion!
I ran the relevant parts of the benchmark and the performance appears to be comparable - maybe ~10% slower than the previous version. I imagine this is due to external factors, but since it's such a small difference compared to going from exponential to linear, I don't think it's worth looking into it. I haven't updated the benchmark results in the commit with the most recent code since doing so took a couple hours last time I tried, and the differences are not huge. Let me know if you'd like me to update this.
Looking forward to hearing your thoughts!

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10552/
Test PASSed.

Test build #105276 has started for PR 24068 at commit 2119c12.

Test build #105276 has finished for PR 24068 at commit 2119c12.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105276/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10667/
Test PASSed.

Test build #105404 has started for PR 24068 at commit a146b05.

Test build #105404 has finished for PR 24068 at commit a146b05.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105404/
Test PASSed.

@cloud-fan I took a stab at a slightly different approach to structuring the code in https://github.com/IvanVergiliev/spark/pull/2/files . The idea is to implement filtering and building in the same match expression, with an enum that tells us whether to perform a filter or a build operation. This has the following benefits:

All the logic for a given predicate is grouped logically in the same place. You don't have to scroll across the whole file to see what the filter action for an And is while you're looking at the build action.
You can't really add a new predicate to the set of filtered predicates without also defining a Build action for it - this fails the exhaustiveness check on ActionType.

The only annoying part is the need for the Either to allow the filter and build actions to return different types. I experimented a bit with using type members on ActionType so that I can define the correct return type for each action and then use that as the return type for performAction, but I don't think this is really expressible in Scala. I don't think this is a big problem.
Overall I'm pretty happy with that version of the code. Let me know if that looks reasonable to you, and if so -  I can merge that branch into this one and we can then wrap up the review.

that sounds like a good idea! Can you merge it and fix the conflict? thanks!

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11071/
Test PASSed.

Test build #105816 has started for PR 24068 at commit 28feafe.

Test build #105816 has finished for PR 24068 at commit 28feafe.

This patch fails due to an unknown error code, -9.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105816/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11079/
Test PASSed.

Test build #105823 has started for PR 24068 at commit b19792c.

@cloud-fan this should be ready for review.

I merged the changes to have filtering and building in the same function / the same case-match.
I rebased against master and applied the relevant changes.
https://github.com/apache/spark/pull/24598 had added tree trimming as part of OrcFilters.convertibleFilters. I reverted that as we do trimming in the actual predicate building now so there's no need to do it in both places.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11080/
Test PASSed.

Test build #105824 has started for PR 24068 at commit 3d1b28e.

Test build #105823 has finished for PR 24068 at commit b19792c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105823/
Test FAILed.

Test build #105824 has finished for PR 24068 at commit 3d1b28e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105824/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11119/
Test PASSed.

Test build #105863 has started for PR 24068 at commit aeaf8a1.

Test build #105863 has finished for PR 24068 at commit aeaf8a1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105863/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11160/
Test PASSed.

Test build #105907 has started for PR 24068 at commit c12ed11.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11208/
Test PASSed.

Test build #105956 has started for PR 24068 at commit 1735c0c.

Double checking something I mentioned in passing a while ago: after we agree this is the final version of the code, we should copy it to the hive/ subtree as well, right?

Test build #105960 has started for PR 24068 at commit 641b662.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11212/
Test PASSed.

Test build #105956 has finished for PR 24068 at commit 1735c0c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105956/
Test PASSed.

Test build #105960 has finished for PR 24068 at commit 641b662.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105960/
Test PASSed.

we should copy it to the hive/ subtree as well

Yes, I agree it's a little annoying...

Test build #106009 has started for PR 24068 at commit 524a1e1.

Updated. AFAICT the remaining actions are:

decide whether to remove the filterPushDownBenchmarkWithColumn benchmark. Let me know and I'll delete it if the consensus is that it's not a good fit for this suite.
update the hive/ subtree. Let me know when the rest looks good and I'll apply the changes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11259/
Test PASSed.

The code change LGTM, let's update the hive one as well, thanks for your great work!
For the benchmark, I'd like to remove it, but want to hear from @dongjoon-hyun as well.

Test build #106009 has finished for PR 24068 at commit 524a1e1.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106009/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11295/
Test PASSed.

Test build #106045 has started for PR 24068 at commit 524a1e1.

Test build #106045 has finished for PR 24068 at commit 524a1e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106045/
Test PASSed.

cc @gengliangwang @dongjoon-hyun

Added  a comment https://github.com/apache/spark/pull/24068/files#r289701087

Thanks for the hard work! I have thought of a simpler approach: we can reuse the method convertibleFilters in createBuilder. I will try it with the benchmark in this PR and get back here.

@IvanVergiliev I agree we should automate running these micro benchmark tests to ensure we do not have the regressions. However, Apache Spark does not do it now. In my company, we have a different framework that run the microbenchmark suites for Spark to ensure we do not have perf regressions.
The micro benchmark suites this PR changed are used to compare the perf difference among the similar but related cases. For example, turning on/off a specific conf. You can create a new perf suite for showing the cost per filter and put them in a single test case? We can check Relative to know whether it has a regression?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11489/
Test PASSed.

Test build #106240 has started for PR 24068 at commit 27ed4f9.

@cloud-fan I applied the changes to the hive/ subtree as well. It seems like it wasn't updated to reflect some of the recent changes to the other files (for example, it doesn't seem to have the quoted attribute name support) - I only applied the minimum amount of changed required to get the same algorithmic behaviour. I haven't applied the quoted attribute name support and other differences.
@gatorsmile I see what you're saying. I got rid of the benchmark that only does the filter conversion step since that indeed seemed a bit out of place with the rest. The remaining new benchmark is Pushdown benchmark with unbalanced Column which tests the full filter-with-pushdown sequence that the rest of the benchmarks cover. This remaining benchmark is almost the same as Pushdown benchmark with many filters which was added in https://github.com/apache/spark/pull/22313 with the introduction of the buildTree function. I think it makes sense to have these two benchmarks next to each other. Let me know if you'd like me to move both of them to a new benchmark file. (I have a small concern that if we move them to a separate file, it'll be easier to forget to run them when making changes - because, as you say, the process is not fully automated at the moment. Thus, I slightly prefer keeping them where they are, but I'm open to moving them too.)

@gatorsmile also added reproduction steps (modelled after https://github.com/apache/spark/pull/22313 ) to the commit description. 30 filters go from 7 minutes to 300ms 🙂

Test build #106240 has finished for PR 24068 at commit 27ed4f9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106240/
Test PASSed.

@IvanVergiliev Thank you for your contribution! The PR description looks much better!
@IvanVergiliev and @gengliangwang I have not read the implementation details yet. Could you show us the perf difference between two approaches?

@gatorsmile I take a quick check, the perf is quite close. The key difference between the two approaches is how the code is implemented:  abstraction and readability.

@gatorsmile yep, the performance is pretty much the same because the two implementations are exactly the same at a high level. The two main high-level steps are:

Trim the Filter tree so that unconvertible filters are removed and we don't need to check before building ORC filters in the next step.
Convert the trimmed filter, knowing that it's fully convertible at this point.

The main differences are in code structure. At a high level, the two main differences are:

Whether the trimming and conversion logic are in the same function, or in two separate functions. I actually had this implemented with two separate functions a few commits back - you can see https://github.com/IvanVergiliev/spark/pull/2/files for when I made the change to merge them in the same function. Notice that it's in a separate PR because I realize it's a controversial change. However, as mentioned back then, I think keeping the filtering and building logic in the same method has the following benefits:

• All the logic for a given predicate is grouped logically in the same place. You don't have to scroll across the whole file to see what the filter action for an And is while you're looking at the build action.
• You can't really add a new predicate to the set of filtered predicates without also defining a Build action for it - this fails the exhaustiveness check on ActionType.

@cloud-fan agreed that this does sound like an improvement in https://github.com/apache/spark/pull/24068#issuecomment-494388288 , so I merged the change into the main PR. I still think this has code design benefits over the separate functions, but I'm open to reverting to the previous version if there's consensus that it's better.

Whether we call buildSearchArgument while trimming the tree. While this provides for a shorter implementation, I think it makes the contract of buildSearchArgument very weak. If the trimming and building are independent from each other, we have a clear, well-defined contract for buildSearchArgument that's basically this

We always pass a fully convertible Filter, so buildSearchArgument can just depend on that and return a Builder.

Instead, if we buildSearchArgument in order to trim the filters sometimes as well, we get a way murkier, unclear contract that goes something like:

We mostly pass a convertible Filter, but sometimes we actually pass whatever we get, and depend on buildSearchArgument to figure out if it's convertible or not.

This leads to all kinds of questions:

if we always trim the Filter first, why do we need to return Option when building?

if I want to use the buildSearchArgument method and I see that it returns an Option, I might just as well assume that it will handle non-convertible filters just fine. However, this is only partially true - it can handle some of them fine, and can break for others.

There's no clear single responsibility of the buildSearchArgument method. The name implies that the only responsibility of the method is to build things, but in fact it's also sometimes used for trimming.
So while this saves some code, I think it makes the overall design and separation of concerns much less clear.

tl;dr I've gone through similar passes as the one proposed by @gengliangwang, but I've had specific design reasons to move away from them.
I'm happy to go back to a version where the filtering and building are written in separate methods if people like this better. I'm more sceptical about the reuse of buildSearchArgument for the reasons I explained above, but I can also make this happen if people think it's a better approach.
This got pretty long, but it also summarizes ~50 commits of changes and the discussion surrounding them. Let me know if anything is unclear or doesn't make sense.

@IvanVergiliev I'd suggest we revert all the benchmark changes, and we write a simple microbenmark to test OrcFilters, and post the benchmark code and result in PR description.
Currently we do not run benchmarks automatically for Spark, so perf regressions rely on user reports.

+1 for @cloud-fan 's suggestion. Also, I saw that @gatorsmile also gave the same advice before.

Thanks for the feedback! Just clarifying before I make any changes.

@cloud-fan sounds good, I can get rid of the benchmark changes.

and we write a simple microbenmark to test OrcFilters
Would that be different from the benchmark I added to the PR description per @gatorsmile's suggestion? The one in the PR does have disk access, but I think it's pretty easy to understand, and it does an excellent job at showing that the exponential behaviour goes away. Let me know if you have something else in mind.

Could you make a separate benchmark for both ORC v1 OrcFilters.createFilter and ORC v2 OrcFilters.createFilter.

@dongjoon-hyun it seems this suggestion would no longer be relevant if we get rid of the benchmark changes and only keep the benchmark results in the PR? Let me know if I got this wrong, and what you had in mind if you'd still like me to do this.

@IvanVergiliev I'd suggest something more low-level, e.g.
val filters = ...
val start = System.currentTimeMillis()
OrcFilter.createFilter...
print("time: " + (start - System.currentTimeMillis()))

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11855/
Test PASSed.

Test build #106618 has started for PR 24068 at commit 24dcc24.

@cloud-fan I reverted the benchmark changes, and updated the benchmark in the PR description so it only runs the filter conversion. It's even nicer to look at now, with the old version very visibly increasing 2x on each iteration, and the new one taking roughly 0 ms for each size.
I don't remember if we came to a conclusion on whether I should bring this PR back into a state where it has separate filter and build methods so the followup changes are easier, or if we should merge it as is and do all changes in followups. Let me know what you'd like me to do.
There's one minor caveat with benchmark code. The OrcFilters class is marked private[sql], so I needed to do some Scala gymnastics to enable the benchmark to call it directly. This is the full version of the code:
// Paste this into spark-shell using the `-raw` flag so it gets interpreted as a Scala file and so that we can trick spark-shell into thinking our class is actually in the `sql` package and can thus access `OrcFilters`.
:paste -raw

package org.apache.spark.sql

import org.apache.spark.sql.execution.datasources.orc._
import org.apache.spark.sql.types._
import org.apache.spark.sql.sources._
object OrcFiltersTest {
  def foo(): Unit = {
    val schema = StructType.fromDDL("col INT")
    (20 to 30).foreach { width =>
      val whereFilter = (1 to width).map(i => EqualTo("col", i)).reduceLeft(Or)
      val start = System.currentTimeMillis()
      OrcFilters.createFilter(schema, Seq(whereFilter))
      println(s"With $width filters, conversion takes ${System.currentTimeMillis() - start} ms")
    }
  }
}

Test build #106618 has finished for PR 24068 at commit 24dcc24.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106618/
Test PASSed.

@IvanVergiliev thanks for your great work! merging to master

@gengliangwang feel free to open a PR to simplify the code as you proposed.


[Debugger] Unable to run modules with following launch configurations

Description:
Unable to run modules with following launch configurations
Steps to reproduce:
 {
            "type": "ballerina",
            "request": "launch",
            "name": "GRPC Service",
            "script": "${workspaceRoot}/grpc_service",
            "networkLogs": true
        },
        {
            "type": "ballerina",
            "request": "launch",
            "name": "GRPC Client",
            "script": "${workspaceRoot}/grpc_client",
            "networkLogs": true
        },

Suggested Labels (optional):
Component/Debugger,
Component/VScode Plugin
Sample
https://github.com/ballerina-guides/backend-for-frontend/blob/master/README.md/

Improvements added with SL Alpha3.  Please reopen if you still get the error.


Updated snap and types for react

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@8756be9). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             master    #78   +/-   ##
=======================================
  Coverage          ?     0%
=======================================
  Files             ?     32
  Lines             ?   1544
  Branches          ?    232
=======================================
  Hits              ?      0
  Misses            ?   1544
  Partials          ?      0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8756be9...86df06e. Read the comment docs.


Confusion about the intergration of pytorch.

Hi there,
I read your PyTorch patch, and was confused about the modification, your modification was based on Gloo backend code, which would synchronize the stream every time the procedure call the AllReduce function. Isn't it inefficient? It seems that NCCL backend would put all the AllReduce operations into an async work queue which does not require synchronizing the stream before every AllReduce call.
Best regards

That is a very good observation.
Currently, SwitchML does not deal with GPU buffers directly and requires that they be copied to host memory similar to how GLOO operates. That is why we use the same logic as GLOO where we let PyTorch copy to host side memory, perform a synchronous allreduce, then copy memory back.
You are correct in that we can increase the efficiency by letting SwitchML handle the copies and letting PyTorch use the asynchronous all reduce operation. We are already working on a SwitchML PPP and a new PyTorch Patch that does exactly what you are describing. We're still testing and ironing out some kinks but hopefully we can push it soon.

Thank you for replying. Anyway, this work is brilliant.

Thanks. Feel free to reach out if you have any other problems.


Follow symlinks

Hi,
Netlify-cli follows symlinks, netlifyctl should probably have the same behavior.

can you expand on this issue? is it deploying a site?

I just remembered that Go's Walk logic doesn't follow symlinks to avoid infinite loops. I'll try to find a different solution, but I'm not sure if we'll fix this any time soon.

I was a little confused by the OP, since in testing it appears that netlify-cli does not follow symlinks. Is that the case?  If so it should be documented. It would be nice if it didn't have this limitation, I rely on symlinks for a particular site, and won't be able to deploy to Netlify without symlink folowing.


input chinese，error

when I input  Chinese,it will show the behind error:
vue2-autocomplete.js:14 Uncaught TypeError: Cannot read property 'length' of null
at a.getData (vue2-autocomplete.js:14)
at a.n [as getData] (vue.min.js:6)
at a.input (vue2-autocomplete.js:14)
at a.n [as input] (vue.min.js:6)
at s.on.input (vue2-autocomplete.js:14)
at HTMLInputElement.t (vue.min.js:6)

Hi @herrdu,
Thanks for giving me an issue. But, I need some reproduce.
BTW. you can check our new version of autocomplete.

@herrdu Check whether this work for you.
https://github.com/BosNaufal/vue2-autocomplete/pull/64


False Positives in "Detect Outbound LDAP Traffic" detection

Running the Detect Outbound LDAP Traffic detection yields multiple false positives.
We might want to add logic to the current rule to check the source of the traffic is from an RFC1918 network address?
For example, in my environment, I'll get a lot of events returned from people scanning my border firewall (24.230.x.x in the picture below) for an open port on TCP/389 or TCP/636:

@ccl0utier -
This detection is a hunting analytic. It is not intended to create notables. It is the intention that Hunting analytics are used by analysts as jumping-off-points looking for behavior that is not normal for their environment, which is why the majority of them are broad and do not filter out as many things as possible that could be false positives in any one environment.
Sensor placement is incredibly important for these analytics and cannot be prescribed in SPL. Dependent on sensor placement, I would suggest utilizing All_Traffic.direction to match the intended purpose of the search, or putting values in the detect_outbound_ldap_traffic_filter filter macro to remove results that are known-benign.

@ljstella Thanks, that makes perfect sense.  I missed the fact that it was a hunting analytic.


fix: ensure prepare_command only appends one -c arguemnt to shell

Fish errors out on multiple -c invocations where zsh, bash don't.
I originally had a check but instead dropped it in app.rs. I very much don't know rust so feel free to adapt as needed :)
Shout out to @onedr0p for figuring out how this was broke and finding the redundant appends <3

Fixes #155

@anthr76 Thank you for the PR. LGTM!
@onedr0p Thank you for the review & investigation!


A naive GuidedConv implementation

I guess the source code for this repo would never be released, and since people are confused by "Guided Conv Module" in the paper, I'm going to share my naive PyTorch implementation based on CSPN code.
Note: This implementation was based on my own understanding of the paper, I'm not sure if it's correct. Besides, I got worse result with this module compared to ordinary concat/add fusion method.
Let me know if you have any questions or suggestions.
class _GuidedConv(nn.Module):
    def __init__(self):
        super(_GuidedConv, self).__init__()

        self.pad_left_top = nn.ZeroPad2d((1, 0, 1, 0))
        self.pad_center_top = nn.ZeroPad2d((0, 0, 1, 0))
        self.pad_right_top = nn.ZeroPad2d((0, 1, 1, 0))
        self.pad_left_middle = nn.ZeroPad2d((1, 0, 0, 0))
        self.pad_right_middel = nn.ZeroPad2d((0, 1, 0, 0))
        self.pad_left_bottom = nn.ZeroPad2d((1, 0, 0, 1))
        self.pad_center_bottom = nn.ZeroPad2d((0, 0, 0, 1))
        self.pad_right_bottom = nn.ZeroPad2d((0, 1, 0, 1))

    def forward(self, x, cw: list, cc: list):
        """
        `x`: input feature maps with size `[B, C_in, H, W]`
        `cw`: `C_in` channel-wise kernels, each with size [B, 3*3, H, W]
        `cc`: `C_out` cross-channel 1*1 kernels, each with size [B, C_in]
        """

        # stage-1: weight x with kernels in `cw`
        tmp = []
        for i in range(len(cw)):
            feat = self._compose_feat(x[:, i, :, :].unsqueeze_(1))
            feat *= cw[i]
            tmp.append(torch.sum(feat, dim=1, keepdim=True))
        tmp = torch.cat(tmp, dim=1)  # [B, C_in, H, W]

        # stage-2: weight tmp with kernels in `cc`
        out = []
        for i in range(len(cc)):
            weight = cc[i].unsqueeze_(-1).unsqueeze_(-1)
            out.append(torch.sum(tmp * weight, dim=1, keepdim=True))

        return torch.cat(out, dim=1)  # [B, C_out, H, W]

    def _compose_feat(self, feat: torch.FloatTensor):
        [H, W] = feat.shape[2:]
        output = [feat]

        # left-top
        output.append(self.pad_left_top(feat)[:, :, :H, :W])
        # center-top
        output.append(self.pad_center_top(feat)[:, :, :H, :])
        # right-top
        output.append(self.pad_right_top(feat)[:, :, :H, 1:])
        # left-middle
        output.append(self.pad_left_middle(feat)[:, :, :, :W])
        # right-middle
        output.append(self.pad_right_middel(feat)[:, :, :, 1:])
        # left-bottom
        output.append(self.pad_left_bottom(feat)[:, :, 1:, :W])
        # center-bottom
        output.append(self.pad_center_bottom(feat)[:, :, 1:, :])
        # right-bottom
        output.append(self.pad_right_bottom(feat)[:, :, 1:, 1:])
        # concat
        output = torch.cat(output, dim=1)  # [B, 3*3, H, W]

        return output

Worse results as well. But CSPN works as a refinement module.

I think it is really hard to re-implement guided conv with info from paper only.
Too many details are missing.
For example,

We don't know the params of this 'standard convolution layer', and whether it is followed by BN, activation, etc.

Worse results as well. But CSPN works as a refinement module.

is there an open source version of CSPN?

Worse results as well. But CSPN works as a refinement module.
is there an open source version of CSPN?

https://github.com/XinJCheng/CSPN


Make wrapper scripts also executable by group, other.

Fixes #20.

The build failures are unrelated to the changes. I'll fix them later.
Thanks @tesujimath !

You're welcome @gqmelo.  Thanks for merging.


photoflow: fails to build:

Encountered photoflow build failure when tested reverse dependencies of unrelated change. Hydra also sees the failure since May: https://hydra.nixos.org/build/154595755
/build/source/src/operations/denoise.cc:30:10: fatal error: vips/cimg_funcs.h: No such file or directory
   30 | #include <vips/cimg_funcs.h>
      |          ^~~~~~~~~~~~~~~~~~~

cc @MtP76
Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute: photoflow

#139970


Query API - 3 - added asFlow variants to ordinary and scalar queries

Blocked by https://github.com/realm/realm-kotlin/pull/608

Added asFlow for ordinary queries - thaw creates ElementResults and asFlow uses them via the notifier.
Added asFlow for scalar queries - thaw creates ScalarResults and asFlow uses them via the notifier. The aggregator values are computed using the C-API functions exposed through realm_results_t - it could be improved if realm_query_t exposed these through the C-API, but since we still need results for a subscription, I think it's fair to just leave it like this.
Fixed bug in swig generated code when there is an exception being thrown but not returned the correct value which caused the exception to be swallowed.

Aggregator flow tests for the empty set

It might not be the best in terms of visibility but they are already there, they can be found in asFlowAggregatorAssertions. I have added some documentation to the method for clarity.

Ordinary flow test for empty set

Same as before, they were there, but now I've split the test in two for better visibility.

And what about flow-termination? I might have asked this before, but I fail to recall it and why it maybe wouldn't be required.

I have added those in the latest commit 👍

And what about flow-termination? I might have asked this before, but I fail to recall it and why it maybe wouldn't be required.

Now I recall. I guess I confused it with object notifications where we actually complete the flow if the observer object is delete. But here the query is not tied to an object, so should be good.


Missing css document

The "Getting started" section refers to a css file "videojs.chapter-thumbnails.min.css".
This file has not been included in the package.
A "dist" folder with this css file and the js files would have been a nice.

@Grotgod
npm i videojs-chapter-thumbnails

ls -la node_modules/videojs-chapter-thumbnails/dist

It is available through npm, but I do not commit the dist directory.

Closing issue. Can reopen if there are still questions.


gone margin end not set when using clone() method

I have set app:layout_goneMarginEnd on my view and when I clone the ConstraintSet gone margins are not cloned. I've found my view by its id in the ConstraintSet and all gone margins are set to -1.
    <TextView
        android:id="@+id/title"
        android:layout_width="0dp"
        android:layout_height="wrap_content"
        app:layout_constraintEnd_toStartOf="@+id/icon"
        app:layout_constraintStart_toEndOf="@+id/icon"
        app:layout_constraintTop_toTopOf="@+id/background"
        app:layout_goneMarginEnd="@dimen/gone_margin"/>

Constraint constraintSet = new ConstraintSet();
constraintSet.clone(this);

When I set it manually it works fine
constraintSet.setGoneMargin(title.getId(), ConstraintSet.END, goneMargin);

Got the same issue


Fix #546

The problem is that the lib could also be in /usr/lib64 (that's where it is on my VM).
I believe just using lqscintilla2 if the two previous exists() didn't succeed would be the best option here? Since even the FHS can't prevent all problems.

While I agree it's good to have a catch all, you would still get segfaults if you use lqscintilla2 because it's the wrong library on linux. in the case of ubuntu/fedora it's called lqt5scintilla2 on arch linux it's lqscintilla2-qt5. I'm not sure about other distros.

Then maybe disable Qscintilla if none of the 3 (4 including lib64) files are found? (by removing the define and printing some error message)
Since it won't compile otherwise, and that lib isn't really critical.
Thanks for all the PR btw, it really helps :smile:

I think something like this works fairly well? More of what you were thinking of? And no problem. I like the program and bugs aren't fun to fix by yourself!

Yeah I think like this it will cover all use cases while still preventing using an invalid lib and segfaulting!


merge Q+A's into statement

Questions and answers need(?) to be merged into statements in order to classify them with a sentiment analysis tool.

A simple first thing to do before training/finding a separate RNN/CNN for this is to just merge the questions and answers by concatenating them, e.g.
(Q: "Do you like apples?") + (A: "Yes.") ---> (S: "Do you like apples? Yes.")
The question is whether the sentiment of liking apples registers as positive without the full merging of the answer,
(S: "I like apples.") or (S: "Yes, I like apples)" or (S: "They enjoy eating apples."), etc.

It's becoming clear that there are two routes here. You can either:

Use the response words ("Yes", "No", "Strongly Agree", "Strongly Disagree", etc.) and combine them with the question to get a statement, and perform a single sentiment analysis to get a number in [-1,+1].
Determine the orientation of the question (Does an affirmative answer indicate a conservative or liberal lean? or contra) as a binary variable {-1,+1}, and map the responses separately to number in [-1,+1], and multiply them together.

The second option seems way cleaner, for two reasons. The sentiment analyzer is going to be noisy, so it's a little better to not trust it fully to return the actual sentiment. Rather, we trust the response data for the actual valuation. However, though it is noisy, it should still reliably give an indication, a hint, of the sentiment direction. For instance, if Q+"YES" yields -.052 and Q+"NO" yields +.031, then I can be pretty sure that an affirmative answer to this question is a conservative leaning answer.
If the responses are {"strongly agree", "agree", "neutral","disagree","strongly disagree"} which easily map to {-1, -1/2, 0, +1/2, +1}, I have to decide, do I try to merge the word answers into "I strongly agree that ... blah blah", and take the output of the sentiment analyzer, a number in [-1,+1]?
If I determine that an affirmative answer is a conservative answer (orientation = -1), do I then assign orientmag=(-1)(+1)=-1 to "strongly agree". Do I include the fact that Q+"YES" was only -.052, and not fully -1? That is, even though an affirmative answer to the question indicates a conservative lean, the lean is not that strong. In other words, the question is nearly perpendicular to the axis [CONS,LIB].

For now, it is imperative to just put basic structure in place with placeholder functions to perform all of these, and not worry at all about the quality of the plot yet. That can and should be tuned later.


Update grammar

Adding two constructs to the grammar.

DROP FUNCTION
FILLFACTOR as an index option

Most files are autogenerated. It makes most sense to review the changes only in tsql.g4

@EnchoMishinevQC I'd suggest that next time you split this into two commits one having the change in the grammar and the other adding the generated files. Then it's faster and easier to just see the relevant commit with the changes of the grammar ;)

@EnchoMishinevQC I'd suggest that next time you split this into two commits one having the change in the grammar and the other adding the generated files. Then it's faster and easier to just see the relevant commit with the changes of the grammar ;)

Good thinking. Did it for this PR as well.

FILLFACTOR as an index option

I don't see with what change this has been addressed - other than that lgtm!

Index options allow an arbitrary simple_id construct. The FILLFACTOR wasn't part of it - it's added in this PR.


How does this compare to the other rollup-plugin-swc

Hey, I see there is another plugin for swc. Do you know how they compare? https://github.com/mentaljam/rollup-plugin-swc

So here is why I made my own rollup swc plugin:

It can reads tsconfig.json and jsconfig.json file automatically, so no extra configuration
Unulike https://github.com/mentaljam/rollup-plugin-swc (which run swc minify on every transform) my plugin only run swc minify on renderChunk, which minify all chunks at once.

A comparison has been added in 29c7dface24ac4781af6340ebca80aabfd5317fa.


Throwing Errors adding line series elements to canvas

Thanks to @beto-rodriguez my code hasn't really changed since the last issue I came across. Sadly I've hit a new error where the UI crashes when a view with a Live-Chart on it is navigated to twice.
To reproduce this I do the following:

Load WPF application -> Home Page is displayed (Live Chart)
Navigate to other view (no live chart)
Navigate back to home page
Navigate to other view (no live chart)
Navigate back to home page -> Crash

This is the stack trace I'm getting:
at System.Windows.FrameworkElement.AddLogicalChild(Object child)
at System.Windows.FrameworkElement.ChangeLogicalParent(DependencyObject newParent)
at System.Windows.Controls.UIElementCollection.AddInternal(UIElement element)
at System.Windows.Controls.UIElementCollection.Add(UIElement element)
at LiveCharts.Charts.Chart.PrepareCanvas(Boolean animate) in C:\Dev\Project\Trunk\Sources\Charts\Charts\Chart.cs:line 739
at LiveCharts.Charts.Chart.ClearAndPlot(Boolean animate) in C:\Dev\Project\Trunk\Sources\Charts\Charts\Chart.cs:line 249
at LiveCharts.Charts.Chart.InitializeSeries(Chart chart) in C:\Dev\Project\Trunk\Sources\Charts\Charts\Chart.cs:line 830
at LiveCharts.Charts.Chart.SeriesChangedCallback(DependencyObject o, DependencyPropertyChangedEventArgs eventArgs) in C:\Dev\Compliance Check\Trunk\Sources\Charts\Charts\Chart.cs:line 804
at System.Windows.DependencyObject.OnPropertyChanged(DependencyPropertyChangedEventArgs e)
at System.Windows.FrameworkElement.OnPropertyChanged(DependencyPropertyChangedEventArgs e)
at System.Windows.DependencyObject.NotifyPropertyChange(DependencyPropertyChangedEventArgs args)
at System.Windows.DependencyObject.UpdateEffectiveValue(EntryIndex entryIndex, DependencyProperty dp, PropertyMetadata metadata, EffectiveValueEntry oldEntry, EffectiveValueEntry& newEntry, Boolean coerceWithDeferredReference, Boolean coerceWithCurrentValue, OperationType operationType)
at System.Windows.FrameworkTemplate.ReceivePropertySet(Object targetObject, XamlMember member, Object value, DependencyObject templatedParent)
at System.Windows.FrameworkTemplate.<>c__DisplayClass0.<LoadOptimizedTemplateContent>b__6(Object sender, XamlSetValueEventArgs setArgs)
at System.Xaml.XamlObjectWriter.OnSetValue(Object eventSender, XamlMember member, Object value)
at System.Xaml.XamlObjectWriter.Logic_ApplyPropertyValue(ObjectWriterContext ctx, XamlMember prop, Object value, Boolean onParent)
at System.Xaml.XamlObjectWriter.Logic_DoAssignmentToParentProperty(ObjectWriterContext ctx)
at System.Xaml.XamlObjectWriter.Logic_AssignProvidedValue(ObjectWriterContext ctx)
at System.Xaml.XamlObjectWriter.WriteEndObject()
at System.Xaml.XamlWriter.WriteNode(XamlReader reader)
at System.Windows.FrameworkTemplate.LoadTemplateXaml(XamlReader templateReader, XamlObjectWriter currentWriter)

I've found that if I comment out lines 737 to 743 in Chart.cs (version 0.5.10) the error doesn't show itself however I suspect this will inhibit me from displaying additional Series or changes to the collections.

Thank you very much for this detailed report, 0.6 will be released soon, it is a full review of live charts I have made important changes and I am really happy with the result, it should be much more stable and scalable, I will fix this as soon as 0.6 is ready.  I am sure you will love 0.6 much easier and powerful sintax

I will try to force this error, just one question when you say "load view" you mean a new window?, or just a partial change in current window? Maybe using a ContentControl

partial change in current window, we have a shell window where we use a content control bound to a viewmodel to switch the views with Caliburn.Micro.

0.6 is a full review of the code, it is now much cleaner I have not been able to force this bug, are you still getting it with last version, 0.6.1 is even cleaner, I would appreciate if you test it with it (not nugget yet)

I've updated the project to use 0.6.1 and the bug still showed itself.
   at System.Windows.FrameworkElement.ChangeLogicalParent(DependencyObject newParent)
   at System.Windows.FrameworkElement.AddLogicalChild(Object child)
   at System.Windows.Controls.UIElementCollection.AddInternal(UIElement element)
   at System.Windows.Controls.UIElementCollection.Add(UIElement element)
   at lvc.Charts.Chart.PrepareCanvas(Boolean animate) in C:\Dev\Project\Trunk\Sources\Charts\CoreComponents\Chart.cs:line 735
   at lvc.Charts.Chart.ClearAndPlot(Boolean animate) in C:\Dev\Project\Trunk\Sources\Charts\CoreComponents\Chart.cs:line 265
   at lvc.Charts.Chart.InitializeSeries(Chart chart) in C:\Dev\Project\Trunk\Sources\Charts\CoreComponents\Chart.cs:line 846
   at lvc.Charts.Chart.SeriesChangedCallback(DependencyObject o, DependencyPropertyChangedEventArgs eventArgs) in C:\Dev\Project\Trunk\Sources\Charts\CoreComponents\Chart.cs:line 800
   at System.Windows.DependencyObject.OnPropertyChanged(DependencyPropertyChangedEventArgs e)
   at System.Windows.FrameworkElement.OnPropertyChanged(DependencyPropertyChangedEventArgs e)
   at System.Windows.DependencyObject.NotifyPropertyChange(DependencyPropertyChangedEventArgs args)
   at System.Windows.DependencyObject.UpdateEffectiveValue(EntryIndex entryIndex, DependencyProperty dp, PropertyMetadata metadata, EffectiveValueEntry oldEntry, EffectiveValueEntry& newEntry, Boolean coerceWithDeferredReference, Boolean coerceWithCurrentValue, OperationType operationType)
   at System.Windows.FrameworkTemplate.ReceivePropertySet(Object targetObject, XamlMember member, Object value, DependencyObject templatedParent)
   at System.Windows.FrameworkTemplate.<>c__DisplayClass0.<LoadOptimizedTemplateContent>b__6(Object sender, XamlSetValueEventArgs setArgs)
   at System.Xaml.XamlObjectWriter.OnSetValue(Object eventSender, XamlMember member, Object value)
   at System.Xaml.XamlObjectWriter.Logic_ApplyPropertyValue(ObjectWriterContext ctx, XamlMember prop, Object value, Boolean onParent)
   at System.Xaml.XamlObjectWriter.Logic_DoAssignmentToParentProperty(ObjectWriterContext ctx)
   at System.Xaml.XamlObjectWriter.Logic_AssignProvidedValue(ObjectWriterContext ctx)
   at System.Xaml.XamlObjectWriter.WriteEndObject()
   at System.Xaml.XamlWriter.WriteNode(XamlReader reader)
   at System.Windows.FrameworkTemplate.LoadTemplateXaml(XamlReader templateReader, XamlObjectWriter currentWriter)

Then we modified the containing view model to inherit from ViewAware rather than just PropertyChangedBase and that fixed our issue.

Thanks for reporting, I will work on it

I have not test this but I think I founf the solution

specially this lines should solve the problem
https://github.com/beto-rodriguez/Live-Charts/commit/cb819deded4da08d3999280780fa667ad2fb903b#diff-7cabf91fc05cf7e480b95dffd3ac5605R883
https://github.com/beto-rodriguez/Live-Charts/commit/cb819deded4da08d3999280780fa667ad2fb903b#diff-c148adab5d542cc147f6f9a3ccc9e63cR242

Hey there, I'm getting nearly same error by navigating second time to my view. Here is my exception:
System.InvalidOperationException: Bei dem angegebenen Element handelt es sich bereits um das logische untergeordnete Element eines anderen Elements. Führen Sie zuerst eine Trennung durch.
bei System.Windows.FrameworkElement.ChangeLogicalParent(DependencyObject newParent)
bei System.Windows.FrameworkElement.AddLogicalChild(Object child)
bei System.Windows.Controls.UIElementCollection.AddInternal(UIElement element)
bei LiveCharts.CoreComponents.Chart.PrepareCanvas(Boolean animate)
bei LiveCharts.CoreComponents.Chart.InitializeSeries(Chart chart)
bei LiveCharts.CoreComponents.Chart.PrepareCanvas(Boolean animate)
bei System.Windows.Threading.DispatcherTimer.FireTick(Object unused)
bei System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
bei System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
Have you fixed this anyway? Can I help you investigating this error? Thx.

I  am practically sure th will dissapear in 0.6.4 I hope I can publish 0.6.4 today

That will be great. Thanks a lot!

I already tested this with new UnitTesting project. this is solved. here is a test mocking this case, https://github.com/beto-rodriguez/Live-Charts/blob/master/UnitTesting/General/SeriesSharedInCharts.cs
the problem was that old canvas was staying in memory and had your series as VisualChilds (I am not really sure how), but now as you can see in the test I force this case and the series should only stay in the second chart. I have not tested this in UI but I am sure it will only plot in second chart wich is what normally wpf does in this cases.
Thanks again for reporting, let me know if you still have any troubles with this issue.
P.S. this will be published with 0.6.4

Thank you! I've tested it in my application and it works correctly!


merge package from arlac77/npm-package-template

.travis.yml

chore(travis): remove node versions 12.2.0
chore(travis): add node versions -10,-11,-12,12.3.1

Coverage increased (+0.01%) to 37.28% when pulling 13415e3c2f720fc15025608dacad78bf5be37c5b on npm-template-sync-1 into 989ed57180fa35ea4373bf8cb72d77332c57fdd6 on master.