SAS

Viya4 Deployment Version Details
2023.10 LTS on on prem k8 cluster
Ansible Variable File Details
SAS API Access
V4_CFG_SAS_API_KEY: ""
V4_CFG_SAS_API_SECRET: ""
V4_CFG_ORDER_NUMBER: ""
V4_CFG_CADENCE_VERSION: "2023.10"
V4_CFG_CADENCE_NAME: "lts"
CR Access
V4_CFG_CR_USER:
V4_CFG_CR_PASSWORD:
V4_CFG_CR_URL: jfrog
Steps to Reproduce
deploy infra thorugh k8 iac on vpshere
deploy viya through sas deployment project.
proxy exists
Expected Behavior
sas pods should be starting
Actual Behavior
we see that no pods are created in sas namespace and when i run kubectl describe sasdeployment -n viyaontw i see below error
Messages:
Error loading entitlements file: "https://ses.sas.download/ses/entitlements.json"
Failed to get 'https://ses.sas.download/ses/entitlements.json'
Get "https://ses.sas.download/ses/entitlements.json": dial tcp <IP_ADDRESS>:443: connect: connection timed
Additional Context
No response
References
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

does anyone has similar experience on why this could happen. k8 clusters is installed through iac and we are deploying sas through this git hub project. and we have sas images in private jfrog repo and we are using certmanager. we are really struck at this issue.
Unfortunately the ingress as well does not pick an external ip and pending state.
Any advise would help or someone else experience that used similar deployment

Hi @SASCloudLearner ,
I have no experience with this specific configuration but it seems like SAS is trying to reach out to https://ses.sas.download/ses/entitlements.json and is blocked (by the proxy?). Not sure why it is reaching out to SAS CR is you are using JFrog. On the other hand, the parameters you posted here seem to be lacking. Not sure if this is just for masking sensitive data or this is really how you entered the values in ansible-vars.yaml. For example:
V4_CFG_SAS_API_KEY: ""
V4_CFG_SAS_API_SECRET: ""
V4_CFG_ORDER_NUMBER: ""
V4_CFG_CR_USER:
V4_CFG_CR_PASSWORD:

In addition you wrote V4_CFG_CR_URL: jfrog is "jfrog" the true URL for accessing the container registry in JFrog? Does it not require authentication? Perhaps you should attach the full ansible-vars.yaml you used.

Hello Eyal,
Thank you for your response.
Yes, i removed the values as its sensitive data. And we have a proxy and on Jump server we set the http_proxy/ https_proxy as well though we still get ssl cert error when we do curl and also though we install the sas ca cert in there.
I believe viayadeployment is not expected to reach out to sas download site when we have internal jfrog container repo.
Here are the details from ansible-vars file and we tried various options but with same results. Also,  any idea about why ingress could not be able to pickup external ip. we are using kubevip.
environment:
https_proxy: http://10.0.x.x:8080
http_proxy: http://10.0.x.x:8080
no_proxy: "{{no_proxy}}"
Cluster
PROVIDER: custom
CLUSTER_NAME: k8cluster
NAMESPACE: viyans
#Requirememts, extra added
V4_CFG_CADENCE_VERSION: "2023.10"
V4_CFG_CADENCE_NAME: "lts"
V4_CFG_DEPLOYMENT_ASSETS: "xxxxxxx4_deploymentAssets_2024-04-26T080944.tgz"
V4_CFG_CERTS: "/sas-deployment/deploy/sandbox/assets/SASViyaV4_xxxx_certs.zip"
V4_CFG_ORDER_NUMBER: "xxxx"
MISC
DEPLOY: true # Set to false to stop at generating the manifest
LOADBALANCER_SOURCE_RANGES: ['xxx.x/32','xxxx/32']
Jump
JUMP_SVR_HOST: 10.0.x.x
JUMP_SVR_USER: myuserid
RWX Filestore
V4_CFG_RWX_FILESTORE_ENDPOINT: 10.0.x.x
V4_CFG_RWX_FILESTORE_PATH: /export
Storage
V4_CFG_MANAGE_STORAGE: true
SAS API Access
V4_CFG_SAS_API_KEY: "xxxxxxxxxx"
V4_CFG_SAS_API_SECRET: "xxxxxxxxxxxxxxx"
CR Access
V4_CFG_CR_USER:docker-user
V4_CFG_CR_PASSWORD: xxxxxxxxx
V4_CFG_CR_URL: https://internalcontainerjfrogrepo
Ingress
V4_CFG_INGRESS_TYPE: "ingress"
V4_CFG_INGRESS_FQDN: "viyaserver.dqdn"
#V4_CFG_TLS_MODE: full-stack # [full-stack|front-door|ingress-only|disabled]
#V4_CFG_INGRESS_MODE: public
#INGRESS_NGINX_CONFIG:
controller:
service:
externalTrafficPolicy: Cluster
V4_CFG_POSTGRES_SERVERS:
default:
internal: false
admin: postgres
password: "mypasswd"
fqdn: exteralpostrges
server_port: 5432
ssl_enforcement_enabled: false
database: SharedServices
LDAP
V4_CFG_EMBEDDED_LDAP_ENABLE: true
Consul UI
V4_CFG_CONSUL_ENABLE_LOADBALANCER: false
SAS/CONNECT
V4_CFG_CONNECT_ENABLE_LOADBALANCER: false
TLS
V4_CFG_TLS_GENERATOR: "cert-manager"
V4_CFG_TLS_CERT: "/sas-deployment/deploy/sandbox/viya4-deployment/ingresscert.crt"
V4_CFG_TLS_KEY: "/sas-deployment/deploy/sandbox/viya4-deployment/ingresscert.key"
V4_CFG_TLS_TRUSTED_CA_CERTS: "/sas-deployment/deploy/sandbox/assets/D0_Root_CA_-_G01.pem"
#CAS MPP Settings
V4_CFG_CAS_WORKER_COUNT: 3
Monitoring and Logging
uncomment and update the below values when deploying the viya4-monitoring-kubernetes stack
V4M_BASE_DOMAIN: <base_domain>
Viya Start and Stop Schedule
uncomment and update the values below with CronJob schedule expressions if you would
like to start and stop your Viya Deployment on a schedule
V4_CFG_VIYA_START_SCHEDULE: "0 7 * * 1-5"
V4_CFG_VIYA_STOP_SCHEDULE: "0 19 * * 1-5"

Hi @SASCloudLearner ,
Here's what I think is happening. I am not 100% sure perhaps others can chime in.
Since you provided values for V4_CFG_DEPLOYMENT_ASSETS and V4_CFG_CERTS parameters so it means you downloaded them yourself before running this project but you did not provide value for V4_CFG_LICENSE parameter then the project tries to download the license file by itself (using V4_CFG_SAS_API_KEY and V4_CFG_SAS_API_SECRET that you provided) as explained here: https://github.com/sassoftware/viya4-deployment/blob/main/docs/CONFIG-VARS.md#sas-software-order and probably then the proxy error happens.
Can you try to manually download the license file as well and point to it using the V4_CFG_LICENSE parameter like you pointed to the assets file and the certs file? Note that if you run the project using Docker, you need to read this as well: https://github.com/sassoftware/viya4-deployment/blob/main/docs/user/DockerVolumeMounts.md

Hey @SASCloudLearner,
Just to verify, you stated your cluster was behind a proxy and it looks like connection is blocked to external hosts? You can verify if this is true by whether or not the baseline,install tasks ran successfully since those actions have tasks that install a some helm charts.
The deployment operator pod will download the entitlements from ses.sas.download during it's execution unless the "repositoryWarehouse" URL is specified in the sasdeployment.yaml file to get it from elsewhere.
https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=dplyml0phy0dkr&docsetTarget=p0uytz3rj1l4ysn1e8n73u95stqu.htm#p0x9m01w44xo9cn13xsxqttwmtj2
We have an open feature request, #372 to add support for this repositoryWarehouse option to better support theses dark site/air gapped type deployments
Also thanks @miaeyg for providing some additional areas to look into to debug this issue.

Hi @SASCloudLearner ,
Here's what I think is happening. I am not 100% sure perhaps others can chime in.
Since you provided values for V4_CFG_DEPLOYMENT_ASSETS and V4_CFG_CERTS parameters so it means you downloaded them yourself before running this project but you did not provide value for V4_CFG_LICENSE parameter then the project tries to download the license file by itself (using V4_CFG_SAS_API_KEY and V4_CFG_SAS_API_SECRET that you provided) as explained here: https://github.com/sassoftware/viya4-deployment/blob/main/docs/CONFIG-VARS.md#sas-software-order and probably then the proxy error happens.
Can you try to manually download the license file as well and point to it using the V4_CFG_LICENSE parameter like you pointed to the assets file and the certs file? Note that if you run the project using Docker, you need to read this as well: https://github.com/sassoftware/viya4-deployment/blob/main/docs/user/DockerVolumeMounts.md
Update 1: Examine the contents of this file in the deployment folder: "site-config/cr_access.json" does it point to JFrog correctly? You should have this file generated for you. Do you also have in "kustomization.yaml" a reference to a file named "sas-image-pull-secrets.yaml"?
Update 2: Verify you have in your "kustomization.yaml" reference to a file named "mirror.yaml" and that this file contains correct references to your JFrog CR. I assume you mirrored SAS LTS 2023.10 to JFrog using the SAS Mirror Manager utility, right?

Hi @SASCloudLearner ,
Here's what I think is happening. I am not 100% sure perhaps others can chime in.
Since you provided values for V4_CFG_DEPLOYMENT_ASSETS and V4_CFG_CERTS parameters so it means you downloaded them yourself before running this project but you did not provide value for V4_CFG_LICENSE parameter then the project tries to download the license file by itself (using V4_CFG_SAS_API_KEY and V4_CFG_SAS_API_SECRET that you provided) as explained here: https://github.com/sassoftware/viya4-deployment/blob/main/docs/CONFIG-VARS.md#sas-software-order and probably then the proxy error happens.
Can you try to manually download the license file as well and point to it using the V4_CFG_LICENSE parameter like you pointed to the assets file and the certs file? Note that if you run the project using Docker, you need to read this as well: https://github.com/sassoftware/viya4-deployment/blob/main/docs/user/DockerVolumeMounts.md
Update 1: Examine the contents of this file in the deployment folder: "site-config/cr_access.json" does it point to JFrog correctly? You should have this file generated for you. Do you also have in "kustomization.yaml" a reference to a file named "sas-image-pull-secrets.yaml"?
Update 2: Verify you have in your "kustomization.yaml" reference to a file named "mirror.yaml" and that this file contains correct references to your JFrog CR. I assume you mirrored SAS LTS 2023.10 to JFrog using the SAS Mirror Manager utility, right?

Hi @jarpat I believe you are correct, I forgot about this... I recall the workaround is to avoid using the deployment operator by setting:
V4_DEPLOYMENT_OPERATOR_ENABLED: false
I also used:
DEPLOY: false
And then I deployed manually using "kubectl". This was done with the advice of Josh Coburn!

Hi Eyal,
I already included the V4_CFG_LICENSE yesterday but still had the same issue. Also mirror.yaml/ sas-image-pull-secrets.yaml are included in kustomization.yaml. And the cr_access.json details looks right as well.
Last week, i did try to run the kubectl commands manually as the kustomization.yaml looked complete and the deployment only fails during re-conciling and saw that pods/services etc were created. we had some other issues with ingress (not able to pick external ip) and other things so could not validate that deployment successfully and also was not sure if that would be missing.
So from your experience, running the kubectl commands against the cluster-wide/cluter-local/namespace works fine? or Is there any additional things to note/ perform?
Thanks Again for your reponse.
Thanks,
Raghu

Hey @SASCloudLearner,
Just to verify, you stated your cluster was behind a proxy and it looks like connection is blocked to external hosts? You can verify if this is true by whether or not the baseline,install tasks ran successfully since those actions have tasks that install a some helm charts.
The deployment operator pod will download the entitlements from ses.sas.download during it's execution unless the "repositoryWarehouse" URL is specified in the sasdeployment.yaml file to get it from elsewhere. https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=dplyml0phy0dkr&docsetTarget=p0uytz3rj1l4ysn1e8n73u95stqu.htm#p0x9m01w44xo9cn13xsxqttwmtj2
We have an open feature request, #372 to add support for this repositoryWarehouse option to better support theses dark site/air gapped type deployments
Also thanks @miaeyg for providing some additional areas to look into to debug this issue.

Hi @jarpat,
Thats right. we have a proxy for cluster to be able to access the internet.
when we set the http_proxy, https_proxy, that seems to help while downloading the sas images from sas container register and push it to jfrog using mirrormgr. But we could not figure out how to include the proxy details in anisble-vars.yaml or other yaml files so deployment operator could take those values during recociling process.
And as the deployment.yaml is only created after we do the viya installation through ansible, it replaces i believe the file if we make changes and run the ansible-playbook again unless we use kubectl commands.
Thank you for reference to the open feature. I will keep an eye on that, incase there is any update as this option would be really helpful while deploying at most of the customer location where access to internet is limited.
Thanks,
Raghu

Hi @SASCloudLearner
Have you tried to add/modify your ansible-vars.yaml to include these two lines?
V4_DEPLOYMENT_OPERATOR_ENABLED: false
DEPLOY: false
If not, please try and re-run the "viya,install" again pointing to a new folder so it will create new kustomization.yaml and then try to install this kustomization.yaml manually with kubectl commands. I suggest you also delete all created objects in the K8S cluster and also the SAS Viya namespace so you start fresh. This worked for me in an air gapped deploy in AWS with ECR used as the Container Registry (equivalent of your JFrog).
I hope this helps.

Hi @miaeyg
I tried this way and seems like its working. But i cannot confirm as the firewall exception to ses.sas.download has been made. I will have to ask them to turn it off but for now they are busy with our other issue on the cluster. This is the new error. seems like some issue with either kube-vip/ coredns/kubernetes api server/ calico etc. try to fix this. Thanks again for the SAS variables though as that seems to be work.
Error from server (InternalError): error when creating "/work/deploy/manifest.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Unknown Host

Closing as duplicate of #372, repositoryWarehouse work will be tracked there.


"Invalid cross-device link" when running cargo lambda new blah on Manjaro

   ~/projects  cargo lambda new sa-solver-sns-result-ws-dispatcher
? Is this function an HTTP function? No
? AWS Event type that this function receives sns::SnsEvent
Error:
× failed to move package: from "/tmp/.tmpX6qXqg" to "sa-solver-sns-result-ws-dispatcher"
╰─▶ Invalid cross-device link (os error 18)
~/projects  cargo lambda new sa-solver-sns-result-ws-dispatcher
? Is this function an HTTP function? No
? AWS Event type that this function receives sns::SnsEvent
Error:
× failed to move package: from "/tmp/.tmpX6qXqg" to "sa-solver-sns-result-ws-dispatcher"
╰─▶ Invalid cross-device link (os error 18)
   ~/projects  uname -a                                                                                 1 ✘  18s   base 
Linux gauss 5.10.105-1-MANJARO #1 SMP PREEMPT Fri Mar 11 14:12:33 UTC 2022 x86_64 GNU/Linux
   ~/projects  cargo --version                                                                                 ✔  base 
cargo 1.62.0-nightly (dba5baf 2022-04-13)

Thanks for opening this issue!
It looks like Rust file:rename can fail in some Linux kernels, according to a similar issues in rustup: https://github.com/rust-lang/rustup/issues/1239
The solution seems to be to capture the error and do a Copy+Delete in those cases.

Aah, interesting. I'd expect that having /tmp on a different device would be not uncommon on *nix systems.
Rustup's fix involved them implementing some util functions themselves to fall back to copy+delete on a failed move/rename, but this seems like the kind of thing that should be in an existing library though rather than copy/pasting code from there.
fs_extra's move_dir seems like a candidate but I'm not sure if fs_extra is still maintained.

Ha, just seen your PR after writing the above, thanks for the quick turnaround!

@sdd I'll probably release a new version in the next couple of days. Meanwhile you can install it by cloning the repo:
git clone https://github.com/cargo-lambda/cargo-lambda
cd cargo-lambda
cargo install --force --path crates/cargo-lambda-cli

Will give it a go tomorrow, thanks!

Hi @calavera - unfortunately that change didn't fix the problem. If I change rename to the below then it works - but it requires the unstable #![feature(io_error_more)]:
pub fn rename<P, Q>(from: P, to: Q) -> io::Result<()>
where
    P: AsRef<Path>,
    Q: AsRef<Path>,
{
    match fs::rename(&from, &to) {
        Ok(ok) => Ok(ok),
        Err(e) => match e.kind() {
            io::ErrorKind::CrossesDevices => {
                match copy_and_delete(from, to) {
                    Ok(()) => Ok(()),
                    Err(_) => Err(e),
                }
            },
            _ => Err(e),
        },
    }
}

Sounds like ErrorKind::Other does not get returned anymore: https://github.com/rust-lang/rust/pull/85746
I think we can check against the error code directly for now until this is io_error_more is stabilized: https://github.com/rust-lang/rust/issues/86442#issuecomment-889391007

Aah, this works, without needing io_error_more:
pub fn rename<P, Q>(from: P, to: Q) -> io::Result<()>
where
    P: AsRef<Path>,
    Q: AsRef<Path>,
{
    match fs::rename(&from, &to) {
        Ok(ok) => Ok(ok),
        Err(e) if Some(libc::EXDEV) == e.raw_os_error() => {
            match copy_and_delete(from, to) {
                Ok(()) => Ok(()),
                Err(_) => Err(e),
            }
        },
        Err(e) => Err(e),
    }
}


Memory list UI

This PR branches off of recommendations so only the last commit has new changes.
Added display for showing users' past lists:

can show all past lists / all past lists of a given name / all past lists in a given time frame
can show the most recent lists
if no list of the name exists, it just shows all lists (basically, sometimes dialogflow cannot understand that "show me past grocery lists" means lists titled "grocery" and not "past grocery" which is why this exists)

@marcusschwartz Sorry, I think I fixed all of those in the previous branch when you mentioned it in the Recommendations PR but then this one branches off of that and I didn't rebase it. So should be fixed now.


Investigate calculating test coverage

We have a variety of different tests in Parsec, it would be good to know what is our test coverage. That would help us knowing if we need to do more or less.

I'm currently looking at tarpaulin for coverage, trying to see if/how our test framework works with it.
Seems to work by calling it like this:
RUST_LOG=info cargo tarpaulin --forward --command build --features="mbed-crypto-provider,direct-authenticator" --run-types bins --timeout 3600 -- -c e2e_tests/provider_cfg/mbed-crypto/config.toml

And then running the tests from a separate thread, followed by a pkill parsec.

It's worth considering whether we want to have the code coverage checks running in the CI and then having the results published to something like Codecov

Is it possible to calculate it for all of our tests and not just the end to end ones?

It's worth considering whether we want to have the code coverage checks running in the CI and then having the results published to something like Codecov

Would be nice! Personally I would prefer generating the results nightly to check what is the global coverage status instead of making it a requirement to merge a PR

Is it possible to calculate it for all of our tests and not just the end to end ones?

As in, stress tests, persistency tests etc? I think so, I have a plan for how to do it

Personally I would prefer generating the results nightly to check what is the global coverage status instead of making it a requirement to merge a PR

Yep, I don't think it's needed per PR, we just need to add it to the nightly stuff

As in, stress tests, persistency tests etc? I think so, I have a plan for how to do it

Ah yes those but I was also wondering about unit tests?

I'll try and figure that afterwards, currently trying with just integration testing - adding unit tests would be more problematic because it'd have to combine the results from multiple tarpaulin runs

I tried to run the service under tarpaulin and then run all the integration tests for every provider, sequentially, but it really didn't like it too much... It was getting a bunch of segfaults, the first ones I saw were due to using drop(...) in main.rs when reloading the service. I think there were more issues in other parts of the code, but I dropped it - will try with grcov which maybe is more straight forward to get working.

Code coverage can be seen here


two queues created

two queues were created for each type of sensor.
A queue was increased to 10 frames.

two queues were created for each type of sensor. A queue was increased to 10 frames.

I see the capacity is 1, please update

It was 1 in the previous code version, but I checked with size=10 yesterday and found it failing the test.
I will run the test with queue size=5 and update you.

two queues were created for each type of sensor. A queue was increased to 10 frames.

I see the capacity is 1, please update
It was 1 in the previous code version, but I checked with size=10 yesterday and found it failing the test.
I will run the test with queue size=5 and update you.

It was 1 in the previous code version, but I checked with size=10 yesterday and found it failing the test. I will run the test with queue size=5 and update you.

No need for more then 1


Use bundled gtest

Suggest to use bundled gtest in tests
Use dmlc bundled gtest in all CI scripts
Make clang-tidy to use dmlc embedded gtest

I have some issue with ctidy (it seems I broke it) which I am struggling to reproduce

I managed to reproduce the issue but still no clue what and why it is happening.
So I removed the move of ctidy on embedded gtest for now

@trams Could you provide a link to the failure?

@trivialfis sure. Here is a link https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-4900/3/pipeline/
I see this
[2019-09-26T22:23:09.697Z]  /workspace/tests/cpp/data/test_simple_csr_source.cu:2:10: error: 'gtest/gtest.h' file not found [clang-diagnostic-error]
[2019-09-26T22:23:09.697Z] #include <gtest/gtest.h>
I looked at commands json Cmake generated and it looked fine. It included the folder with googletests headers

@trams The tidy shouldn't look at test code.  See https://github.com/dmlc/xgboost/blob/2fa8b359e06783ed5824824ec8ffb32c1d4e1383/tests/ci_build/tidy.py#L122 .
We currently don't lint the unittests.

Wait I'm revisiting my tidy script, sorry for the noise.

@trams The tidy is running on the source code of googletest itself...

Could you clear out the gtest source code from build directory?

What do you exactly mean by that?
I have the issue with Travis CI checks as well as local checks. It check gtest source tree indeed but it does not fail on it (it produces only warnings). For some reason it complains on some *.cu files and only on some
I do not exactly know how bundled gtests work (only in theory). It has been implemented by @hcho3 . Could you help, @hcho3 ?
Also I suggest for now to focus on this pull request and deal with updating tidy scripts later

I'll take a look

@trams I think I have a fix. Can you mark the checkbox that says "Allow edits from maintainers"?

Here is the patch to fix clang-tidy:
diff --git a/Jenkinsfile b/Jenkinsfile
index 6eab50fe..91d6fc35 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -116,7 +116,7 @@ def ClangTidy() {
     def docker_binary = "docker"
     def dockerArgs = "--build-arg CUDA_VERSION=9.2"
     sh """
-    ${dockerRun} ${container_type} ${docker_binary} ${dockerArgs} tests/ci_build/clang_tidy.sh
+    ${dockerRun} ${container_type} ${docker_binary} ${dockerArgs} python3 tests/ci_build/tidy.py
     """
     deleteDir()
   }
diff --git a/tests/ci_build/clang_tidy.sh b/tests/ci_build/clang_tidy.sh
deleted file mode 100755
index 5c96a78b..00000000
--- a/tests/ci_build/clang_tidy.sh
+++ /dev/null
@@ -1,18 +0,0 @@
-#!/bin/bash
-
-export GTEST_PKG_NAME=release-1.8.1
-export GTEST_DIR_NAME=googletest-${GTEST_PKG_NAME}  # uncompressed directory
-export GTEST_ZIP_FILE=${GTEST_PKG_NAME}.zip	    # downloaded zip ball name
-
-rm -rf gtest googletest-release*
-
-wget -nc https://github.com/google/googletest/archive/${GTEST_ZIP_FILE}
-unzip -n ${GTEST_ZIP_FILE}
-mv ${GTEST_DIR_NAME} gtest && cd gtest
-cmake . -DCMAKE_INSTALL_PREFIX=./ins && make
-make install
-
-cd ..
-rm ${GTEST_ZIP_FILE}
-
-python3 tests/ci_build/tidy.py --gtest-path=${PWD}/gtest/ins
diff --git a/tests/ci_build/tidy.py b/tests/ci_build/tidy.py
index d51d0fd2..ee2fa2f5 100755
--- a/tests/ci_build/tidy.py
+++ b/tests/ci_build/tidy.py
@@ -29,15 +29,12 @@ class ClangTidy(object):
     '''
     clang tidy wrapper.
     Args:
-      gtest_path: Full path of Google Test library.
       cpp_lint: Run linter on C++ source code.
       cuda_lint: Run linter on CUDA source code.
     '''
-    def __init__(self, gtest_path, cpp_lint, cuda_lint):
-        self.gtest_path = gtest_path
+    def __init__(self, cpp_lint, cuda_lint):
         self.cpp_lint = cpp_lint
         self.cuda_lint = cuda_lint
-        print('Using Google Test from {}'.format(self.gtest_path))
         print('Run linter on CUDA: ', self.cuda_lint)
         print('Run linter on C++:', self.cpp_lint)
         if not self.cpp_lint and not self.cuda_lint:
@@ -61,8 +58,7 @@ class ClangTidy(object):
         os.mkdir(self.cdb_path)
         os.chdir(self.cdb_path)
         cmake_args = ['cmake', '..', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON',
-                      '-DGOOGLE_TEST=ON', '-DGTEST_ROOT={}'.format(
-                          self.gtest_path)]
+                      '-DGOOGLE_TEST=ON', '-DUSE_DMLC_GTEST=ON']
         if self.cuda_lint:
             cmake_args.extend(['-DUSE_CUDA=ON', '-DUSE_NCCL=ON'])
         subprocess.run(cmake_args)
@@ -108,6 +104,8 @@ class ClangTidy(object):
                         '--cuda-gpu-arch=sm_' + capability)
             elif components[i].find('--std=c++11') != -1:
                 converted_components.append('-std=c++11')
+            elif components[i].startswith('-isystem='):
+                converted_components.extend(components[i].split('='))
             else:
                 converted_components.append(components[i])

@@ -156,6 +154,7 @@ class ClangTidy(object):
                 return False
             isxgb = path.find('rabit') == -1
             isxgb = isxgb and path.find('dmlc-core') == -1
+            isxgb = isxgb and (not path.startswith(self.cdb_path))
             if isxgb:
                 return True

@@ -235,13 +234,11 @@ if __name__ == '__main__':
     parser = argparse.ArgumentParser(description='Run clang-tidy.')
     parser.add_argument('--cpp', type=int, default=1)
     parser.add_argument('--cuda', type=int, default=1)
-    parser.add_argument('--gtest-path', required=True,
-                        help='Full path of Google Test library directory')
     args = parser.parse_args()

     test_tidy()

-    with ClangTidy(args.gtest_path, args.cpp, args.cuda) as linter:
+    with ClangTidy(args.cpp, args.cuda) as linter:
         passed = linter.run()
     if not passed:
         sys.exit(1)
diff --git a/tests/cpp/common/test_enum_class_param.cc b/tests/cpp/common/test_enum_class_param.cc
index da5ddd67..dd3fc256 100644
--- a/tests/cpp/common/test_enum_class_param.cc
+++ b/tests/cpp/common/test_enum_class_param.cc
@@ -31,7 +31,7 @@ TEST(EnumClassParam, Basic) {
     {"foo", "frog"}, {"bar", "10"}
   };
   // try initializing
-  param.Init(kwargs);
+  param.Init(kwargs);  // NOLINT(clang-analyzer-core.UndefinedBinaryOperatorResult)
   ASSERT_EQ(param.foo, Foo::kFrog);
   ASSERT_EQ(param.bar, 10);

Sorry for a slow response.
This pull requests allows to be edited by maintainers so feel free to go ahead and modify anything
Meanwhile I'll try to resubmit move ctidy to use embedded google test
@hcho3 thank you for a detailed analysis !

Thank you! I was a bit confused by tests not passing

@trams There was outage in the test farm (#4921). It is fixed now.


Error

What operating system are you using?
windows
What browser are you using?
chrome
Describe the bug
[GoogleGenerativeAI Error]: Error fetching from [451 Unavailable For Legal Reasons]
What prompt did you enter?
No response
Console Logs
No response
Participation

[X] I am willing to submit a pull request for this issue.

Same issue here.

删除部署在Vercelde的palm-proxy，这个代理会被禁用或者封号，重新部署palm-netlify-proxy到Netlify。


Accessing PropTypes via the main React package is deprecated.

Warning: Accessing PropTypes via the main React package is deprecated. Use the prop-types package from npm instead.
This warning occurs from React version v15.5.0
https://facebook.github.io/react/blog/2017/04/07/react-v15.5.0.html

Fix is in Pull Request


Use fixed uid/gid for vtpm container

Use the fixed ids create in dom0 to run the VTPM container, this allows to run the container as non-root user and have control over the access control as described in #3986 .
linuxkit-525400123456:~# ps aux | grep vtpm
 1540 root      0:00 /usr/bin/containerd-shim-runc-v2 -namespace services.linuxkit -id vtpm -address /run/containerd/containerd.sock
 1561 vtpm      0:00 {init.sh} /bin/sh /usr/bin/init.sh
 1629 vtpm      0:00 /usr/bin/vtpm_server
 3297 root      0:00 grep vtpm

LGTM
Should we also update the hash for other references to eve-dom0-ztools ? Or are the others up to date?

don't think so, I'll do in another PR.

@eriknordmark apart from vtpm, only pillar uses lfedge/eve-dom0-ztools, I'll wait for @andrewd-zededa  #3996 and then update both vtpm and pillar.


Move the random init on Individual into a classmethod make_random

Right now we support two __init__ methods on Individual, with the following signatures:
    def __init__(
        self,
        data: ProblemData,
        penalty_manager: PenaltyManager,
        rng: XorShift128,
    ) -> None:

    def __init__(
        self,
        data: ProblemData,
        penalty_manager: PenaltyManager,
        routes: List[List[int]],
    ) -> None:

The first is what I call the "random init", the second the "route init". Having multiple overloaded __init__ methods really screws up typing and generating docs, as well as IDE support. We can do better to express the intent of the random init by having it bound as a classmethod instead.
I propose something like this:
    def __init__(
        self,
        data: ProblemData,
        penalty_manager: PenaltyManager,
        routes: List[List[int]],
    ) -> None:

   @classmethod
   def make_random(
        cls,
        data: ProblemData,
        penalty_manager: PenaltyManager,
        rng: XorShift128,
    ) -> Individual:

This should not require changes on the C++ side, other than in the bindings. Intended usage on the Python side is individual = Individual.make_random(data, pm, rng).

Pybind does not yet support classmethods out of the box (this issue: https://github.com/pybind/pybind11/issues/1693), so I am using a workaround in #165.


SentenceTokenizer works badly with numbered lists

SentenceTokenizer works badly with numbered lists. For example this text will produce 10 sentences:
1.  Where is the bar located?
2.  What are the bar's opening hours?
3. Is there an entrance fee in the bar?
4. What styles of music are typically played in the bar?
5. Does the bar offer any discounts for students?

Yes, the sentence tokenizer looks at the syntax of a text based on puncuation and cannot distinguish between a period used for a numbered and a period used the end a sentence. Another more advanced mechanism would be needed to parse numbered list as well as ordinary sentences ending in a period. Won't fix, that is not in this parser built on regular expressions.


JumpCloud identity sync

[x] Test OIDC with JumpCloud
[ ] JumpCloud user/group sync

@bmanifold -- got the go ahead on this one, may need a quick sync to chat about UI changes. Can come after #4506

API keys in JumpCloud are per admin as opposed to per-account, so will need to call this out in the docs.

Update from customer. All-in on SCIM integration!


[Proposal] CSRF checks on GET routes

The Vulnerabilities on GET Routes
For historical reasons some GET routes in Gitea do more than getting info and actually change state, for example /user/logout, and GET routes with action/:action in it.  Current CSRF handler only checks POST routes, and thus leaves those GET routes vulnerable.
One particular route admin/notices/empty should probably be a POST i/o GET route.
Strategies
We are working on a PR to address these vulnerabilities, and would like the community feedback on the best strategy.
Option 1: reuse existing CSRF token + referrer-policy header
In this option we will append the existing CSRF token to vulnerable GET routes as a query string, validate the token. A referrer-policy header can be set to same origin to prevent abuse of the token.
This option has less changes to existing code.
Option 2: generate a new JWT token with a key exclusive for this purpose
In this option we will generate a new JWT token and send to the browser via a new cookie.  This will have more changes to the code.
Questions
security concerns on reusing csrf token on URL
We are thinking to implement option 1, and have code already done. It has less code changes and hopefully can be merged soon. Are there any concerns on this option 1?
response code or redirect
What should we do if the required token is missing or invalid? Should we redirect to home page, or return 403? Browsers treat 403 error code differently, for example Chrome on Mac displays the following message  which may not be what you want.
This site can’t be reached
The web page at https://gitea.on-my-super-domain.dummy/user/logout might be temporarily down or it may have moved permanently to a new web address.
ERR_INVALID_RESPONSE

Potential Impact
Both options require an extra handler on vulnerable routes to check the token in query string, and minor changes in URL in templates. While the PR can include the new token in default templates, this is potentially a breaking change to people using custom templates. The custom templates need to be updated to include the token in generated URL.

This is a dupe ticket. I'm on mobile right now so I can get to it, but it was decided to ensure CSRF on those routes.
When I get to a computer I'll link ticket.

Hello,
appreciate the quick response. Before creating this issue, I have searched and read all open and closed issues with the keyword csrf, and haven't found anything similar. (maybe it has been deleted?) This is about CSRF checks AFTER users log in, not on login route. The mischief can be done with a link to the unprotected route, on issue/profile pages on the same domain.
If someone is already working on it, could you share the link to the issue and discussions?
If it is not the same issue, do you mind re-opening the issue?
Gitea (and gogs) has some unconventional use of http verbs, such as performing delete action on GET requests.

It's hard to search for because the security researcher who reported it deleted their account, and because it included some sensitive details removed the details from the ticket. However the it's also been reported through the<EMAIL_ADDRESS>email so the owners (and a couple maintainers) have the vulnerability details, and the ticket is kept open as a placeholder until it can be resolved.
I'm back at a computer now and the ticket is #4357

I actually have read that issue, which is about unprotected post routes in API. This issue is about the unprotected GET routes of UI in the browser. So I thought these are totally different issues. Do you mind reopening it for community comments so a fix can be decided and submitted soon?
The researcher (cezar??) brought my attention to those CSRF issues. I am especially concerned about them because our application is multi user environment. Not sure why the account has been deleted. I hope the researcher can test and verify the fixes if the researcher is still hanging around.
I have done something about token check in api routes,  and have proposed to limit API access to token only in reqToken handler. I will prepare a pull request on that since it is a separate issue.  One of the maintainers mentioned that a few API GET routes are used in the user/repo search in dashboard. these routes do not have reqToken handler, and will continue to work without token. But in the long run I think it is a good practice to always supply a token when making requests to api routes.  People make less mistakes when the logic is simple.
I have a small team and limited resources, and prefer not to maintain a separate repo, or manually resolve the merge conflicts.

Just in case someone will be reading this issue in the future, we have opted for rewriting the code and using a separate key and token exclusive for vulnerable GET routes. it is cheap and fast to generate and verify a token stored in another cookie. The consequences of losing CSRF token for POST requests are huge. It is a risk not worth taking.

This issue is about the unprotected GET routes of UI in the browser. So I thought these are totally different issues.

Yes, they are totally different issues. That's why I opened #4337, where I explained that, for example, embedding an image targeting /user/logout it's possible to logout an user with a GET request. So as @techknowlogick said, both CSRF POST on API and CSRF GET on UI are already reported to the developers.

Not sure why the account has been deleted.

I made a new account ( the one I'm using now ) without all the information the old one had. You can say thanks to Microsoft for this.

I hope the researcher can test and verify the fixes if the researcher is still hanging around.

After I've reported the XSS issue I stopped following the Gitea development, also because I don't use it anymore ( I don't need a private git hosting at the moment ). But I can help in fixing the issues I sent, so I'm still here.
You asked for feedback.
As @msg7086 pointed out here, every endpoint you call "vulnerable GET routes" should be POST instead of GET, so there's no need in CSRF Tokens on GET. Why? Because GET is only for "getting" resources and displaying them. Take a look at Gitea Swagger API: all the GET requests shows information like user emails, user's name, etc. There aren't any actions peroformed on GET there so there isn't any dangerous stuff.
So, logout, watching repos, following users, instance actions in the Admin settings ( issue reported by another researcher but I can't find it ) and the other routes<EMAIL_ADDRESS>received should be changed to POST with a CSRF check. Seems pretty easy to me as a fix.

I am especially concerned about them because our application is multi user environment.

Sure, but thanks to @lunny security issues are taked more seriously than when I made my first reports. And thanks to the developers and community members, high priority issues that are still open now are getting fixed.
Of course, if you're really concerned beacuse of a multi user environment, I don't recommend even the latest 1.5.1 version as there are CSRF vulnerabilties both on API and some Admin settings, but you already know this. You can help thus, for example with your Pull 4840 on distrusting the cookies in API authentication. Not sure why that one is still blocked BTW.
Thanks @beeonthego.
PS: Regarding version 1.5.1, thank you all.

@cezar97 Thanks for the insight, and welcome back! I am glad you are still here. If you can create a new account using the username of deleted account, you should keep this one. Otherwise another person might use it and pretend to be you.
I have read issue #4337 but couldn't find it when I created this issue to prepare a PR. The content has been deleted and it does not contain csrf or xsrf keyword now. The credit goes to you for reporting the csrf vulnerabilities on GET routes.
Thank you for being patient and persistent. I think Gitea as a community should have responded faster to address the security issues (I am one of the users and I am guilty of this too).  Some fixes are easy, for example my PR #4840 only has 3 lines of code. It will probably only take a few minutes for someone already familiar with the history and the code of this project.
PR #4840 needs a change in access policy and test suites. It has failed some of the tests because the tests assume access to API routes with cookie. The same vulnerability exists in Gogs API routes. I have submitted the same PR to Gogs (matching gogs var and file name) and the PR has passed all tests there. I will see which team of maintainers respond to the PR faster. Gitea maintainers may have forgotten the pain of waiting for PR to be merged, the same reason they have forked Gitea from Gogs.

@techknowlogick Is someone else already working on these GET routes or changing them to POST routes? If someone is already working on it, I will focus on something else.

That's what I did. It's the same username :).
I know I didn't make things easy by deleting the first account, anyway I didn't write a meaningful title in that issue either.
I know some fixes are easy, but hopefuly if community contributes we can help the developers. BTW as said, things are improving even if slowly.
Regarding Gogs, I've also contacted the developer with issues that affected it ( for example it doesn't have private emails so the leak that got fixed in 1.5.1 doesn't apply ). It was more friendly, but none of the issues were fixed. I only got a label with priotity/critical ( Issue 5355 in Gogs repository ). So until the XSS is fixed, it doesn't really matter if your CSRF PR gets merged, sadly.

@beeonthego seems that someone saved the first edit of the above deleted report ( #4337 ).
If you're interested you can find it at http://web.archive.org/web/20180628164433/https://github.com/go-gitea/gitea/issues/4337.

@cezar97 In one of your issues deleted, you mentioned about the possibility of theft in CSRF tokens designed for POST routes. Developers have to send the tokens to the browser as page content in one way or another ( in http head or body content, or inject into javascript code).  When a victim visits a malicious external link, the attacker can use a script to send a GET request to the server, parse the response and obtain the token. Browsers will send the victim's cookie automatically, even if the attacker can not access the cookie. So these tokens are no secrets anyway.  It is the browser's same origin policy that prevents the attacker from sending POST requests using stolen csrf token and cookie. What do you think?
For this reason, a CSRF token on GET routes has to be different one, if the page contains important information and can not change to a POST route.
Appreciate your wisdom in this area.

@cezar97 Thank you for the insight. You have been extremely helpful. The discussion is getting deep now. I hope this thread will help someone to understand and implement CSRF properly in their own app in the future.
Regarding cross site GET requests, browsers actually DO SEND the cookie along with the request even if the resource is on another host. I have tested this on Chrome for Mac Version 68.0.3440.106. The developer console displays a warning icon and message "Provisional headers are shown", and it DOES NOT show all the request headers. This is the same whether browser caching is on or off. However when I examine the request headers on nginx proxy, the cookies ARE clearly there in the cross site GET request! Other browsers are probably doing the same. If you are curious, you can test different browsers,  check the request headers on server, and share the results.
It seems that browsers do not distinguish whether a GET request is made by entering a link in address bar, clicking on browse history/bookmark, or is initiated from a page on another site. A browser checks CORS policy, and if the request is allowed,  it sends cookie along with the request to fetch resources. This is for a better UX.
The implication is that when a victim visits a malicious external site, the site have full access to all the resources available on GET routes. Browsers send the cookie automatically. The malicious site does not have direct access to cookie, but it can take advantage of browsers' behaviour, and steal information or valuable/private code.
The only defense is to have a token on all GET routes too, no matter whether the routes perform action or not. That is why most internet banking sites have ugly URL with long query strings.

Regarding cross site GET requests, browsers actually DO SEND the cookie along with the request even if the resource is on another host.

Now I understand what you mean. Yes, that's how the things works with every browser. Requests to a site are send with the cookies to that site only.
But CSRF check on GET can't fix this, it just make sure that the Requests are legitimate and send by the user and not a malicious website as you said, because the malicious website doesn't have the Token. Even if they aren't legitimate and CSRF blocks them server side, cookies are still send by the browser.
The point is that even if cookies are send on GET, there is no way for a malicious website to get the request headers where cookies are stored without javascript, but as protection, cookies aren't sent if the request is from a script, as said in another comment.
So if GET routes follow the best practice of not performing actions, an external site can't get the request with the cookies or the response with the information requested either.
In other words, this is how things works, not a vulnerability.
Hope I answered your doubts :)

@cezar97 Thank you very much for the patience and detailed explanations. I do have a better understanding now.
HTML tags of embedded resources (such as img, style, script etc) are not covered by same origin policy. For these tags, browsers send cookies along with cross site GET request without asking. These are the GET routes I referred to before.
You are right, GET routes should not perform action or change state.  Changing the routes to POST involves a lot of changes in the front end JavaScript, the templates, and test suites.
Internally we have modified go-macaron/csrf  module in the vendor folder and context module,  and added an extra token on these GET routes in templates for a quick fix. I will upload the go-macaron/csrf module to my repo, just in case someone will be interested in fixing their app too.
There won't be a PR as eventually the routes should be changed to POST. Changes in vendor folder may never get merged in upstream repo, and thus a PR will never pass the dependency tests.

@beeonthego but than on the other hand logout will not work without javascript (some people do use gitea without js)

So instead of Javascript a form is preferable.

or you could do /logout?csrf=123213123131 and on the logout route if the csrf is not present then present a page that has the logout link with the csrf

If it's easier for you to implement, then it's OK. But as @beeonthego mentioned, referrers with same-origin should be enforced to prevent leaking the Token to others site, considering also the Open Redirects.
And as anyone noticed or reported, seems there's a remaining CSRF on the Admin Panel. Here's is the Gogs link with the report: https://github.com/gogs/gogs/issues/5367.

@Siesh1oo Thanks for sharing the trick. I have read about it in your comments in another issue.
No matter how to implement it, people using custom templates will have to update templates manually. It is a breaking change.
In our app we have custom oAuth2 redirect routes (GET) and have to protect with a token in query string. To reduce the risk of exposing the token by referrer, we use a separate short-life token for such GET requests only, and render a normal page without protected content if the token is missing or expired.
I have uploaded the modification to go-macaron/csrf module under this branch
https://github.com/beeonthego/csrf/tree/extra-token-on-get-action-routes
I am not sure how to proceed. Changes in vendor folder may never be merged in upstream repo.

@techknowlogick : Technically possible, but be sure to understand the caveats https://www.owasp.org/index.php/Cross-Site_Request_Forgery_(CSRF)_Prevention_Cheat_Sheet#Disclosure_of_Token_in_URL

@beeonthego but than on the other hand logout will not work without javascript (some people do use gitea without js)

When Javascript is disabled, gitea does not even show the logout button (nor any other functional button) -- is gitea supposed to be functional without javascript?

Oh, you are right logout is not visible :) Than yeah we can do that with JavaScript. I would prefer form with post tho


Get errors from SQL hangs the UI

sometimes, we will get errors from SQL eg. accessing tables you don't have access to. (database_firewalls). Today the tool just hangs

appear again on Master commit 48cb2bcecffff82ab150e89d3343bfff4fc4b387, will take a look


Missing Class comments

closes #126

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 48.515%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
490

Relevant Lines:
1010

💛  - Coveralls


Propagate errors that arise updating objects

When an attempt to update an existing Kubernetes object fails and the server returns an error, don't just log the error, but propagate it to callers of kubecfg.createOrUpdate as well.
Fixes #252.

I didn't see an existing unit test that would cover this case. If you think we should have one, please let me know.


Clicking 'Open Folder' doesn't do anything with 'Use AzCopy for Improved Blob Upload and Download' checked

Storage Explorer Version: 1.5.0
Platform/OS Version: Windows 10/ Linux Ubuntu 16.04/ MacOS High Sierra
Architecture: ia32
Build Number:20180826.5
Commit: d6717c99
Regression From: Not a regression
Steps to Reproduce:

Navigate to 'Experimental' menu -> Select ‘Use AzCopy for Improved Blob Upload and Download’.
Open one blob container which contains one blob at least -> Download a blob.
After downloading successfully -> Click ‘Open Folder’ in Activities panel.

Expected Experience:
Clicking ‘Open Folder’ works well.
Actual Experience:
Clicking 'Open Folder' doesn't do anything.

Fixed in 20180927.3.


🐛 Bug Report: Issues when Backstage is run in devcontainer

📜 Description
When I run Backstage in docker and expose ports 3000 and 7007, the application doesn't show up in my browser.
However, when I install it locally, it works just fine.
👍 Expected behavior
When I run yarn dev inside the docker image with 3000 and 7007 ports exposed, the Backstage app should appears in my browser.
👎 Actual Behavior with Screenshots
I have a blank screen with the loading bar

👟 Reproduction steps
To reproduire the issue:

Create a Nodejs devcontainer in vscode
Run npx @backstage/create-app@latest
cd to the directory where the app was created
Run yarn dev
The browser opens automatically and it keeps trying to load the page

📃 Provide the context for the Bug.
No response
🖥 Your Environment
No response
👀 Have you spent some time to check if this bug has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Are you willing to submit PR?
None

@sonikro do you see anything wrong here in regard to devcontainers? 🤔

I've been fighting this exact problem for a few weeks now. If I use the dev tools in Firefox I can see that only the first 2 network requests succeed, the others never start at all. I suspect it's an issue between React and Firefox, because I switched to using Safari and it works every time. I can also switch to a new container in Firefox (like a Chrome profile) and get the bundle to load, but that would also mean logging into my OAuth provider every time I had to switch so I went with using Safari instead.
I did also move my config over to http://<IP_ADDRESS> as the baseUrl in case it was something to do with the container attempting DNS resolution, but that did not solve the problem. I also hit an issue early on where VSCode stopped automatically exposing port 7007 and only 3000 was open, which I resolved by adding both ports to the config in devcontainer.json.

Can you share what DevContainer specification are you using? Are you using the one available in the Contrib folder? https://github.com/backstage/backstage/tree/master/contrib/.devcontainer
I work on Backstage inside of a DevContainer pretty much every day with no problems, but there are some settings you might want to setup in your devcontainer specification to make sure you are forwarding the ports from the container to your host machine.
Also, what is your environment? Are you using Windows + WSL2 + Docker Desktop, or Macbook with Docker Desktop? All of that information might change how the docker layer is exposing the ports to your host computer.

I haven't reported this (until I saw this issue and wanted to share my experience with @AnasSahel) because I assumed it was an interaction between React and Firefox. The HTTP request never actually starts in my case, so logically I can't blame the Backstage backend for not serving the file when requested. Switching browser without touching the devcontainer let me launch immediately. So I blamed the Firefox cache, but I have little to no experience with React or JS in general, so I don't know if there is something in how Backstage interacts with it that may contribute to the problem.

Hello all,
Thanks for your answer. I tried switching between browsers but the result is the same.
I'm under MacOS, with Docker Desktop.
Here are my configurations:
File: devcontainer.json
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/javascript-node-postgres
{
  "name": "backstage-lab01",
  "dockerComposeFile": "docker-compose.yml",
  "service": "app",
  "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
  "forwardPorts": [3000, 7007]

  // Features to add to the dev container. More info: https://containers.dev/features.
  // "features": {},

  // Use 'forwardPorts' to make a list of ports inside the container available locally.
  // This can be used to network with other containers or with the host.
  // "forwardPorts": [3000, 5432],

  // Use 'postCreateCommand' to run commands after the container is created.
  // "postCreateCommand": "yarn install",

  // Configure tool-specific properties.
  // "customizations": {},

  // Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
  // "remoteUser": "root"
}

File: docker-compose.yml
version: '3.8'

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile

    volumes:
      - ../..:/workspaces:cached

    # Overrides default command so things don't shut down after the process ends.
    command: sleep infinity
    env_file:
      - .env

    # Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function.
    network_mode: service:db

    # Use "forwardPorts" in **devcontainer.json** to forward an app port locally.
    # (Adding the "ports" property to this file will not forward from a Codespace.)

  db:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_DB: ${POSTGRES_DB}

    # Add "forwardPorts": ["5432"] to **devcontainer.json** to forward PostgreSQL locally.
    # (Adding the "ports" property to this file will not forward from a Codespace.)

volumes:
  postgres-data:

File: Dockerfile
FROM mcr.microsoft.com/devcontainers/javascript-node:1-20-bookworm

# [Optional] Uncomment this section to install additional OS packages.
# RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
#     && apt-get -y install --no-install-recommends <your-package-list-here>

# [Optional] Uncomment if you want to install an additional version of node using nvm
# ARG EXTRA_NODE_VERSION=10
# RUN su node -c "source /usr/local/share/nvm/nvm.sh && nvm install ${EXTRA_NODE_VERSION}"

# [Optional] Uncomment if you want to install more global node modules
# RUN su node -c "npm install -g <your-package-list-here>"

And here is the structure of my project in vscode

Thanks all for your help

Hello all,
I fixed it. The app uses ipv6, thus the port is not forwarded correctly.
I changed the app-config.yaml to force ipv4 and now it's working
app:
  title: Scaffolded Backstage App
  baseUrl: http://localhost:3000
  listen:
    host: <IP_ADDRESS> # Added line, and now it's working

Glad this was fixed! Closing the issue


Add a button to "save changes dialog"

... as shown in the picture. The main idea - to shorten scenario with "copy".
Now we should "copy", then find (or open new) sql window, then "paste".
Maybe it could be suitable to add an option to change behaviour of "copy" button (copy to clipboard or copy to new sql window) - in the options dialog.

thanks for suggestion

Done

Verified


Wrong cache-key used for caching

While trying to switch to default cache-key in https://github.com/obsproject/obs-studio/pull/5137.
We were trying remove the cache-key settings when this issue happen.
An unexpected issue has occurred:
Cache Size: ~1123 MB<PHONE_NUMBER> B)
/usr/bin/tar --use-compress-program zstd -d -xf /__w/_temp/a957e912-7502-4f61-a192-d3e489b47e8d/cache.tzst -P -C /__w/obs-studio/obs-studio
Cache restored successfully
Restored cache with key: -x86_64

https://github.com/bilelmoussaoui/flatpak-github-actions/blob/9b33ba80aec2fc2652e05f15f19929fa98182889/flatpak-builder/index.js#L209-L212
It seems that cacheKey even if empty does not fullfill the condition and generate a wrong cache key which ends up being -x86_64 in our situation.

I believe this one was fixed in #48, it just hasn't had a release since that was merged. I've been using .../flatpak-builder@master and it seems to work as expected.

Closing as already fixed, I will try to do a new release in the upcoming weeks


Crash if OpenVRConfig isn't instantiated in Godot first

This is not by design :)
Found out on stream yesterday that if OpenVRConfig isn't initialized within Godot first that the engine crashes, presumably because certain default values aren't set up as the underlying singleton should be created even if the GDNative object isn't.
Workaround for now is to make sure you execute:
var openvr_config = preload("res://addons/godot-openvr/OpenVRConfig.gdns");
if openvr_config:
	print("Setup configuration")
	openvr_config = openvr_config.new()

before arvr_interface.initialize() is run.

Yes, actually now that OpenVRConfig does more, we'd want to instantiate it anyway all the time.

Indeed, which is why I never ran into this before but I guess we can't guarantee its instantiated.
Haven't found the cause yet though


Logical math error (MathLib)

This issue is Arcsinh() was not implemented correctly. If a 1 was passed into the function, it it would return 1.15 or some number similar which is not accurate. .88 would be the accurate return value.
This issue is for the MathLib Library

Issue patched and uploaded. Platformio has not been updated yet. I am working on testing more of the functions to verify the reliability of the code. This issue will be closed but please beware the library may still have a few logical math errors.


Feature/humanization

Adds item textures
Adds localizations

@Grissess can you un-fuck this and rebase it off dev? I forgot to pull from dev and it's angry now..


Tag a new release on PyPi?

Hi!
I recently discovered your package and really like it.
I've seen you have recently added support for inheriting the dispatch types from the type annotations. That's very nice!
As I'd like to use it in one of my packages, could you tag a new release on PyPi?
Thanks!

Hey @PhilipVinc!
I'm very glad to hear that you like the package! The current master still has a few kinks to work out, which is what I'm currently working on. I hope to push a new release to PyPi within the next 7 days. :)

Hey @PhilipVinc!
I've just tagged a new release on PyPI, so closed this issue.


Ubuntu 17.10 support

Hi,
please make Ubuntu Server 17.10 support.
Thanks..

rtinst 1.5.3 now works with fresh Ubuntu 17.10 installs


New pull request as per requests from live-lab team for CW24 HOL

Require a new PR for emergency PR submission for CW24 as per requests from live-lab team
Also, all minor formatting issues are resolved and live-lab retested by the Oracle Health team
Old PR here - https://github.com/oracle-livelabs/oracle-health/pull/7/files

@rghosh9 please resolve conflicts

Hi @rghosh9 Please add your WMS ID and see my other comments. Also, you have conflicts that must be solved.

WMS IS 11685

Need help on resolving these conflicts? Not sure. Any insights you can share?


How do I save detected face?

How do I save detected face?
Is there a special function for that?

Hi @martinenkoEduard, Is it implemented through python, JavaScript or others? Could you please help me in reproducing the same with complete steps at our end. Thank you!

Hi @martinenkoEduard, Is it implemented through python, JavaScript or others? Could you please help me in reproducing the same with complete steps at our end. Thank you!

#https://www.youtube.com/watch?v=jn1HSXVmIrA

from unittest import result
import cv2
import mediapipe as mp

mp_face_detection = mp.solutions.face_detection
mp_drawing = mp.solutions.drawing_utils
faceDetection = mp_face_detection.FaceDetection()

# Create a VideoCapture object and read from input file
# If the input is the camera, pass 0 instead of the video file name
cap = cv2.VideoCapture('Faces from around the world.mp4')

# Check if camera opened successfully
if (cap.isOpened()== False):
  print("Error opening video stream or file")

# Read until video is completed
while(cap.isOpened()):
  # Capture frame-by-frame
  ret, frame = cap.read()
  if ret == True:

    # rgb_frame = frame[:, :, ::-1]
    imgRGB = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = faceDetection.process(imgRGB)
    #print(results)
    if results.detections:
      for id, detection in enumerate(results.detections):
        print(detection)
        mp_drawing.draw_detection(frame,detection)

    # face_locations = face_recognition.face_locations(rgb_frame)

    # for top, right, bottom, left in face_locations:
    # # Draw a box around the face
    #     cv2.rectangle(frame, (left, top), (right, bottom), (0, 0, 255), 2)

    # Display the resulting frame
    cv2.imshow('Frame',frame)

    # Press Q on keyboard to  exit
    if cv2.waitKey(25) & 0xFF == ord('q'):
      break

  # Break the loop
  else:
    break

# When everything done, release the video capture object
cap.release()

# Closes all the frames
cv2.destroyAllWindows()

Hi @martinenkoEduard, MediaPipe does return detections, or face images, or rendered detection rectangles
but does not have any APIs for storage of these returned results. Thank you!


too many open files with restic cache

I ran restic -r REPO cache and after so time the output switched from
tree 4012a865 stored
tree 51b731e0 stored
tree 2cb44bf2 stored
tree 5d58cbe6 stored
tree 222c739a stored
tree 4aaa5840 stored
tree 403ccf4e stored
tree 27f0012d stored
tree 26678210 stored
tree 22cce287 stored
tree 577534ed stored
tree 438c26ad stored
tree 62d005af stored
tree 52c6e0dd stored
tree 2aa8c1d8 stored
tree 4b2e3f15 stored
tree 3b1ec46a stored
tree 3ee3edca stored
tree 64e9f611 stored
tree 4e753b99 stored
tree 3f68b4c8 stored

to
tree 65a8538b cache error: open /home/gordin/.cache/restic/trees/65a8538ba96f1c1ddda73598220b21d867bc7f19efea6f1a511ca11d17c8f419: too many open files
  load error: open /run/media/gordin/backup/elchstation.restic/trees/65/658b133e8ac99832c12d233e6e8ac6fe19ddb10e56fdcaf24e7d0171b01a5d63: too many open files
tree 65b5e3af cache error: open /home/gordin/.cache/restic/trees/65b5e3af2a29b252931a856eb4847d42a7a2a2900cc2d677ddb2a97114d651b8: too many open files
tree 6591b4ba cache error: open /home/gordin/.cache/restic/trees/6591b4bacdc47c4ff44de7724793d5a63bac6d3429ef7c9988e563e08fd5fe41: too many open files
tree 65af2414 cache error: open /home/gordin/.cache/restic/trees/65af2414a91d7757c8a35bd85941fc8aa4af788d0cf833d1db8e4e80e35579fd: too many open files
tree 65adc69f cache error: open /home/gordin/.cache/restic/trees/65adc69f66815a78f394a9ea00b8c9c342234341a70e411e3712c773a338c552: too many open files
tree 65591aa3 cache error: open /home/gordin/.cache/restic/trees/65591aa3ec492331d7f0524b22d06316a20a71ccfd5381f2abae1d98dbf5c63c: too many open files
tree 65bf323a cache error: open /home/gordin/.cache/restic/trees/65bf323a676020bf3936329e2929b9eea222ed9b56b1609e10c1bdb3d1de1eb1: too many open files
tree 65ccd1d9 cache error: open /home/gordin/.cache/restic/trees/65ccd1d992bd37d0113149f42bc53e9c88b666b365e02d314bd10825571e8819: too many open files
tree 655923a4 cache error: open /home/gordin/.cache/restic/trees/655923a42cbc3e540a13943faca54321a0f036351107f83ed8de2c9affc8dab4: too many open files
tree 6572fae5 cache error: open /home/gordin/.cache/restic/trees/6572fae565bbba595a206911227a67604e6f4f42fbf8d1e80ded0cdfa6aeb212: too many open files
tree 6569d0a6 cache error: open /home/gordin/.cache/restic/trees/6569d0a642c50243590a9fdab29eaa2c59b2162e8c19865457d3c36f221ebc31: too many open files
tree 65c414ef cache error: open /home/gordin/.cache/restic/trees/65c414eff641e4e6b406d72b9d26ddf1d15d38fe39c663a4b515306027d06f72: too many open files
...
tree 745853ec cache error: open /home/gordin/.cache/restic/trees/745853ecad86cf64e7c3e785995001d4b8532ffa4b904964e40b386275e30bb9: too many open files
tree 745b9d6f cache error: open /home/gordin/.cache/restic/trees/745b9d6f8c5ceeb12e13b7519b159146caed5efa3fc5acb236efa433719ab670: too many open files
tree 74621c9e cache error: open /home/gordin/.cache/restic/trees/74621c9e02235c8f79eb4ee2cb9657acc363cceb93a6fb32cd95cb36a3f35603: too many open files
tree 74604a97 cache error: open /home/gordin/.cache/restic/trees/74604a97b1104267ef029a1bb51d65679d676ca4d985228a0c8edeff222e3933: too many open files
tree 74787003 cache error: open /home/gordin/.cache/restic/trees/74787003e58f43eace677f06ca67bfae1a757a356025e057da4e53c471af0340: too many open files
tree 7482217f cache error: open /home/gordin/.cache/restic/trees/7482217f46e1f6a07d85285ee8fa5d274d676e1fd2c42834caafb99e61a948e0: too many open files
tree 7481c21d cache error: open /home/gordin/.cache/restic/trees/7481c21d6539390b44710458a66e25c5194f4ad19398e626e8c2edc78d897a36: too many open files
tree 7480c07a cache error: open /home/gordin/.cache/restic/trees/7480c07a9492718c06065ec43cedb6222cfbbf13ee6d9c991f8323ee2e89cd8e: too many open files
tree 74827a6f cache error: open /home/gordin/.cache/restic/trees/74827a6fce87366c5e7e087ef3b6a38a160da22a33ca72086bacf9ce8330ce34: too many open files
tree 7485be19 cache error: open /home/gordin/.cache/restic/trees/7485be199d2ac69f30b9583b8b881ffc4ed37cad2224b9ea673b391ded23ed2f: too many open files
  store error: open /home/gordin/.cache/restic/trees/2171d7b6ed3ed8eac4c7872f3e0e2e21baf73955f9c26c62ceac85adf53d700e: too many open files
tree 7485b5da cache error: open /home/gordin/.cache/restic/trees/7485b5da6c2c1163a55fea5471b71a862672cddb1925b25a46d638709f4feb98: too many open files
tree 74864cf9 cache error: open /home/gordin/.cache/restic/trees/74864cf9f7d11c331563c2fa06307405cd2be842e79ca4d11f9b7c73a00cc4d1: too many open files
  store error: open /home/gordin/.cache/restic/trees/2880bdce522c0bad17bea21e9a0e927739a3a08998ae2f4527b6636236b78135: too many open files
tree 748cf0ff cache error: open /home/gordin/.cache/restic/trees/748cf0ff2d98c418a4090c3f894f7dad7aaf3e0e2e37e174cc70558ba15410f7: too many open files
tree 748f0bdf cache error: open /home/gordin/.cache/restic/trees/748f0bdf776da65954fb264a9a9144a6ea6fbd358b869859388fcaad4c673901: too many open files
tree 7495ca9c cache error: open /home/gordin/.cache/restic/trees/7495ca9c713ca28eb184ca8e0b21780d75641ebc25cd29596e755d8d153dc9bf: too many open files
tree 74954c52 cache error: open /home/gordin/.cache/restic/trees/74954c52bd42ffdd5f2cdafb7c1bc3761c77221de7668ada3514dd891f322e2c: too many open files
tree 74a57262 cache error: open /home/gordin/.cache/restic/trees/74a572621916e388c4b30c01aa396d07325d8852880bf1a6f95bb9c0c8de274f: too many open files
tree 74a395ef cache error: open /home/gordin/.cache/restic/trees/74a395ef17637bc3322c53db89faf983249937230e1811ff9f7cd09369caed32: too many open files

Wow, interesting. i'll investigate that.

This is fixen in b17840c6ee9d1d3c80d1cbd2dacf5c26fb53ca51, I refactored the cache command.


NodeClaim stuck in 'Unknown' (Cannot disrupt NodeClaim: state node doesn't contain both a node and a nodeclaim)

Description
Observed Behavior: Karpenter raises a new EC2, but it doesn't connect to the EKS - Instead it's stuck in status 'Unknown': "Cannot disrupt NodeClaim: state node doesn't contain both a node and a nodeclaim"
Expected Behavior: The node is added to the EKS cluster
Reproduction Steps (Please include YAML): EC2NodeClass + Nodepool
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: karpenter
  namespace: kube-system
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["2"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: karpenter
      expireAfter: 720h  # 30 * 24h = 720h
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: karpenter
  namespace: kube-system
spec:
  amiFamily: AL2  # Amazon Linux 2
  role: "KarpenterNodeRole-<cluster>"  # replace with your cluster name
  subnetSelectorTerms:
    - tags:
        Environment: test
        Tier: public
  securityGroupSelectorTerms:
    - tags:
        "aws:eks:cluster-name": <cluster>
  amiSelectorTerms:
    - id: ami-00710ab8f493e2428
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        iops: 10000
        encrypted: false
        deleteOnTermination: true
        snapshotID: snap-0ec4fd6705eea533e
  tags:
    env: test
    Name: balance-test-karpenter

Extra info:
I'm trying to replace my Auto-Scaler with Karpenter.
I gave the nodes the exact same:

IAM role.
Security-Group.
EBS volume is based on the same snapshot.
Exact same AMI.

I've added the required roles to aws-auth:
- groups:
  - system:nodes
  - system:bootstrappers
  rolearn: arn:aws:iam::<account_id>:role/KarpenterNodeRole-<cluster>
  username: system:node:{{EC2PrivateDNSName}}

I've entered the EKS worker node and ran:
journalctl -u kubelet
but no entry was added there.
I tried changing the roles name, tried adding permissions to the role, tried adding permissions to the SG, Nothing, the nodes just refuse to connect.
Karpenter logs:
{"level":"INFO","time":"2024-11-25T15:53:54.818Z","logger":"controller","message":"found provisionable pod(s)","commit":"a2875e3","controller":"provisioner","namespace":"","name":"","reconcileID":"6703d662-e9b1-4f99-9c55-e72f0aaa6b7e","Pods":"over-provisioning/over-provisioning-6d568b6cf8-7tqjr, over-provisioning/over-provisioning-6d568b6cf8-8p8bf","duration":"181.987056ms"}
{"level":"INFO","time":"2024-11-25T15:53:54.818Z","logger":"controller","message":"computed new nodeclaim(s) to fit pod(s)","commit":"a2875e3","controller":"provisioner","namespace":"","name":"","reconcileID":"6703d662-e9b1-4f99-9c55-e72f0aaa6b7e","nodeclaims":1,"pods":1}
{"level":"INFO","time":"2024-11-25T15:53:54.819Z","logger":"controller","message":"computed 1 unready node(s) will fit 1 pod(s)","commit":"a2875e3","controller":"provisioner","namespace":"","name":"","reconcileID":"6703d662-e9b1-4f99-9c55-e72f0aaa6b7e"}
{"level":"INFO","time":"2024-11-25T15:53:54.842Z","logger":"controller","message":"created nodeclaim","commit":"a2875e3","controller":"provisioner","namespace":"","name":"","reconcileID":"6703d662-e9b1-4f99-9c55-e72f0aaa6b7e","NodePool":{"name":"karpenter"},"NodeClaim":{"name":"karpenter-wxbnk"},"requests":{"cpu":"1780m","memory":"2418Mi","pods":"6"},"instance-types":"c4.large, c5.large, c5.xlarge, c5a.large, c5a.xlarge and 55 other(s)"}
{"level":"INFO","time":"2024-11-25T15:53:58.329Z","logger":"controller","message":"launched nodeclaim","commit":"a2875e3","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"karpenter-wxbnk"},"namespace":"","name":"karpenter-wxbnk","reconcileID":"01c8fd77-f4cb-4572-a600-333737c2caeb","provider-id":"aws:///us-east-2b/i-052e9c4c5d91c8767","instance-type":"c7i-flex.large","zone":"us-east-2b","capacity-type":"spot","allocatable":{"cpu":"1930m","ephemeral-storage":"89Gi","memory":"3114Mi","pods":"29"}}

(No errors AFAIK)
The auto-scaler still works as expected though - I raise the deployment replicas to 1, and everything works as expected.
Is this a bug? Or am I missing anything.
I've looked at all the other topics about this, all the solutions are "Oh I missed some tag" - I've checked all the tags again and again, it's not the issue.
Any help would be great.
Versions:

Chart Version: 1.0.8
Kubernetes Version (kubectl version): 1.31.3 (EKS)

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

I was able to fix this, I'll elaborate on what I did:

I've created a different nodegroup (And didn't use the old one that my Cluster autoscaler used)
I updated the AMI - Instead of using the one my ASG used, I'm using the latest:

Ami Selector Terms:
  Name:  amazon-eks-node-1.31-*
# Instead of the id: ami-00710ab8f493e2428

Removed the old snapshot used, and just let Karpenter generate one (GP3), and removed the encryption of the EBS
Updated the aws-auth:

Removed the 'KarpenterControllerRole-' iam identity mapping (It's not required there, I don't know why I've added it in the first place)
Deleted and re-added my 'KarpenterControllerRole-' iam identity mapping - I think I was missing the line 'username: system:node:{{EC2PrivateDNSName}}'

TBH, I'm not so sure what caused it to work, it might be related to more than 1 thing.

@TSjhkim sure, this is my updated EC2NodeClass (Created via Terraform):
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: karpenter
  namespace: kube-system
spec:
  amiFamily: AL2  # Amazon Linux 2
  role: "KarpenterNodeRole-${CLUSTER_NAME}"  # replace with your cluster name
  metadataOptions:
    httpPutResponseHopLimit: 2 # Enable IMDSv2
  subnetSelectorTerms:  # Adjust this to your subnets tags
    - tags:
        Environment: test
        Tier: private
  securityGroupSelectorTerms:  # Adjust this to your SG tags
    - tags:
        "aws:eks:cluster-name": ${CLUSTER_NAME}
  amiSelectorTerms:
    - name: amazon-eks-node-1.31-*
  blockDeviceMappings:
    - deviceName: /dev/xvda # Not giving it a snapshot will cause it to just create a default EBS
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        iops: 10000
        encrypted: false  # Disabling encryption to our EBS
        deleteOnTermination: true
  tags: # Tags to add to all newly created ec2's
    env: test
    Name: ${CLUSTER_NAME}-karpenter

I've added comments in the YAML


Escape Quotes in Cells

When encoding or serializing data to CSV format, we don't escape double-quote characters ("), which will causes the data to be parsed incorrectly. We need to escape the quote characters in the cell contents by adding another double-quote before it:
"Exactly!" He replied

Becomes this:
""Exactly!"" He replied

@calebkleveter I've been reviewing what I understand to be the spec for CSV (RFC 4180). It looks like if you want to include quotes anywhere in what we consider a cell then you are supposed to wrap the whole cell in quotes as well. See Sections 2.6 and 2.7. Do the changes to the encoder and decoder/parser in #9 account for this?
I was looking at tackling this issue along side the config work I was working on, but if you've got a good approach for this then that's even better!

The parser and decoder handle escaped quotes in all contexts, even if the cell is not wrapped in double quotes. The serializer and encoder automatically surround all cells in quotes already. Once the PR for this issue is merged, all cases for double quotes should be handled.

@calebkleveter That's great!
So it would handle all of these examples?
full name,quote type
the hulk,no quotes
"aqua man","wrapped in quotes"
"""iron man","""wrapped and leading quote"
"spyder man""","wrapped and trailing quote"""
"bat""man","wrapped and""inserted quote"

Yes, it would. If you parsed that into a dictionary, it would look like this:
[
    "full name": ["the hulk", "aqua man", "\"iron man", "spyder man\"", "bat\"man"],
    "quote types": ["no quotes", "wrapped in quotes", "\"wrapped and leading quotes", "wrapped and trailing quotes\"", "wrapped and\"inserted quote"]
]


[HOTFIX][SPARK-14402] Fix ExpressionDescription annotation

What changes were proposed in this pull request?
Fix for the error introduced in https://github.com/apache/spark/commit/c59abad052b7beec4ef550049413e95578e545be:
/Users/jacek/dev/oss/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:626: error: annotation argument needs to be a constant; found: "_FUNC_(str) - ".+("Returns str, with the first letter of each word in uppercase, all other letters in ").+("lowercase. Words are delimited by white space.")
    "Returns str, with the first letter of each word in uppercase, all other letters in " +
                                                                                ^

How was this patch tested?
Local build

Can one of the admins verify this patch?

ok to test

Test build #55069 has started for PR 12192 at commit 218bf51.

Test build #55069 has finished for PR 12192 at commit 218bf51.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55069/
Test FAILed.

Merged build finished. Test FAILed.

Is this really the patch to cause the issue or is this more an interim hiccup?

@tdas another streaming failure

Hi, @jaceklaskowski and @marmbrus .
Thank you for fixing this. I didn't notice there is such a problem.
Sorry for late response!
I have another PR having the same syntax. I will fix that, too.
Thank you again.

Jenkins retest this please

Test build #55207 has started for PR 12192 at commit 218bf51.

Test build #55207 has finished for PR 12192 at commit 218bf51.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55207/
Test PASSed.

Hi, All.
The original code is made by me, so I feel guilty. :)
I think Jenkins build does not show additional information for this PR since our Jenkins build on master branches works fine.

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/  (JDK8 + Scala 2.11)
https://amplab.cs.berkeley.edu/jenkins/view/Spark QA Compile/job/spark-master-compile-maven-scala-2.10/ (JDK8 + Scala 2.10).

And, we know the JDK7 does not have this issue according to the Spark mailing list. So, I ran the following command and found the current master branch compiles without problem. This seems not a Scala compilation problem.
spark:master$ git clean -fdx
spark:master$ java -version
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
spark:master$ ./build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.2 -Phive -Phive-thriftserver -DskipTests clean compile
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:38 min
[INFO] Finished at: 2016-04-07T10:29:20-07:00
[INFO] Final Memory: 76M/871M
[INFO] ------------------------------------------------------------------------

After that, just package also works fine.
spark:master$ ./build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.2 -Phive -Phive-thriftserver -DskipTests package
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:02 min
[INFO] Finished at: 2016-04-07T10:50:44-07:00
[INFO] Final Memory: 79M/853M
[INFO] ------------------------------------------------------------------------

The real root cause of this error occurs during Scala doc generation by doc-jar plugin. Is it possible to solve the real root cause by changing pom.xml to reorder the plugin execution stages or something else? I've already changed my another PR #12185 (DESC FUNCTION implementation)  according this PR's approach, but it's a little too limited approach for me.

Just for a record, here is the current error message from doc-jar. (The version of doc-jar is the latest.)
[INFO] <<< scala-maven-plugin:3.2.2:doc-jar (attach-scaladocs) < generate-sources @ spark-catalyst_2.11 <<<
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:doc-jar (attach-scaladocs) @ spark-catalyst_2.11 ---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=64m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
/Users/dongjoon/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:626: error: annotation argument needs to be a constant; found: "_FUNC_(str) - ".+("Returns str, with the first letter of each word in uppercase, all other letters in ").+("lowercase. Words are delimited by white space.")
    "Returns str, with the first letter of each word in uppercase, all other letters in " +
                                                                                ^

I agree it's a bit of a strange problem. @dongjoon-hyun do you agree that this change is worth merging? it seems like it fixes the immediate symptom of a problem, even if we're not sure why the problem exists in the first place. It's not hacky so seems worth merging.

Sure. I agree for merging.
The underlying problem will be investigated and resolved someday.

Please please merge it as soon as possible as I'm suffering from not having it in master every time I do the build :(

Merged to master


Add ids from ElasticAPM if present

ECS tracing fields: https://www.elastic.co/guide/en/ecs/current/ecs-tracing.html

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #14 opened

Start Time: 2021-01-15T12:28:18.119+0000

Duration: 12 min 31 sec

Commit: 39f843cd7ac3f4977fc255a7f24f52db7c6c9943

Test stats :test_tube:

Test
Results

Failed
0

Passed
216

Skipped
0

Total
216

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #14 opened

Start Time: 2021-01-15T12:28:18.119+0000

Duration: 12 min 31 sec

Commit: 39f843cd7ac3f4977fc255a7f24f52db7c6c9943

Test stats :test_tube:

Test
Results

Failed
0

Passed
216

Skipped
0

Total
216


Error in compare two faces

Hi, i tried to compare two faces with compare2faces in #11, but i got this error :

hi, faces must be a pytorch tensor


Add manual confirmation for releases

Today the release process involves tagging and pushing a commit to main, the rest is taken care of by automation. This level of automation has been helpful, however, it would be ideal to add a manual confirmation that can only be acted on by a select few repo admins.
Since we're in the midst of the release pipeline refactor in #310 it would be ideal to get this work tested under that PR.

The GHA workflow has been updated so that toolbox team members must approve a release


Dependency upgrade action

Description
This PR implements a custom action to upgrade the dependencies and open a PR using pip-compile directly instead of dependabot's pip option. This respects the restrictions in pyproject.toml and ensure the secondary dependencies are consistent with the primary ones.
I tested this by running it on push, which can be seen in , and resulted in PR #44
Limitations:

The CI workflows do not run in the resultant PR, to fix this we should use the RSE CI bot (see these docs)
If we want these to auto-merge, we will need to add the appropriate PR user to the auto-merge.yml workflow

Fixes #29
Type of change

[ ] Documentation (non-breaking change that adds or improves the documentation)
[ ] New feature (non-breaking change which adds functionality)
[ ] Optimization (non-breaking, back-end change that speeds up the code)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Breaking change (whatever its nature)

Key checklist

[ ] All tests pass (eg. python -m pytest)
[ ] The documentation builds and looks OK (eg. python -m sphinx -b html docs docs/build)
[ ] Pre-commit hooks run successfully (eg. pre-commit run --all-files)

Further checks

[ ] Code is commented, particularly in hard-to-understand areas
[ ] Tests added or an issue has been opened to tackle that in the future. (Indicate issue here: # (issue))

A question: is it possible that an update by a Dependabot results into some failing checks, because something existing becomes incompatible with the bot update?

Yes, exactly. For example, #8 is failing because pip can't resolve the dependencies when installing them. This is the error:
ERROR: Cannot install -r dev-requirements.txt (line 45) and idna==3.10 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested idna==3.10
    coldfront 1.1.6 depends on idna==3.6

The other one: Will the PR created by this action be automatically merged or you need to merge it manually? - Ok I just saw PR#44 and it is not auto-merged. So that answers.

It will require a little additional work to make the resultant PR auto-merge because GitHub has a safety measure to prevent workflows being triggered by other workflows (to prevent infinitely calling calling workflows)

A question: is it possible that an update by a Dependabot results into some failing checks, because something existing becomes incompatible with the bot update?

Yes, exactly. For example, #8 is failing because pip can't resolve the dependencies when installing them. This is the error:

Ok. On PR#8, I can see it says Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself - It is fascinating to know that it can sense if there are any conflicts in the PR and it also tries to resolve them!

Also was Renovate a non-starter?

@cc-a Do you not think we need to make the changes to make the CI workflows run in the resultant PR?


Optimistically query and replace files atomically

Fixes #316
Hi, I'm trying to learn about iceberg.
One option to perform the copy-on-write/eager updates is to provide the queryTime snapshot to the ReplaceFiles SnapshotProducer. Then if the snapshot is not the queryTime snapshot, fail the commit.
I'm hoping this doesn't get retried because it is an IllegalStateException and not a CommitFailedException.
excerpt from base SnapshotProducer
      Tasks.foreach(ops)
          .retry(base.propertyAsInt(COMMIT_NUM_RETRIES, COMMIT_NUM_RETRIES_DEFAULT))
          .exponentialBackoff(
              base.propertyAsInt(COMMIT_MIN_RETRY_WAIT_MS, COMMIT_MIN_RETRY_WAIT_MS_DEFAULT),
              base.propertyAsInt(COMMIT_MAX_RETRY_WAIT_MS, COMMIT_MAX_RETRY_WAIT_MS_DEFAULT),
              base.propertyAsInt(COMMIT_TOTAL_RETRY_TIME_MS, COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT),
              2.0 /* exponential */)
          .onlyRetryOn(CommitFailedException.class)
          .run(taskOps -> {

Thanks for working on this, @johnclara!
We already have a solution internally that we were about to share as well. We decided to go with a new API instead of extending RewriteFiles because it uses DataOperations.REPLACE, meaning the assumption is that files are removed and replaced without changing the data. Therefore, we created ModifyFiles with DataOperations.MODIFY.
We are accepting baseSnapshotId in the constructor instead of accepting it in the method. That way, it is transparent to the user and we can still track new files during retries. Also, we are accepting a row filter to do conflict resolution. Setting that row filter to alwaysTrue will be equivalent to what you have (i.e. retries will always fail). On top, that row filter allows us to do more fine-grained conflict resolution. For now, we limit the scope to partitions, meaning that if you are modifying files in one partition but a concurrent operation commits a new file to another partition, we can still commit successfully. Later, we will extend this logic beyond partitions by leveraging column stats and InclusiveMetricsEvaluator to detect conflicts.
Let me share what I have so that we can discuss it in detail.

Sounds good! Looking forward to seeing what you all have :)


To Do #2

Future Steps:

Reading papers that conducted transaction-based research on the whole Ethereum blockchain to check how they did it. What are the requirements of the full node I/O we need (these papers) :

https://arxiv.org/abs/1812.05934
https://arxiv.org/abs/2101.06204
https://www.usenix.org/conference/usenixsecurity21/presentation/perez
https://www.usenix.org/conference/usenixsecurity20/presentation/zhang-mengya
https://arxiv.org/abs/2005.08278
https://www4.comp.polyu.edu.hk/~csxluo/SODA.pdf

Adding beacon proxies

Try to find a way to detect the admin of upgradeable proxies and then check if the admin is an EOA, Multi-Sig, or governance token (number of governors of the system: EOA is one. Multi-sig is easy to understand. For governance tokens, we need 2 things: 1.  Distribution and allocation of the token to different addresses  2. Threshold of the governance token).

We can find the number of admin changes on the proxies we found.

We can find the number of upgrades that happened to a proxy contract. We can also check for the amount of upgrade each time happened to the proxy contract by using a code difference tool and comparing the old implementation contract with the newer version

Finding a way to detect Call-based upgradeable contracts

Other patterns are not recognizable ( I have my reasons)

Reading about Diamonds and Create2 upgradeability methods

Add the description
Update the evaluation table
Check if there is a way to find contracts  that use this method

Running a full node maybe!

Add differences between delegate call and call based approaches on the Devs' view (how much they should change their code)

We can do research on the plausibility as well because it's important for upgrade (for measurement study part)

Sereum: Protecting Existing Smart Contracts Against Re-Entrancy Attacks

They ran a full archival node and as they said they had a reasonable I/O
They have used a forked version of Geth Client (they add some modules to it)
Their model is a hybrid analysis similar to ours (Transaction trace + Bytecode Analysis)
Our model need a decompiler in addition
They re-executed 78 million transactions involved in 4.5 million Blocks
They randomly sampled from their outcomes and check them manually like what we did

The Eye of Horus: Spotting and Analyzing Attacks on Ethereum Smart Contracts

They ran a full archival Geth client
The used debug_traceTransaction method
They changed the Tracer module of the Geth to just have their desired data on the output of transaction traces (for example they removed stack counter,memory for every instruction, etc.)
They used the Ethereum ETL framework to retrieve a list of transactions for every smart contract deployed up to block 10 M and collected a total of 697,373,206 transactions and 3,362,876 contracts.
They filtered contracts without transactions and removed transactions that have a gas limit of 21,000 (do not execute code). Also they skipped all the transactions that were part of the 2016 denial-ofservice attacks, as these incur high execution times
Their finaal dataset is 1,234,197 smart contracts consisting of 371,419,070 transactions
Their Setup:  64 GB of memory and an Intel(R) Core(TM) i7-8700 CPU with 12 cores clocked at 3.2 GHz, running 64-bit Ubuntu 18.04.5 LTS. (used Geth version 1.9.9)

Smart Contract Vulnerabilities: Vulnerable Does Not Imply Exploited

They patched Geth client to speed up getting transaction traces

I think they changed the tracer module of Geth to have their just their desired output

maybe a helpful link on stackoverflow: https://ethereum.stackexchange.com/questions/40191/how-to-install-a-custom-tracer-in-geth/40465

tracer module: https://github.com/ethereum/go-ethereum/tree/master/eth/tracers

check how Horus changed the trace module

they used data set of other papers which is not related to our topic

TXSPECTOR: Uncovering Attacks in Ethereum from Transactions

like other researches, they modified the Geth client to just have desired traces for transactions.

unlike others, they modified EVM  instead of trace module to log their desired field of transactions and op-codes

" The traces were collected on an L8s v2 instance on the Microsoft Azure Cloud, with 8 VCPUs, 64GB RAM and 2TB SSD, running Ubuntu 18.04. Trace Extractor ran a full Ethereum node to collect bytecode-level traces, from the 0-t h block to the 7, 200, 000-t h block. Note that Ethereum has around 10180000 number of blocks (as in June 1st 2020) and it keeps growing exponentially. We cannot collect all of them for our experiment because it takes a huge amount of storage and also processing time. We therefore stop collecting at 7,200,000-th block, which has resulted in the size of 1,577 GB, containing 397,269,533 transactions in total. We store them in a Trace DB (implemented atop MongoDB) by Trace Extractor.

They said that based on their logic it takes 1 second for each transaction to be processed!!

SODA: A Generic Online Detection Framework for Smart Contracts

they modified the Geth client to retrieve their desired data

They have collected information of the internal transactions by adding code into the interpretation handler of the EVM instructions (CALL, CALLCODE,Create, Delegatecall, staticcall, self destruct) that can create internal transactions

I'm not sure if we can collect the proxies just by Geth's EVM modification because we need two consecutive instructions.

Relate the upgrade to the attacks or anything like this


add option to send notes to selected relays

Super cool! This Dialog is getting cluttered, though. Let's keep pushing!


Updating to ASP.NET 5 Beta 8

Updating website to work with beta 8 version of ASP.NET.  Also updated deployment scripts to match the latest templates for resource deployment.

LGTM


TOSOT AC support

Hi.
Can you support the TOSOT air conditioners as well? It is nearly the same as the Gree, the app name is TOSOT+, but it looks like ewpe. If you need any help for testing, info, debug etc, I'm ready to do it!
Thank you!

My Tosot airco works like a charm now. Small issue in de bridge software, see the binding issue there, with the solution.

Sounds good! Can you tell me what was the issue, and where is the solution?

Please see the issue regarding the binding in the ewpe bridge software as I already mentioned. Same developer.

Ohh, I see, I found now! Thanks

Now it should be fine as the original issue is closed


Go to definition stuck and taking up processor resource

Issue Type: Bug
Right click and select "Go to Definition". Sometimes VSCode gets stuck looking for the definition and doesn't find it somehow but it keeps trying, taking up a significant amount of processing resource. Laptop's fan starts going crazy, and there's no way to stop it unless I close vscode and reopen.
Metadata:
macOS mojave (was happening high sierra too)
javascript
react.
VS Code version: Code 1.27.2 (f46c4c469d6e6d8c46f268d1553c5dc4b475840f, 2018-09-12T07:04:11.329Z)
OS version: Darwin x64 18.0.0

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-4278U CPU @ 2.60GHz (4 x 2600)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: enabledrasterization: enabledvideo_decode: enabledvideo_encode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
7, 7, 7

Memory (System)
8.00GB (0.05GB free)

Process Argv
/Applications/Visual Studio Code.app/Contents/MacOS/Electron

Screen Reader
no

VM
0%

Extensions (5)

Extension
Author (truncated)
Version

php-intellisense
fel
2.3.5

auto-close-tag
for
0.5.6

ftp-simple
hum
0.6.7

highlight-matching-tag
vin
0.4.4

html-css-class-completion
Zig
1.17.1

Does this reproduce in the latest VS Code insiders build with all extensions disabled?


Add download and usage info to README

@thso/@chalin does this look OK? It's been a long time since I've used dart_doc_syncer...

LGTM


Module not found errors when using vite/vinxi and might-fail

Steps to reproduce:

Start a new solid start project: bun create solid
Select solid start, basic template, use typescript
Import any might fail functions and use in any tsx file

I remember running into this issue in tanstack start as well, which also uses vinxi
I'm unsure if this is a problem with vinxi or with might-fail, but considering no other packages I install give me this issue, I assume
it's the latter

Aparently @iliaamiri has fixed this issue because he's awesome. I'll review the pr and run some tests. In the mean time, jsr doesn't have these issues, so you can use the jsr version:
bunx jsr add @might/fail

import { mightFail } from "@might/fail"

In fact, it might be worth always using the jsr version with bun and deno projects since there's no extra ts compiling as there is with node.

@Vakiyama This is fixed in version https://github.com/meech-ward/might-fail/releases/tag/0.6.3

Thank you Sam!