[BUG] Re-creating the CC GKE cluster after deploying the landing-zone solution requires removal of the requireShieldedVM org policy - kubernetes intent engine will replace it back post-create

see https://github.com/ssc-spc-ccoe-cei/gcp-tools/issues/32
Describe the bug
Re-creating the CC GKE cluster after deploying the landing-zone solution requires removal of the requireShieldedVM org policy - kubernetes intent engine will replace it back post-create
After deploying the LZ successfully - deleted the CC cluster, attempt CC re-create - hitting shieldedVM org policy
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/landing-zone/environments/common/policies/org-policies.yaml#L84
Existing deployment OK

To Reproduce
Steps to reproduce the behavior:
1 - Took a clean organization, installed the CC cluster, deployed the landing-zone solution
2 - deleted the CC cluster
3 - attempt recreation of the CC cluster
4 - run into shieldedVM org policy (previously set by the landing-zone solution)
Expected behavior
Deleting the cluster obviously retains org policies - it seems that the landing-zone will disallow a default CC cluster install - which from a security perspective is normal
So the issue is that the org policy must be removed before attempting to deploy the CC cluster again.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Fix is to apply a false flag for the org policy before creating the CC cluster - then the lz solution will return the policy once gke is back up
michael@cloudshell:~ (landing-zone-controller-e4g7d)$ gcloud anthos config controller create landing-zone-controller --location northamerica-northeast1 --network kcc-controller --subnet kcc-regional-subnet
Create request issued for: [landing-zone-controller]
Waiting for operation [projects/landing-zone-controller-e4g7d/locations/northamerica-northeast1/operations/operation-1663188232198-5e8a92fc658f2-a614c3a2-993952f2] to complete...failed.
ERROR: (gcloud.anthos.config.controller.create) unexpected error occurred while waiting for SLM operation [projects/krmapihosting-slm/locations/northamerica-northeast1/operations/operation-1663188237478-5e8a93016ea42-84c2a403-f292d464]: errored while waiting for operation: projects/krmapihosting-slm/locations/northamerica-northeast1/operations/operation-1663188237478-5e8a93016ea42-84c2a403-f292d464: Operation failed with error:
generic::invalid_argument: terraform apply failed, error: exit status 1, stderr:
2022/09/14 13:44:18 [DEBUG] Using modified User-Agent: Terraform/0.12.31 Cloud SSA

Error: Error waiting for creating GKE cluster:
        (1) Not all instances running in IGM after 17.043475418s. Expected 1, running 0, transitioning 1. Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-landing-z-default-pool-dbc4594b-xn0b' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/landing-zone-controller-e4g7d. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information
        (2) Not all instances running in IGM after 18.435053483s. Expected 1, running 0, transitioning 1. Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-landing-z-default-pool-9a70dc96-s8dd' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/landing-zone-controller-e4g7d. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information
        (3) Not all instances running in IGM after 25.990797918s. Expected 1, running 0, transitioning 1. Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-landing-z-default-pool-a012fae7-q2vd' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/landing-zone-controller-e4g7d. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information.

  on main_gke.tf line 32, in resource "google_container_cluster" "acp_cluster":
  32: resource "google_container_cluster" "acp_cluster" {

, stdout:
google_container_cluster.acp_cluster: Creating...
google_container_cluster.acp_cluster: Still creating... [10s elapsed]
google_container_cluster.acp_cluster: Still creating... [20s elapsed]

delete the org policy first - as we are now governed by the landing-zone policies - before bringing the anthos cluster back up
I'll write up a procedure tomorrow on
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/architecture.md#deleting-the-anthos-cluster

Verifying managed org + arete
set the shieldedVM org policy on the folder above the project
dmin_@cloudshell:~ (pubsec-declarative-toolkit-arg)$ arete create pdt-arg-kcc --region=us-east1 --project=pubsec-declarative-toolkit-arg
12:05PM INF Creating Config Controller Cluster....
12:07PM FTL  error="Create request issued for: [pdt-arg-kcc]Waiting for operation [projects/pubsec-declarative-toolkit-arg/locations/us-east1/operations/operation-1663243533000-5e8b60ff59e80-4465a0a6-1a7e94d9] to complete..................................................................................................................failed.ERROR: (gcloud.anthos.config.controller.create) unexpected error occurred while waiting for SLM operation [projects/krmapihosting-slm/locations/us-east1/operations/operation-1663243538361-5e8b610476d35-b492d7e8-0ef90d34]: Subnetwork \"projects/pubsec-declarative-toolkit-arg/regions/us-east1/subnetworks/kcc-regional-subnet\" is not valid for Network \"kcc-controller\"."
admin_@cloudshell:~ (pubsec-declarative-toolkit-arg)$

delete the vpc first
dmin_@cloudshell:~ (pubsec-declarative-toolkit-arg)$ arete create pdt-arg-kcc --region=us-east1 --project=pubsec-declarative-toolkit-arg
12:12PM INF Creating Config Controller Cluster....
12:14PM FTL  error="Create request issued for: [pdt-arg-kcc]Waiting for operation [projects/pubsec-declarative-toolkit-arg/locations/us-east1/operations/operation-1663243944384-5e8b6287ad75d-42a3c722-5e3a7f66] to complete....................................................................................................................failed.ERROR: (gcloud.anthos.config.controller.create) unexpected error occurred while waiting for SLM operation [projects/krmapihosting-slm/locations/us-east1/operations/operation-1663243949570-5e8b628c9f84e-d1c7f96f-bd84e4a5]: Network \"kcc-controller\" does not exist."

recreating the network, subnet, cluster via advanced install in this case

Alternatively instead of deleting the Org Policy you could add a new policy to exempt the Config Controller project.
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
 kind: ResourceManagerPolicy
 metadata: # kpt-merge: config-control/require-shielded-vm
   name: exempt-require-shielded-vm
   namespace: config-control # kpt-set: ${management-namespace}
   labels:
     guardrail: 'false'
   annotations:
     cnrm.cloud.google.com/blueprint: kpt-pkg-fn
 spec:
   constraint: constraints/compute.requireShieldedVm
   booleanPolicy:
     enforced: false
   projectRef:
     name: project-id
``

A good option will prototype it.
For now the option to pre-delete the policy on the folder or project will handle the case where the org has the shieldedVM policy already in place (security managed orgs) - before we even run the landing-zone.
We can rely on the kubernetes intent engine to re-instate the policy once the GKE cluster comes back up "to be verified - but it should work"
I like the idea of doing it once in the yaml though - will prototype

Regression testing 2nd ol.dev cluster with policy off - to get around pre-verified
michael@cloudshell:~ (landing-zone-controller-1z583)$ gcloud anthos config controller create landing-zone-controller --location northamerica-northeast1 --network kcc-controller --subnet kcc-regional-subnet
Create request issued for: [landing-zone-controller]
Waiting for operation [projects/landing-zone-controller-1z583/locations/northamerica-northeast1/operations/operation-1663253856902-5e8b8774fde41-143290de-ba33c54f] to complete...failed.
ERROR: (gcloud.anthos.config.controller.create) unexpected error occurred while waiting for SLM operation [projects/krmapihosting-slm/locations/northamerica-northeast1/operations/operation-1663253862363-5e8b877a32fe2-4b3b7ed6-c4e25ead]: errored while waiting for operation: projects/krmapihosting-slm/locations/northamerica-northeast1/operations/operation-1663253862363-5e8b877a32fe2-4b3b7ed6-c4e25ead: Operation failed with error:
Error: Error waiting for creating GKE cluster:
        (1) Not all instances running in IGM after 5m48.550577091s. Expected 1, running 0, transitioning 1. Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-landing-z-default-pool-2c30d04a-7rnq' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/landing-zone-controller-1z583. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information

wait for cluster to delete
rerun
gcloud anthos config controller create landing-zone-controller --location northamerica-northeast1 --network kcc-controller --subnet kcc-regional-subnet

Issue on 2nd org is I had an existing 2nd cc cluster - the org policy was getting reset to true right after I reset it  - as expected

delete the older 2nd cluster first before recreating the 1st
michael@cloudshell:~ (lz-20220910-oldev)$ gcloud anthos config controller delete pdt-oldev --location=northamerica-northeast1
You are about to delete instance [pdt-oldev]

Do you want to continue (Y/n)?  y

Delete request issued for: [pdt-oldev]
Waiting for operation [projects/lz-20220910-oldev/locations/northamerica-northeast1/operations/operation-1663257217373-5e8b93f9c9490-0c43d609-fb7c1ccd] to complete...working

Potential solution to this is when deploying the LZ we bundle some custom org policies to exempt the CC project. IE compute.requireShieldedVm, compute.vmCanIpForward, and compute.restrictVpcPeering

self notes
Use case options: GKE (clean/with-cc), region, org policies, VPC/subnet (default/custom)
use cases (TB selected)

UC00: clean GKE, region 1, no org policies, vpc default - verify
UC01: clean GKE, region 1, no org policies, vpc custom
UC02: clean GKE, region 1, org polices, vpc default
UC03: clean GKE, region 1, org policies, vpc custom
UC04: clean GKE, region 2, no org policies, vpc default = OK
UC05: clean GKE, region 2, no org policies, vpc custom
UC06: clean GKE, region 2, org polices, vpc default
UC07: clean GKE, region 2, org policies, vpc custom
UC08: GKE with CC, region 1, no org policies, vpc default
UC09: GKE with CC, region 1, no org policies, vpc custom
UC0a: GKE with CC, region 1, org polices, vpc default
UC0b: GKE with CC, region 1, org policies, vpc custom - current
UC0c: GKE with CC, region 2, no org policies, vpc default - verify
UC0d: GKE with CC, region 2, no org policies, vpc custom
UC0e: GKE with CC, region 2, org polices, vpc default
UC0f: GKE with CC, region 2, org policies, vpc custom - current

Region 1 - JIRA issued (VPC custom or VPC default)
Region 2 -

michael@cloudshell:~ (landing-zone-controller-1z583)$ gcloud anthos config controller create landing-zone-controller6 --location northamerica-northeast2 --network default --subnet default
Create request issued for: [landing-zone-controller6]

247385736

Update docs to override both org policies affecting CC create - at the least priv project level
Do the same for arete - here adjust the advanced install docs
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/docs/advanced-install.md#advanced-install
gcloud anthos config controller create $CLUSTER --location $REGION --network $NETWORK --subnet $SUBNET
        (3) Not all instances running in IGM after 32.489803877s. Expected 1, running 0, transitioning 1. Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-pdt-default-pool-7f80211c-6j85' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/controller-arg-0919. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information.

PR testing
Attempt to start the CC server - after a previous LZ deployment
Cluster fails to come up (in this case we are using na2) due to the requireShieldedVm org policy
Fix is to override the org policy only at the cc project level (the project running the cc cluster as detailed in the advanced install - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/advanced-install.md
Before:
gcloud anthos config controller create $CLUSTER --location $REGION --network $NETWORK --subnet $SUBNET
        (3) Not all instances running in IGM after 32.489803877s. Expected 1, running 0, transitioning 1. Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-pdt-default-pool-7f80211c-6j85' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/controller-arg-0919. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information.

turn off the org policy

After:

admin_@cloudshell:~/wse_github/20220829/pubsec-declarative-toolkit (controller-arg-0919)$ gcloud anthos config controller create $CLUSTER --location $REGION --network $NETWORK --subnet $SUBNET
Create request issued for: [pdt]
Waiting for operation [projects/controller-arg-0919/locations/northamerica-northeast2/operations/operation-1663614985004-5e90c8c39673f-15d8d376-d129e48d] to complete...working..
Waiting for operation [projects/controller-arg-0919/locations/northamerica-northeast2/operations/operation-1663614985004-5e90c8c39673f-15d8d376-d129e48d] to complete...done.
Created instance [pdt].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-pdt.

Testing managed arg... na1 after a previously deleted na2

admin_@cloudshell:~/wse_github/20220829/pubsec-declarative-toolkit (controller-arg-0919)$ export REGION=northamerica-northeast1
admin_@cloudshell:~/wse_github/20220829/pubsec-declarative-toolkit (controller-arg-0919)$ export SUBNET=pdt1
admin_@cloudshell:~/wse_github/20220829/pubsec-declarative-toolkit (controller-arg-0919)$ export NETWORK=pdt1
admin_@cloudshell:~/wse_github/20220829/pubsec-declarative-toolkit (controller-arg-0919)$ gcloud compute networks create $NETWORK --subnet-mode=custom
Created [https://www.googleapis.com/compute/v1/projects/controller-arg-0919/global/networks/pdt1].
NAME: pdt1
SUBNET_MODE: CUSTOM
BGP_ROUTING_MODE: REGIONAL
IPV4_RANGE:
GATEWAY_IPV4:
Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network pdt1 --allow tcp,udp,icmp --source-ranges <IP_RANGE>
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network pdt1 --allow tcp:22,tcp:3389,icmp
admin_@cloudshell:~/wse_github/20220829/pubsec-declarative-toolkit (controller-arg-0919)$ gcloud compute networks subnets create $SUBNET  --network $NETWORK --range <IP_ADDRESS>/16 --region $REGION
Created [https://www.googleapis.com/compute/v1/projects/controller-arg-0919/regions/northamerica-northeast1/subnetworks/pdt1].
NAME: pdt1
REGION: northamerica-northeast1
NETWORK: pdt1
RANGE: <IP_ADDRESS>/16
STACK_TYPE: IPV4_ONLY
IPV6_ACCESS_TYPE:
INTERNAL_IPV6_PREFIX:
EXTERNAL_IPV6_PREFIX:
admin_@cloudshell:~/wse_github/20220829/pubsec-declarative-toolkit (controller-arg-0919)$ gcloud anthos config controller create $CLUSTER --location $REGION --network $NETWORK --subnet $SUBNETCreate request issued for: [pdt]
Waiting for operation [projects/controller-arg-0919/locations/northamerica-northeast1/operations/operation-1663619025979-5e90d7d15c868-7017e151-bd630fe9] to complete...working...

Testing on completely clean org eventfield.io
using
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/docs/advanced-install.md
Welcome to Cloud Shell! Type "help" to get started.
To set your Cloud Platform project in this session use “gcloud config set project [PROJECT_ID]”
michael@cloudshell:~$ gcloud config set project pubsec-declarative-tk-ei
Updated property [core/project].
michael@cloudshell:~ (pubsec-declarative-tk-ei)$ history
    1  gcloud config set project pubsec-declarative-tk-ei
    2  history
michael@cloudshell:~ (pubsec-declarative-tk-ei)$ git clone https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git
Cloning into 'pubsec-declarative-toolkit'...
remote: Enumerating objects: 4542, done.
remote: Counting objects: 100% (38/38), done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 4542 (delta 18), reused 6 (delta 1), pack-reused 4504
Receiving objects: 100% (4542/4542), 1.74 MiB | 15.62 MiB/s, done.
Resolving deltas: 100% (2800/2800), done.
michael@cloudshell:~ (pubsec-declarative-tk-ei)$ cd pubsec-declarative-toolkit/
cli/       docs/      examples/  .git/      .github/   services/  solutions/

cartyc commented 7 minutes ago
@fmichaelobrien instead of documenting this I think we should have a default org policy as part of the package that adds the exclusion to the Config Controller host project. What do you think?
sounds good - I would rather an automated approach with special handling for the CC project,  I'll look at testing via a boolean yaml first
gcloud beta resource-manager org-policies set-policy --organization<PHONE_NUMBER>45 policy.yaml

PR #138 and PR #139 are recommendations for NA2 - we still have not narrowed down the issue with NA1 on a re-create of the CC cluster (beyond what we fixed for billing, requireOsLogin and requireShieldedVms
The use case is create the CC in NA1 - deploy the LZ, delete the CC, Create the CC.  NA1 works until we deploy the LZ, once I get the LZ up on NA2 we can validate the CC recreate and LZ update on NA2

Review also https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/setup-kcc.sh#L153

for #446
remove also constraints/compute.requireShieldedVm

Actually for 1 of the 2 - the historical Shielded - we are good with the following override
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-9428)$ kubectl get gcp -n policies
NAME                                                                                AGE     READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project   9m36s   True    UpToDate   9m8s


Relying on Homebrew configuration paths instead of /opt/boxen/config

Whilst trying to update nginx in boxen/puppet-nginx#51, I've found that the convention of putting configuration files into /opt/boxen/config is causing it to fail on newer versions of OSX (whereas it works fine on older installations).
==> ./configure --prefix=/usr/local/Cellar/nginx/1.10.2-boxen1 --with-http_ssl_module --with-pcre --with-ipv6 --with-cc-opt=-I/usr/local/opt/pcre/include -I/usr/local/opt/openssl/include --with-ld-opt=-L/usr/local/opt/pcre/lib -L/usr/local/opt/openssl/lib --conf-path=/opt/boxen/config/nginx/nginx.conf --pid-path=/opt/boxen/data/nginx/nginx.pid --lock-path=/opt/boxen/data/nginx/nginx.lock --with-http_v2_module
==> make
==> make install
Last 15 lines from /Users/dtbaker/Library/Logs/Homebrew/nginx/03.make:

make -f objs/Makefile install
test -d '/usr/local/Cellar/nginx/1.10.2-boxen1' || mkdir -p '/usr/local/Cellar/nginx/1.10.2-boxen1'
test -d '/usr/local/Cellar/nginx/1.10.2-boxen1/sbin' \
		|| mkdir -p '/usr/local/Cellar/nginx/1.10.2-boxen1/sbin'
test ! -f '/usr/local/Cellar/nginx/1.10.2-boxen1/sbin/nginx' \
		|| mv '/usr/local/Cellar/nginx/1.10.2-boxen1/sbin/nginx' \
			'/usr/local/Cellar/nginx/1.10.2-boxen1/sbin/nginx.old'
cp objs/nginx '/usr/local/Cellar/nginx/1.10.2-boxen1/sbin/nginx'
test -d '/opt/boxen/config/nginx' \
		|| mkdir -p '/opt/boxen/config/nginx'
cp conf/koi-win '/opt/boxen/config/nginx'
cp: /opt/boxen/config/nginx/koi-win: Operation not permitted
make[1]: *** [install] Error 1
make: *** [install] Error 2

Swapping the --conf flag to use /usr/local/etc (like upstream Homebrew does) allows the installation to work as expected.
I'm wondering if we want to keep the /opt/boxen/config convention or whether we can move the Boxen puppet modules to instead manage the file under /usr/local/etc.
@MikeMcQuaid do you have any context on whether this move would cause issues in Homebrew?

From a historical perspective, we always wanted to keep boxen-managed things in /opt/boxen to improve discoverability and to aid in nuking. However, I think that line's been blurred for a long time now, and this proposal doesn't sound unreasonable.
There are a couple of non-trivial things to consider, though:

script/nuke would now need to know to clear out some configs from /usr/local/etc
inconsistency in where configs are stored will hurt; you'll want to go all in, and update all core modules at once.

we always wanted to keep boxen-managed things in /opt/boxen to improve discoverability

This was my primary concern. If we started allowing these configuration files in shared locations it's going to be very difficult to track which system is managing the file and I'm thinking it will introduce race conditions where (for example) boxen runs, updates homebrew which in turn updates the config file and we would be left with the homebrew version of the configuration, not Boxen. We could mitigate this in a couple of ways but it seems pretty hacky to do.

This breaks currently because as of Homebrew 1.3.0 you cannot install into arbitrary locations on the disk from inside a formula. As a result an etc directory is your only option. I'd suggest etc/"boxen" to avoid clashes.

That seems pretty reasonable @MikeMcQuaid, I'll POC it out. Thanks! 🌟

Closing this since it will be addressed in boxen/our-boxen#855


OpenShift S2I cannot push image on EC2

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.492 s
[INFO] Finished at: 2016-12-07T15:39:09+00:00
[INFO] Final Memory: 31M/267M
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "openshift" could not be activated because it does not exist.
Copying built war files into /wildfly/standalone/deployments for later deployment...
Copying all war artifacts from /opt/app-root/src/target directory into /wildfly/standalone/deployments for later deployment...
'/opt/app-root/src/target/example-camel-cdi-4.4.0.war' -> '/wildfly/standalone/deployments/example-camel-cdi-4.4.0.war'
Copying all ear artifacts from /opt/app-root/src/target directory into /wildfly/standalone/deployments for later deployment...
Copying all rar artifacts from /opt/app-root/src/target directory into /wildfly/standalone/deployments for later deployment...
Copying all jar artifacts from /opt/app-root/src/target directory into /wildfly/standalone/deployments for later deployment...
...done

Pushing image <IP_ADDRESS>:5000/wildfly-camel/camel-cdi:latest ...
Registry server Address:
Registry server User Name: serviceaccount
Registry server Email<EMAIL_ADDRESS>Registry server Password: <<non-empty>>
error: build error: Failed to push image: Get https://<IP_ADDRESS>:5000/v1/_ping: http: server gave HTTP response to HTTPS client

https://docs.docker.com/registry/deploying/
https://docs.docker.com/registry/insecure/


Wrong type of PK in mysql when using tinyint as type

Describe the bug
Driver: mysql
Version: Checked on 6.1.9, 6.3.6 and 6.3.7 and next
When we have two entities like
@Entity({ schema: 'foo', tableName: 'with_tiny' })
export class WithTiny {
  @PrimaryKey({ name: 'tiny_id', type: TinyIntType })
  id!: number

  @Property()
  name!: string
}

@Entity({ schema: 'foo', tableName: 'not_tiny' })
export class CasualEntity {
  @PrimaryKey({ name: 'not_tiny_id' })
  id!: number

  @ManyToOne(() => WithTiny, { fieldName: 'tiny_id' })
  tiny!: WithTiny
}

Then during initialisation of db it creates them like
  CREATE TABLE `with_tiny` (
    `tiny_id` int unsigned NOT NULL AUTO_INCREMENT, <---- NOTE: That should be tinyint
    `name` varchar(255) NOT NULL,
    PRIMARY KEY (`tiny_id`)
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci

  CREATE TABLE `not_tiny` (
    `not_tiny_id` int unsigned NOT NULL AUTO_INCREMENT,
    `tiny_id` tinyint unsigned NOT NULL,  <---- NOTE: There is correct one
    PRIMARY KEY (`not_tiny_id`),
    KEY `not_tiny_tiny_id_index` (`tiny_id`)
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci

Where the with_tiny.tiny_id should be tinyint and it's int (but in not_tiny table the type is correct) what cause the constraint to fail
DriverException: alter table `foo`.`not_tiny` add constraint `not_tiny_tiny_id_foreign` foreign key (`tiny_id`) references `foo`.`with_tiny` (`tiny_id`) on update cascade; - Referencing column 'tiny_id' and referenced column 'tiny_id' in foreign key constraint 'not_tiny_tiny_id_foreign' are incompatible.

Tried to find if I'm missing anything but I failed to find any information about it :(
Reproduction
https://github.com/tomaszwylezek/micro-orm-pk-tinyint
Follow README.md
What driver are you using?
@mikro-orm/mysql
MikroORM version
6.1.9, 6.3.6, 6.3.7 and next
Node.js version
v18.18.0
Operating system
mac
Validations

[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord.
[X] The provided reproduction is a minimal reproducible example of the bug.

Try this instead:
@PrimaryKey({ name: 'tiny_id', columnType: 'tinyint' })
id!: number

But I can see it won't help, since the support for autoincrement PKs in knex is really weird (and this is a side-effect of how they do things).

Tried it already - same issue.

Overall we had similiar issued with AI Pks and unsigned: false when generating different entities - we "hacked it" by having separate decorator which is wrapping @PrimaryKey and skipping unsigned | type | autoincrement for setup process (but ofc it doesn't fix the issue as the types are wrong across the app then as we have the PK type in both entity definition and in places when used as FK)

Fixed via #6065


Error Trying to Download Symbols Default Project

Describe the bug
[Error - 10:02:00] Please report this issue to https://github.com/microsoft/al/issues including information on how to reproduce it, if possible.
Processing of message 'al/downloadSymbols' failed with error: 'No credentials are available in the security package'
Details:
System.ComponentModel.Win32Exception (0x8009030E): No credentials are available in the security package
at System.Net.NTAuthentication.GetOutgoingBlob(Byte[] incomingBlob, Boolean throwOnError, SecurityStatusPal& statusCode)
at System.Net.NTAuthentication.GetOutgoingBlob(String incomingBlob)
at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Dynamics.Nav.Deployment.Telemetry.TelemetryHttpClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\a\1\s\source\Prod\Microsoft.Dynamics.Nav.Deployment\Telemetry\TelemetryHttpClientHandler.cs:line 61
at Microsoft.Dynamics.Nav.Deployment.Http.NavHttpClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\a\1\s\source\Prod\Microsoft.Dynamics.Nav.Deployment\Http\NavHttpClientHandler.cs:line 35
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)    at Microsoft.Dynamics.Nav.Deployment.ApiClients.PackagesApiClient.SendRequest(IHttpClient client, SymbolReferenceSpecification reference) in D:\a\1\s\source\Prod\Microsoft.Dynamics.Nav.Deployment\ApiClients\PackagesApiClient.cs:line 139    at Microsoft.Dynamics.Nav.Deployment.ApiClients.PackagesApiClient.DownloadPackage(IHttpClient client, SymbolReferenceSpecification specification, String directory, Boolean isSecondLevelDependency) in D:\a\1\s\source\Prod\Microsoft.Dynamics.Nav.Deployment\ApiClients\PackagesApiClient.cs:line 154    at Microsoft.Dynamics.Nav.Deployment.ApiClients.PackagesApiClient.DownloadPackages(ImmutableArray1 references, String targetDir) in D:\a\1\s\source\Prod\Microsoft.Dynamics.Nav.Deployment\ApiClients\PackagesApiClient.cs:line 54
at Microsoft.Dynamics.Nav.Deployment.ReferenceDownloader.NavDevServerPackageDownloader.DownloadPackages(ImmutableArray1 packages, String targetDirectory) in D:\a\1\s\source\Prod\Microsoft.Dynamics.Nav.Deployment\ReferenceDownloader\NavDevServerPackageDownloader.cs:line 32    at Microsoft.Dynamics.Nav.EditorServices.Protocol.LanguageServer.Extensions.DownloadSymbolsRequestHandler.DownloadFiles(DownloadSymbolsRequest request, String cacheDirectory, ImmutableArray1 referencesToDownload) in D:\a\1\s\source\Prod\Microsoft.Dynamics.Nav.EditorServices.Protocol\LanguageServer\Extensions\DownloadSymbolsRequestHandler.cs:line 124
at Microsoft.Dynamics.Nav.EditorServices.Protocol.LanguageServer.Extensions.DownloadSymbolsRequestHandler.ProcessRequestAsync(DownloadSymbolsRequest request, Int32 requestId, CancellationToken cancellationToken) in D:\a\1\s\source\Prod\Microsoft.Dynamics.Nav.EditorServices.Protocol\LanguageServer\Extensions\DownloadSymbolsRequestHandler.cs:line 93
at Microsoft.Dynamics.Nav.EditorServices.Protocol.LanguageServer.Extensions.NavServerRequestHandler2.HandleAsync(T request, Int32 requestId, CancellationToken cancellationToken) in D:\a\1\s\source\Prod\Microsoft.Dynamics.Nav.EditorServices.Protocol\LanguageServer\Extensions\NavServerRequestHandler.cs:line 40    at Microsoft.Dynamics.Nav.EditorServices.Protocol.MessageProtocol.RequestHandlerBase1.HandleAsync(JToken requestContents, Int32 requestId, CancellationToken cancellationToken) in D:\a\1\s\source\Prod\Microsoft.Dynamics.Nav.EditorServices.Protocol\MessageProtocol\RequestHandlerBase.cs:line 63
at Microsoft.Dynamics.Nav.EditorServices.Protocol.RequestRegistry.Process(Message message) in D:\a\1\s\source\Prod\Microsoft.Dynamics.Nav.EditorServices.Protocol\Endpoints\RequestRegistry.cs:line 64
To Reproduce
Trying to download  symbols from a docker container.
The Web Client works properly.
Tried using the AL Extension on the store and tried using the al extension in the docker container.
AL Code to reproduce the issue
{
"version": "0.2.0",
"configurations": [
{
"name": "Your own server",
"request": "launch",
"type": "al",
"environmentType": "OnPrem",
"server": "http://devContainer",
"serverInstance": "BC",
"authentication": "Windows",
"startupObjectId": 22,
"startupObjectType": "Page",
"breakOnError": true,
"launchBrowser": true,
"enableLongRunningSqlStatements": true,
"enableSqlInformationDebugger": true,
"tenant": "default"
}
]
}
{
"id": "d6167eb3-78bb-43a3-b2e4-b1a45a0af402",
"name": "ALProject3",
"publisher": "Default publisher",
"version": "<IP_ADDRESS>",
"brief": "",
"description": "",
"privacyStatement": "",
"EULA": "",
"help": "",
"url": "",
"logo": "",
"dependencies": [
{
"id": "63ca2fa4-4f03-4f2b-a480-172fef340d3f",
"publisher": "Microsoft",
"name": "System Application",
"version": "<IP_ADDRESS>"
},
{
"id": "437dbf0e-84ff-417a-965d-ed2bb9650972",
"publisher": "Microsoft",
"name": "Base Application",
"version": "<IP_ADDRESS>"
}
],
"screenshots": [],
"platform": "<IP_ADDRESS>",
"idRanges": [
{
"from": 50100,
"to": 50149
}
],
"contextSensitiveHelpUrl": "https://ALProject3.com/help/",
"showMyCode": true,
"runtime": "5.0"
}
Expected behavior
Shouln't the VSCode ask me for Windows Credentials so acess the docker container?
5. Versions:

AL Language: v6.1.362735
Business Central: 16.3

As far as I know, Docker containers do not work with Windows credentials. Only user/password is supported. That being said, we should not throw an exception.

Having the same issue with the latest BCContainerHelper pre-release version

Can you provide a script to create a container and a project to reproduce the issue?

This issue has been closed because it needs more information and has not had recent activity. Please open a new issue if you believe this has not been resolved and reference the current issue.


Reverse Tang unlocking

There is one use-case of a remote unlock that might be worth implementing at the booster side.
A remote host boots and reaches a locked root partition. At this point, initrd hangs and waits until the administrator enters the password. As this server is remote, there is no way to enter the password using a keyboard. Instead, the hosts expect the password is passed over the network.
Some initrd implementations have plugins to bringup ssh server at initrd stage e.g. mkinitcpio-dropbear. Setting up an sshd daemon brings a lot complexity to initramfs. It requires a full network setup, probably systemd/udev, the sshd server itself. Complexity is the enemy of maintainability. Complexity is the enemy of security. It would be great to implement the same use case with a simpler architecture.
In fact, booster already allows unlocking drives remotely with Tang technology. Tang is a simple two message protocol to derive keys securely over an insecure communication channel. It makes a lot of sense to reuse it. But instead of having a Tang server we need the initramfs to stop and wait for messages. Then administrator contacts the remote host and executes ECMR key exchange as would normal Tang does.
Here is what the unlock would look like:

the remote hosts opens a port and waits for incoming messages
administrator runs a CLI tool that initiates one ECMR key exchange. The cli would look like booster unlock server1.mycompany.com ./secret.key.

@ShapeShifter499  answering your comment https://github.com/anatol/booster/issues/114#issuecomment-984281535 here
The idea behind Clevis-Tang is that both parties contain some secret value and ECMR protocol allows to derive a secret value over insecure channel. Once client (the clevis side) computes the derived key it can use it for dm-crypt or for wrapping/unwrapping another key.
With this schema Tang server runs as a TCP server and Clevis side connects to it.
With the reverse-Tang the TCP roles changed. Clevis starts up, opens a TCP port that starts listening. It basically asks "are there any Tang's here"? The Tang side connects to Clevis port and then ECMR exchange happens.

I have a new concern about that reverse tang plan.  Do you think a Clevis system could be exploited if it's looking for a Tang server and a malicious one responds?

Do you think a Clevis system could be exploited if it's looking for a Tang server and a malicious one responds?

Malicious Tang could try to unlock the drive but it will be impossible for it unless it has the private key. The private key is what really needed to reconstruct the disk password. Without it the client will not be able to verify the password, it will reject the answer from malicious Tang and wait for another request.
The only problem I see here is DOS attack when a lot of malicious users try to unlock with wrong keys and make the client busy with verifying it while valid user gets stuck because network queue is overflown.

FYI I just release tang.go - the library that implements the server-side of the ECMR exchange. It is needed to get the current feature implemented.
https://github.com/anatol/tang.go

Remote unlock support has been added to the master branch.
To enable remote locking please follow these instructions:

install clevis-extra-pins-git tang.go-git packages from AUR
generate private keys with tang-keys create . sig exc. It creates 2 files: a signing and a keyDerive keys. Please keep these files private.
generate advertisement: tang-keys adv --output advertisement.json sig.jwk exc.jwk. An advertisement is the equivalent of a public signing certificate. This file is used later to encrypt partitions and the file can be shared with other parties.
encrypt the partition itself using advertisement generated above: clevis luks bind -d /dev/fooo remote '{"adv":"advertisement.json", "port":34551}'. This tells booster to stop at the boot and use port 34551 to listen to incoming TCP requests.

now reboot your machine. It will stop and wait for unlock requests. To unlock it run unlock-remote $YOURMACHINE:34551 assets/remote/exc.jwk assets/remote/sig.jwk.
Please give it a try and let me know how does it work for you.

I've made a few additional changes to simplify the command-line tool interface.
Now all tang.go functionality (tang-keys, tangd, unlock-remote ) consolidated into 1 binary - tangctl.
The tool usage looks like this now:

Generate tang private key with tangctl create > key.priv. This is the key used for tangd and to unlock an encrypted machine.
Generate a public key (or in tang lingo it is a "key advertisement") tangctl public key.priv > key.pub. key.pub can be distributed to a third party to encrypt their machines as clevis luks bind -d /dev/fooo remote '{"adv":"key.pub", "port":34551}'

To unlock a remote machine run tangctl unlock $YOURMACHINE:34551 priv.key.


feat(front): 蓝鲸交互规范落地 - 侧栏优化(closed #1552)

Codecov Report

Merging #1563 (c9d77c1) into v2.3.2-dev (5d508f2) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@              Coverage Diff               @@
##           v2.3.2-dev    #1563      +/-   ##
==============================================
- Coverage       73.39%   73.39%   -0.01%
==============================================
  Files             387      387
  Lines           26074    26070       -4
==============================================
- Hits            19137    19134       -3
+ Misses           6937     6936       -1

see 2 files with indirect coverage changes


[1533] Optimise providers show

Context
Fixing performance.
Changes proposed in this pull request
Fix N+1 enrichments query.
Guidance to review
DRAFT. :)
Checklist

[ ] Make sure all information from the Trello card is in here
[ ] Attach to Trello card
[ ] Rebased master
[ ] Cleaned commit history
[ ] Tested by running locally

 Optimise organisation page on Publish


Roll Flutter Engine from 1cdbebee1901 to 2eb873445d26 (3 revisions)

https://github.com/flutter/engine/compare/1cdbebee1901...2eb873445d26
2024-06-07<EMAIL_ADDRESS>Roll Skia from 5fc83884a619 to b7f51dfcc342 (4 revisions) (flutter/engine#53275)
2024-06-07<EMAIL_ADDRESS>Revert "Widen CPU affinity set." (flutter/engine#53274)
2024-06-07<EMAIL_ADDRESS>Roll Skia from e36110f8b451 to 5fc83884a619 (1 revision) (flutter/engine#53273)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC matanl@google.com,rmistry<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


New features to fec_conv

Adding to the above, the new features to be merged into master include rate 1/3 convolutional coding, had, soft, and quantized soft decision Viterbi decoding is refined over the current version. Doc string help as well as a Jupyter notebook Convolutional_Codes.ipynb, is also added.


better treatment of packages that depend on beta (versioning mismatch)

Try the following:

load https://github.com/microbit-apps/pxt-arcadeshield in https://makecode.microbit.org/beta (WORKS)
load https://github.com/microbit-apps/pxt-arcadeshield in https://makecode.microbit.org (FAILS, silently)

The WORKS and FAILS behaviors are expected, but the silent FAIL is not great.  When there is a version mismatch (as above in (2)), we should have a nicer message.

The silent failure is here
pxtapp.js:1  Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'pxt.json')
    at n.getConfigAsync (pxtapp.js:1:279221)
    at pxtapp.js:1:285290

the code path is
            if (!i.startsWith("pub:")) {
                const s = e.patching.upgradePackageReference(t, n, i)
                  , r = e.appTarget.bundledpkgs[s];
                return JSON.parse(r[e.CONFIG_NAME])
            }


systemd-additional-units: Add dbus dependency

Without this we can't be sure the dbus-1 package is available, causing a build issue.

Looks good to me.


Remove keen

Codecov Report

Merging #266 into master will decrease coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #266      +/-   ##
==========================================
- Coverage   97.41%   97.38%   -0.04%
==========================================
  Files          81       80       -1
  Lines        3130     3093      -37
  Branches       24       24
==========================================
- Hits         3049     3012      -37
  Misses         78       78
  Partials        3        3

Impacted Files
Coverage Δ

packages/etherlime/cli-commands/commands.js
98.81% <ø> (-0.11%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 136575a...aacbc6b. Read the comment docs.


Truesight 3.3 Vulnerable Driver leading to Process killing of protected processes.

https://github.com/ph4nt0mbyt3/Darkside

Thanks @ph4nt0mbyt3 for the heads up.


Whoosh and DB LockError while import of users

Try to import 3 users (e.g. enter 3 lines in textarea or use a csv example).
Only 2 users are imprted. Server shows a lot of errors:
...
django.db.utils.OperationalError: database is locked
[2016/06/11 22:19:21] HTTP POST /rest/users/user/ 500 [1.18, <IP_ADDRESS>:37592]
Internal Server Error: /rest/users/user/
...
whoosh.index.LockError

@ostcar Please look at this. It looks like a problem caused by big mode.

There could be two different errors, one with whoosh and one with sqlite. I could only reproduce the one with whoosh.
Django Channels runserver command starts 4 worker threads. So even on the small mode, OpenSlides has to be multithread ready.
On the client side the user import sends different post-request to the server at the same time. So the server tries to create to create the users at the same time and tries to touch the sqlite database and the whoosh index for writing at the same time.
The whoosh problem was expected: https://github.com/OpenSlides/OpenSlides/blob/master/openslides/utils/search.py#L117

You can see the the sqlite error if you scroll up in your server terminal output (comes before the whoosh error).
This symptom of users import (several write operations at one time) could also be happened if several admin users run different write actions in the same time. Right?

@ostcar @normanjaeckel
This issue is a blocker for use current master in production - and also for tests with many users/motions you always run into this lockerror. So we need a fix for this regression asap. Who can look a this issue?

Try this: http://whoosh.readthedocs.io/en/latest/api/writing.html#utility-writers

Please test again. The SQLite error might be still there.

Successfully tested: The whoosh.index.LockError is fixed with PR #2251.

Still open: the DB error django.db.utils.OperationalError: database is locked
Steps to reproduce:
Import users csv example file.
@normanjaeckel Any idea?

https://docs.djangoproject.com/en/dev/ref/databases/#database-is-locked-errors

@normanjaeckel Can you try a pull request?

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(OPENSLIDES_USER_DATA_PATH, 'database.sqlite'),
        'OPTIONS': {
            'timeout': 600000
        }
    }
}

Try this but it does not fix the problem.

I can not reproduce the sqlite error. It works fine on my computer. I read in the documentation of python-sqlite, sqlite and django but could not find anything why I works for me and not for you. @emanuelschuetze can you test this on different operating systems? Maybe it is a special problem with debian based distributions (maybe they use strange compile arguments for sqlite).

@ostcar What were your steps to reproduce it? Please try my steps to reproduce it (I cannot imagine that it depends from the operation system):
Here my steps to reproduce it always on Ubuntu 16.04:

remove personal_data directory (with DB, search index, settings etc.)
run OpenSlides on current master: python manage.py start (new sqlite db is created)
login and go to /users/import
download the csv example and select this csv file (you see 4 example users in the preview)
click button "Import 4 participants"
=> Only 3 of 4 participants are successfully imported. Server terminal shows errors:
sqlite3.OperationalError: database is locked

Full backtrace:
Internal Server Error: /rest/users/user/
Traceback (most recent call last):
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 323, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/channels/handler.py", line 227, in process_exception_by_middleware
    return super(AsgiHandler, self).process_exception_by_middleware(exception, request)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/rest_framework/viewsets.py", line 87, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/rest_framework/views.py", line 466, in dispatch
    response = self.handle_exception(exc)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/rest_framework/views.py", line 463, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/rest_framework/mixins.py", line 21, in create
    self.perform_create(serializer)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/rest_framework/mixins.py", line 26, in perform_create
    serializer.save()
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/rest_framework/serializers.py", line 191, in save
    self.instance = self.create(validated_data)
  File "/home/emanuel/projects/openslides/src/OpenSlides/openslides/users/serializers.py", line 103, in create
    return super().create(validated_data)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/rest_framework/serializers.py", line 867, in create
    setattr(instance, field_name, value)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 481, in __set__
    manager.set(value)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 910, in set
    self.add(*new_objs)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 843, in add
    self._add_items(self.source_field_name, self.target_field_name, *objs)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 1004, in _add_items
    for obj_id in new_ids
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/models/query.py", line 450, in bulk_create
    self._batched_insert(objs_without_pk, fields, batch_size)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/models/query.py", line 1056, in _batched_insert
    using=self.db)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/models/query.py", line 1039, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1060, in execute_sql
    cursor.execute(sql, params)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/emanuel/projects/openslides/src/OpenSlides/.venv/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 323, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database is locked

@normanjaeckel I tested the timeout option in settings.py - same error.

This seems to be a problem with one of our requirements.
I did exactly what you ask and did not get an error. Then I run pip install -r requirements.txt -U and now I get the database is locked error.
Here are the packages, that where updated:
Installing collected packages: Django, beautifulsoup4, asgiref, autobahn, daphne, channels, webencodings, setuptools, html5lib, coverage, mccabe, flake8, pillow, tornado
  Found existing installation: Django 1.9.7
    Uninstalling Django-1.9.7:
      Successfully uninstalled Django-1.9.7
  Found existing installation: beautifulsoup4 4.4.1
    Uninstalling beautifulsoup4-4.4.1:
      Successfully uninstalled beautifulsoup4-4.4.1
  Found existing installation: asgiref 0.13.3
    Uninstalling asgiref-0.13.3:
      Successfully uninstalled asgiref-0.13.3
  Found existing installation: autobahn 0.14.1
    Uninstalling autobahn-0.14.1:
      Successfully uninstalled autobahn-0.14.1
  Found existing installation: daphne 0.13.0
    Uninstalling daphne-0.13.0:
      Successfully uninstalled daphne-0.13.0
  Found existing installation: channels 0.15.0
    Uninstalling channels-0.15.0:
      Successfully uninstalled channels-0.15.0
  Found existing installation: setuptools 23.1.0
    Uninstalling setuptools-23.1.0:
      Successfully uninstalled setuptools-23.1.0
  Found existing installation: html5lib 0.9999999
    Uninstalling html5lib-0.9999999:
      Successfully uninstalled html5lib-0.9999999
  Found existing installation: coverage 4.1
    Uninstalling coverage-4.1:
      Successfully uninstalled coverage-4.1
  Found existing installation: mccabe 0.5.0
    Uninstalling mccabe-0.5.0:
      Successfully uninstalled mccabe-0.5.0
  Found existing installation: flake8 2.6.2
    Uninstalling flake8-2.6.2:
      Successfully uninstalled flake8-2.6.2
  Found existing installation: Pillow 3.2.0
    Uninstalling Pillow-3.2.0:
      Successfully uninstalled Pillow-3.2.0
  Found existing installation: tornado 4.3
    Uninstalling tornado-4.3:
      Successfully uninstalled tornado-4.3
Successfully installed Django-1.9.9 asgiref-0.14.0 autobahn-0.15.0 beautifulsoup4-4.5.1 channels-0.17.2 coverage-4.2 daphne-0.14.3 flake8-3.0.4 html5lib-0.999999999 mccabe-0.5.2 pillow-3.3.0 setuptools-25.1.6 tornado-4.4.1 webencodings-0.5

So to fix this we have to find the problematic dependency, downgrade it and write a bug report.

@FinnStutzenstein Please try to find the problematic python package (and version). Thx.

I have checked all packages alone, so all other packages are the newest ones. I get the Error every time, it seems to be a combination of packages, that causes this error. My workflow:
rm -rf peronal_data/
pip install -r requirements.txt -U //Update the last package to the newest version
pip uninstall -v django
pip install -v django==1.9.7
python manage.py start --no-browser --host <IP_ADDRESS> --port 9999
//testing to import the example.csv

I had problems with beautifulsoap4. While uninstalling I have got this error:
Cannot uninstall requirement beautifulsoap4, not installed So, the error may come from beautifulsoap4 - I will try to check this.
Now I am going to downgrade all mentioned packages to check, whether I can get rid of the this error and try to upgrade each package at once, hoping to find the right package/combination.

@ostcar I could not recreate your status where no error appeared. I have the following packages and versions matching yours:
$ pip freeze
asgiref==0.13.3
autobahn==0.14.1
backports-abc==0.4
beautifulsoup4==4.5.1
channels==0.15.0
coverage==4.1
daphne==0.13.0
Django==1.9.7
djangorestframework==3.3.3
flake8==2.6.2
html5lib==0.9999999
isort==4.2.5
jsonfield==1.0.3
mccabe==0.5.0
natsort==4.0.4
Pillow==3.2.0
pycodestyle==2.0.0
pyflakes==1.2.3
PyPDF2==1.25.1
reportlab==3.3.0
roman==2.0.0
six==1.10.0
sockjs-tornado==1.0.3
tornado==4.3
Twisted==16.2.0
txaio==2.5.1
webencodings==0.5
Whoosh==2.7.4
zope.interface==4.2.0

Maybe you have overseen something? Emanuel said like an older databaseversion or so.

I don't get a working version now neither. At least I can reproduce the error now...

See https://github.com/andrewgodwin/channels/pull/297


info.log have render messages even LOG_RENDERING_PERFORMANCE = False

Graphite info log have renderer messages even if I configure LOG_RENDERING_PERFORMANCE = False
Issue in following code line:
https://github.com/graphite-project/graphite-web/blob/3eded1d74fe36facdad606f5e088023dfecfd410/webapp/graphite/util.py#L238
I believe it must be log.rendering

The line above is a part of logtime decorator which is calling from many places. But it needs to be fixed, though.


could not create directory

Hi, I don't want to be annoying but I really want to find a solution to the problem I'm having since I haven't seen it anywhere else and I'm really interested in using the plugin without having to patch it :sweat_smile:.
When running :LspInstall lang I get the error could not create directory lang. As I had thought about the PR that for some reason the command was being executed with my version of Lua, I downgraded to lua 5.1 to make sure it wasn't that, and the same problem continues.
In case this information is useful for something:

Nvim version: NVIM v0.5.0-dev+1177-g095f5da79
doing os.execute("mkdir -p ".. path) inside Neovim returns true exit 0 as it would in newer versions of Lua (I don't understand why yet, if Lua 5.1 is supposed to be used in Neovim)

Thanks in advance :slightly_smiling_face:

Nvm, I just saw that the compilation that Neovim recommends for nightly builds in Fedora apparently is not the same as the AppImage, now it works as it should!
Note for those who use the agriffis copr: if you want to use this plugin, you will have to use the AppImage provided by Neovim instead.

Thanks for following up on this!


overrides as parameter to activation

evaluator access defers to activation.functions

Check the latest release. I think this provides suitable override definitions. For examples, see the examples in https://github.com/cloud-custodian/cel-python/blob/master/docs/source/integration.rst. These show an approach that seems consistent with Go.


Networking Issues in Kubernetes Immediately Following Upgrades

This is an issue requested from @leodotcloud concerning networking issues and kubernetes we've observed in our environments.
Issues seen:

Ingress controllers are unable to be communicated with. There appears to be a "macsync" issue which is often remedied with a reboot, but not 100% of the time.
Occasional ARP cache issues where network communicates do not transmit properly, which has caused a catastrophic etcd failure in dev.

Steps to Reproduce:

Create a new rancher stack, (e.g. 1.6.4) and install kubernetes (In my example, 1.6)
Be sure to disable the rancher ingress controller, otherwise it gets in the way.
After the cluster is created create a traefik ingress daemonset
Once the traefik ingress is working and responding appropriately, perform an upgrade to rancher and kubernetes (in my example 1.6.5 and its upgrades caused a failure, as well as a test from 1.6.5 to 1.6.6 and k8s 1.7)
Ingress controllers will stop working (sometimes a select few, sometimes all, but always at least one). The container will be running, and listening on port 80, but not doing anything. You cannot netcat, curl, etc, to it.
Network manager container on the host for which the ingress controller will have error messages similar to the following:

8/14/2017 3:54:09 PMtime="2017-08-14T22:54:09Z" level=error msg="macsync: for container: {traefik-ingress-controller-7g72s <IP_ADDRESS> 02:ae:9c:da:8d:76 [<IP_ADDRESS>] [<IP_ADDRESS>:80:80/tcp <IP_ADDRESS>:8080:8080/tcp]    map[io.rancher.container.display_name:traefik-ingress-controller-7g72s io.rancher.container.network:true io.rancher.stack.name:kube-system pod-template-generation:1 controller-revision-hash:546381873 io.kubernetes.container.name:POD io.kubernetes.pod.namespace:kube-system k8s-app:traefik-ingress-lb annotation.kubernetes.io/config.seen:2017-08-09T17:02:35.572280667Z annotation.kubernetes.io/config.source:api io.kubernetes.pod.name:traefik-ingress-controller-7g72s io.kubernetes.pod.uid:8a986fdf-7d24-11e7-a9de-02ae9cca6e8d io.rancher.service.launch.config:io.rancher.service.primary.launch.config name:traefik-ingress-lb annotation.kubernetes.io/created-by:{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"DaemonSet\",\"namespace\":\"kube-system\",\"name\":\"traefik-ingress-controller\",\"uid\":\"a8116811-7c67-11e7-b5f0-02ae9c26e2c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"355234\"}}\n io.kubernetes.docker.type:podsandbox io.rancher.service.deployment.unit:8a986fdf-7d24-11e7-a9de-02ae9cca6e8d] 0 1ab57e95-b7d7-4ad1-a7a5-3dabf8cff02d 3128ecf0-3e08-4345-b463-7bb4bf625ee7 running  af3d191aa9bf463be2601ae0076dbd756556f9c71f3e403a9b40a9395279fed2 1 0 0 [<IP_ADDRESS>] [rancher.internal] []  f80bce3a-404a-418d-abbc-a10b66f4cbe3 map[] false 26fc4bd1-8a7b-4950-8eef-b1fbd5cfa090 {0 0 0  0 0}}, could not lookup interface: Link not found"
8/14/2017 3:54:09 PMtime="2017-08-14T22:54:09Z" level=error msg="macsync: error syncing MAC addresses for the first tiime: in name ns for container af3d191aa9bf463be2601ae0076dbd756556f9c71f3e403a9b40a9395279fed2: Link not found"

Traefik yaml:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    k8s-app: traefik-ingress-lb
  name: traefik-ingress-controller
  namespace: kube-system
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
      name: traefik-ingress-lb
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      containers:
      - args:
        - -d
        - --web
        - --kubernetes
        image: traefik
        imagePullPolicy: Always
        name: traefik-ingress-lb
        ports:
        - containerPort: 80
          hostPort: 80
          name: http
          protocol: TCP
        - containerPort: 8080
          hostPort: 8080
          name: admin
          protocol: TCP
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      hostNetwork: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: traefik-ingress-controller
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
  templateGeneration: 1
  updateStrategy:
    type: OnDelete

Assigning the issue to you as requested @alena1108
Summary:
apiVersion: v1
kind: Pod
metadata:
  name: full
spec:
  containers:
    - name: full
      image: nginx
  hostNetwork: true

The above yaml file will launch a pod in host network, but in the UI/metadata you can see this container is registered on the managed network. Since it's on the managed network, we try to do macsync, etc and doing those incorrectly on a container which doesn't belong to managed network can have random side effects.

@leodotcloud I'll add the missing logic to not place k8s hostNetwork=true container on the managed network, but I am curious how it even worked before? This bug in k8s pods sync has been in rancher for a long time, yet it somehow worked? Did anything got changed in macsync logic on the network manager side?

Available with rancher/server:v1.6.8-rc4 and kubernetes v1.7.4-rancher1-etcdv2 using image rancher/kubernetes-agent:v0.6.5

@galal-hussein Can you also add the validation to check connectivity for the existing K8s host network test case?

With rancher 1.6.7 installed from scratch I have similar issue but without observing errro in network manager container. netstat -tulpn on host shows:
tcp        0      0 <IP_ADDRESS>:8080            <IP_ADDRESS>:*               LISTEN      18957/traefik
tcp        0      0 <IP_ADDRESS>:80              <IP_ADDRESS>:*               LISTEN      18957/traefik

But if you try connect to host:80 you will get icmp with "Host ureachable" error.

Rancher server: v1.6.8-rc5
Fresh installation of kubernetes on this rancher version and deploying host networked pod didn't reveal any issues and i can see that the pod didn't get a 10.42.x.x ip and not in the Rancher managed network.
However when upgrading from v1.6.7 to v1.6.8-rc5 i can see that the old pod still got the 10.42.x.x ip and  its networking seems to be broken, pod used:
apiVersion: v1
kind: Pod
metadata:
  name: nginx2
  labels:
    app: nginx
spec:
  hostNetwork: true
  containers:
  - name: nginx
    image: sangeetha/testnewhostrouting
    ports:
    - containerPort: 80

i can't connect to the host's ip anymore.


Looks like packDestination of DotNetCoreCLI@2 is not working when using the pack command

In my pipeline I am using the following yaml to create nuget packages for my project.
      - task: DotNetCoreCLI@2
        displayName: 'Build Nuget package assemblies'
        inputs:
          command: pack
          packagesToPack: 'assemblies/project1/project1.csproj'
          packDestination: '$(Build.ArtifactStagingDirectory)/nuget/generatedpackages'
          verbosityPack: Detailed

I would expect the nuget packages will be stored in  '$(Build.ArtifactStagingDirectory)/nuget/generatedpackages', which should be resolved to '/home/vsts/work/1/a/nuget/generatedpackages'. Unfortunately the package are stored in the root ''/home/vsts/work/1/a'.
When I look a the log information, it looks like the packDestination is not passed to the command:
Starting: Build Nuget package assemblies
==============================================================================
Task         : .NET Core
Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command
Version      : 2.198.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
==============================================================================
/usr/bin/dotnet pack /home/vsts/work/1/s/assemblies/project1/project1.csproj --output /home/vsts/work/1/a /p:Configuration=Release --verbosity Detailed

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e0667a90-8546-51bc-0e5d-2877eb4adfdd
Version Independent ID: 878272e9-94bd-5055-239a-e5477f26ca1a
Content: Publish NuGet packages with Pipeline tasks or the classic editor - Azure Pipelines
Content Source: docs/pipelines/artifacts/nuget.md
Product: devops
Technology: devops-cicd-artifacts
GitHub Login: @ramiMSFT
Microsoft Alias: rabououn

@JohanKlijn -- Johan, thank you for your well-researched feedback.  Please consider these resources:

Azure DevOps Services
Azure DevOps on Stack Overflow

If you feel this is a product bug, please add it here.
@ramiMSFT -- Rami, please look into this potential documentation issue.

@JohanKlijn Thanks for your feedback! This sounds like an issue with the task. Can you file an issue in Pipelines tasks.

For the people reading this topic, I filed an issue, see https://github.com/microsoft/azure-pipelines-tasks/issues/16171

FYI for any that hit this. The workaround is to set the parameter name to be outputDir which will correctly pass the value through to the cli --output.


Automated cherry pick of #9555: fix(region): fix openstack instance os type

Cherry pick of #9555 on release/3.5.
#9555: fix(region): fix openstack instance os type

Codecov Report

Merging #9557 (4258ed3) into release/3.5 (6a15e02) will increase coverage by 11.92%.
The diff coverage is 1.85%.

@@               Coverage Diff                @@
##           release/3.5    #9557       +/-   ##
================================================
+ Coverage         5.98%   17.91%   +11.92%
================================================
  Files              796       85      -711
  Lines           125693     8921   -116772
================================================
- Hits              7522     1598     -5924
+ Misses          117504     7151   -110353
+ Partials           667      172      -495

Flag
Coverage Δ

aFlag
17.91% <1.85%> (+11.92%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/apigateway/handler/auth.go
0.00% <0.00%> (ø)

.../monitor/alerting/conditions/suggestrulereducer.go
0.00% <0.00%> (ø)

pkg/monitor/alerting/conditions/suggestquery.go
2.00% <2.00%> (ø)

pkg/util/seclib2/seclib.go
80.39% <0.00%> (-5.66%)
:arrow_down:

pkg/compute/models/loadbalancercachedacls.go

pkg/cloudcommon/policy/policy.go

pkg/mcclient/options/k8s/ingress.go

pkg/baremetal/tasks/basedeploy.go

pkg/appsrv/radix.go

pkg/compute/regiondrivers/zstack.go

... and 595 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cab0077...4258ed3. Read the comment docs.


Compensation of solar panel support

Here you find the mechanical calculations of how we compensate the solar panel support.
With this compensation our stepper motor 1 will move a zero  theoretical torque (N*m)
Calcul_CompensacioSuportPlacaSolar.pdf

Hello Nico,
Good job. Just a comment:
The explanations must be translated to English for the final documentation attached to the web page.


[ISSUE #3221]Feature/record log list/clean

about #3221 and #3258
work list:

[x] add clean log api.
[x] record log list by page api.

Codecov Report

Merging #3635 (8ce8186) into master (a27bb8f) will decrease coverage by 0.68%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #3635      +/-   ##
============================================
- Coverage     62.72%   62.04%   -0.69%
+ Complexity     5868     5806      -62
============================================
  Files           888      889       +1
  Lines         24723    24756      +33
  Branches       2263     2264       +1
============================================
- Hits          15507    15359     -148
- Misses         7763     7938     +175
- Partials       1453     1459       +6

Impacted Files
Coverage Δ

...u/admin/config/properties/DashboardProperties.java
14.28% <0.00%> (-10.72%)
:arrow_down:

...admin/controller/OperationRecordLogController.java
50.00% <0.00%> (-25.00%)
:arrow_down:

.../shenyu/admin/model/entity/OperationRecordLog.java
95.00% <ø> (ø)

...he/shenyu/admin/model/event/BatchChangedEvent.java
40.00% <ø> (ø)

.../admin/model/event/dict/BatchDictChangedEvent.java
0.00% <ø> (ø)

...henyu/admin/model/event/dict/DictChangedEvent.java
44.44% <ø> (ø)

...el/event/handle/BatchPluginHandleChangedEvent.java
0.00% <ø> (ø)

...n/model/event/handle/PluginHandleChangedEvent.java
0.00% <ø> (ø)

...odel/event/metadata/BatchMetaDataChangedEvent.java
0.00% <ø> (ø)

...min/model/event/metadata/MetaDataChangedEvent.java
0.00% <ø> (ø)

... and 73 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a27bb8f...8ce8186. Read the comment docs.


Lidar Lite V3 newer than 2016 is not recognized by Pixhawk 2.1

The Lidar Lite V3 rangefinder is an optional rangefinder to be used with the Pixhawk 2.1. It can be enabled by the parameter EKF2_RNG_AID=1 and the Lidar Lite V3 can be initialized by the parameter SENS_EN_LL40LS. I use the I2C protocol.
I currently have 3-4 Lidar Lite V3's that are dated 2016 on the label. These Lidars work every time. The tests I have ran:

Visual: Noticing the craft holds altitude
The Analyze tab in QGroundControl: In QGroundControl, look in the top left corner, click widgets, then analyze. Type distance_sensor_current in the filter box. If the Lidar is working it will display a parameter that can be clicked and data can be viewed. (Note: the Lidar must be connected to the Pixhawk correctly, and both the Pixhawk and the Lidar have to be powered. I will add links to the correct way to wire this at the end)
Arduino test:  Connected the Lidar to the arduino using the provided documentation.  https://learn.sparkfun.com/tutorials/lidar-lite-v3-hookup-guide/all

The 2016 Lidars will pass all of the tests described above. The 2018-2019 Lidars will only pass the arduino test.
I decided to go straight to the source by contacting Garmin. Garmin's response was, "The newer units use a newer hardware design that uses a faster FPGA.  The intent was for it to function as a drop-in replacement for the original, but there are some differences.  We have made updates to the https://github.com/garmin/LIDARLite_Arduino_Library that are intended to account for the differences."
Also I have attached what Garmin sent me about the change.
LLv3-I2C-Guidelines.pdf
Do you guys plan to support this new change? When could I expect the change to be implemented if so? Is there anything I can do in the meantime to fix this? Any and all help is appreciated.

LLv3-I2C-Guidelines.pdf

@ar1040 , I will put some time into this issue in the next week or so and report back here as I make progress.

@ar1040, my hardware is a V3HP and fortunately/unfortunately it works as expected with current PX4:master firmware flashed to a pixhawk 4 (fmu-v5). Where did you obtain your newer units and are they the V3 or V3HP?

@mcsauder, my hardware is a V3. I bought these units from robotshop.com, but they are original Garmins. I can use the PWM protocol and it works, but it only reads 12 meters max, versus 17 meters using I2C so I would prefer to have the extra 5 meters of distance.
I think it is also worth mentioning I am using a Pixhawk Cube and not the Pixhawk 4. I am also using the 1.8.2 f13bbac firmware version. I would like to swap to the 1.9.2 firmware but there are still issues with takeoff and attitude that conflicts with my software, but this is a different issue in itself.

Hi @ar1040 .  This might be very difficult to resolve if we cannot move you to the current release firmware; there are many bugfixes and code changes in the past year that might complicate resolving the issue or render it irrelevant. Could you raise a new issue to deal with the reason you cannot update to new firmware?

@mcsauder  I would like to use the newest firmware version due to the other fixes. I will open a new issue regarding the firmware issues. If the changes for the Lidar are pushed to the latest firmware version, it would still be very helpful, as I plan to switch our units to the latest firmware.

@ar1040 , I've flashed v1.8.2 and and see an issue that might be the same as yours with my fmu-v5 hardware, the ll40ls i2c driver fails to initialize properly in the LidarLiteI2C::probe() call.
So, I think you have two options: 1) Upgrade to v1.9.2, or 2) backport the current driver probe() method to 1.8.2 and keep running with what you have.
I have backported to v1.8.2 by copying the current PX4:master branch probe() method and a few #defines.
You'll need to add these #defines to LidarLiteI2C.h

#define LL40LS_UNIT_ID_HIGH 0x16
#define LL40LS_UNIT_ID_LOW 0x17
/* Device limits */
#define LL40LS_MIN_DISTANCE (0.05f)
#define LL40LS_MAX_DISTANCE_V3 (35.00f)
#define LL40LS_MAX_DISTANCE_V1 (25.00f)

and this variable in the class:
uint16_t	_unit_id;

and copy/paste the current probe() method from here.
As this appears to not be an issue with v1.9.2 and newer code, we can probably close out this issue.  Let us know your thoughts.

@mcsauder , I updated another one of my Pixhawk Cubes to v1.9.2 but I still cannot get a distance sensor reading. I normally test by opening QGC and clicking the widgets tab on the top left, then analyze. I then type "distance" in the search bar of the analyze window. If the sensor is recognized, it gives me 8 (I think) data sets to view. I choose the distance_sensor_current data set and view it to make sure it is reading the distance correctly.
However, even after I updated to v1.9.2 I could not get my Lidar Lite V3 to register.

@ar1040 , can you test again with today's current PX4:master branch?

@mcsauder , I pulled the latest master via QGC (dropdown and selected master). The firmware shows up as 1.9.0dev. I am not sure if I am pulling the correct firmware version. But I tested it anyway. Still, only my 2016 lidar works and none of the 2019s work with I2C (I tried 5 that I had on hand). If I am not testing the correct firmware please let me know.

Hello, I am having the same issue (also with the master branch). One thing I am noticing is that if I boot using an older Lidar I see the data measurements, and if I leave the drone on and connect the newer lidar it works as well. However, when I do a boot using the newer lidar it does not work.
With kind regards,
Lars

@ar1040 ,

@mcsauder, yes my is working now, but only if I run the ll40ls start command in the Mavlink Console, otherwise it never shows in the analyze window.

I've tried mine on the bench connected to I2C A and it starts correctly at boot with SENS_EN_LL40LS set to I2C (2).  Which port are you connected to so that I can try to get as close to your configuration as I can?

@mcsauder

I've tried mine on the bench connected to I2C A and it starts correctly at boot with SENS_EN_LL40LS set to I2C (2). Which port are you connected to so that I can try to get as close to your configuration as I can?<

I am using the only I2C port available on the Pixhawk 2.1 Cube that I am aware of.

@ar1040 ,

@mcsauder, yes my is working now, but only if I run the ll40ls start command in the Mavlink Console, otherwise it never shows in the analyze window.

I've tried mine on the bench connected to I2C A and it starts correctly at boot with SENS_EN_LL40LS set to I2C (2). Which port are you connected to so that I can try to get as close to your configuration as I can?

This is actually common problem as I am also met with this. The follows is what Garmin replied in email:
If the manufacture date on the sensor is after 08/2018, you will need to update your drivers in order to make it work properly.
You can refer to this support topic  article for more info https://support.garmin.com/en-US/?faq=BI0Aphz9cx5OXvG3fxnux6&partNumber=010-01722-00&searchQuery=i2c&tab=topics&textPage=1
Is this solved yet?

I'm refactoring the driver to try to solve this...  Once I get a PR in, would you all help me test?

No, this isn't solved yet.  Please feel free to submit a PR or see also #12695

Hi @HenryHuYu and @ar1040 , would you give current master branch a run and see if this issue is solved?  Thanks!

I think this is fixed now.... or at least I hope it is.  Please re-open if it is still an issue!

@mcsauder, I apologize for the late response. I did pull the master and I am still having the same problem as I can not get data from the lidar unless I start it with the "ll40ls start" command in the mavlink console.

I purchased a lidarlite which is manufactured 2019/01, i'm getting the same issue on 1.10.1
It doesn't show up and also even whether i start the driver it is giving me the following error

it seems my lidarlite has some problem, even in arduino it only gets nack message, not the same issue


Help documentation in modeller for method "append" is wrong

Context
Found in modeller version <IP_ADDRESS>
Problem description
The modeller help documentation (right menu bar) states that the method "append" should be used in this way:
append(DT, DT[]): DT[] ALT+a
This seems to be a wrong documentation because using it this way will lead to deployment issues.
Expectation
Documentation should state that method "append" should be used in the following way:
append(DT[], DT): DT[] ALT+a
working example
This is a working example without deployment issues.

Duplicate of #6 in xyna-process-modeller.
By mistake I opened this issue in the wrong project.


Add CLI opts for frontmatter

This adds CLI options for canonicalUrl, title, and tags, and adds canonicalUrl to frontmatter as an option. I also edited the documentation to show changes.
This closes #9 and #10, as it folds them in.

Sorry about the multiple PRs. Should have scoped this better initially.

Merged in 7ed5c7e57b7a846fddf8f163c72d54d6ebc09181.


Write full docs

I'm sure that there's some Julia utility where we can do full documentation of the package and build a readable site. Just need to investigate further...

Yee

https://nflverse.github.io/NFLData.jl/stable/


Make 32-bit builds and pick the right ones

I'm not sure this is actually worth doing, though, as no-one uses 32-bit machines these days, right?

Also arm64 for Pinebooks etc.

Also arm64 for Pinebooks etc.


VRApp works in the Editor but not in built using Linux

Hi, I am currently trying to build a VR application for Linux using this plugin.
I loaded the Interactions_Example scene and it runs perfectly fine in the Editor: head movements are tracked perfectly together with controllers.
The problem is that  I cannot see my controllers and also buttons do not work in the built Linux application. The only thing that works is the head movement tracking, so I am able to look around but I cannot do anything else.
Setup:

VR Headset: Meta Quest Pro connected through SteamVR using ALVR
Unity version: 22.3.21f1 ( I also tried with 2021.3.36f1 and I get the same result)
Linux OS: Ubuntu 22.04

Step to reproduce:

Launch ALVR and connect your headset using SteamVR
Open a new 3D Core Project in Unity
Import the steam_vr custom package following  #1107
Toggle Initialize XR on Startup  and OpenVR Loader in Edit -> Project Settings -> XR Plugin Management
Set Stereo Rendering Module to Multipass and Mirror View Mode to Open VR in Edit -> Project Settings -> OpenVR
Switch Graphics API to Vulkan for Linux in Edit ->  Project Settings -> Player -> Rendering
Generate SteamVR Inputs through Window -> SteamVR Input -> Save and generate
Press play -> it works correctly
Build app with File -> Build Settings -> Add Open Scenes then select Linux as Target Platform
Launch compiled AppImage -> head movement is tracked but no controllers appear

Update:
I managed to isolate the problem but still no solution: if I run the app built in debug mode and open the generated log file I find:
...
	Editor App Name :
[XR] 	Editor App Key :
[XR] 	Action Manifest Path : /home/nardi/Build/VRTest2_Data/StreamingAssets/SteamVR/actions.json
[XR] 	Stereo Rendering Mode : Multi Pass
[XR] 	Initialization Type : Scene
[XR] 	Mirror View Mode : OpenVR View
[XR] [OpenVR] [path] /home/nardi/VRTest2
[XR] [OpenVR] [Error] Action manifest file does not exist at path (/home/nardi/VRTest2/Assets//home/nardi/Build/VRTest2_Data/StreamingAssets/SteamVR/actions.jsonore)
[XR] [OpenVR] startupInfo: {
	"action_manifest_path": "/home/nardi/VRTest2/Assets//home/nardi/Build/VRTest2_Data/StreamingAssets/SteamVR/actions.jsonore"
}
...

And I noticed the incorrect path that seems to be a concatenation of two paths. It should be only /home/nardi/Build/VRTest2_Data/StreamingAssets/SteamVR/actions.jsonore. But I did not find a way to change this.
This seems to be the main issue because after some not-interesting debug lines there is a list of failures that probably are the reason why controllers are not spawning in the scene:
<b>[SteamVR]</b> GetInputSourceHandle (/unrestricted) error: NoData
UnityEngine.StackTraceUtility:ExtractStackTrace () (at /home/bokken/build/output/unity/unity/Runtime/Export/Scripting/StackTrace.cs:37)
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:LogError (object)
Valve.VR.SteamVR_Input_Source:Initialize () (at /home/nardi/VRTest2/Assets/SteamVR/Input/SteamVR_Input_Source.cs:70)
Valve.VR.SteamVR_Input:Initialize (bool) (at /home/nardi/VRTest2/Assets/SteamVR/Input/SteamVR_Input.cs:163)
Valve.VR.SteamVR:CreateInstance () (at /home/nardi/VRTest2/Assets/SteamVR/Scripts/SteamVR.cs:226)
Valve.VR.SteamVR:Initialize (bool) (at /home/nardi/VRTest2/Assets/SteamVR/Scripts/SteamVR.cs:122)
Valve.VR.SteamVR_Behaviour:InitializeSteamVR (bool) (at /home/nardi/VRTest2/Assets/SteamVR/Scripts/SteamVR_Behaviour.cs:119)
Valve.VR.SteamVR_Behaviour:Awake () (at /home/nardi/VRTest2/Assets/SteamVR/Scripts/SteamVR_Behaviour.cs:100)
...
other errors

After several tests I finally managed to make it work.
The issue seems to be related to the generation of SteamVR inputs (step 7 of previous comment).
WORKAROUND:
I generated the inputs before  opening the Interactions_Example and pressing Play.
If you directly press Play and then you accept the prompt for generating inputs you may face my issue.
My thought is that maybe there is a bug in the way paths to the actions.json are generated.

Hi, thanks for the report. Do you mind giving this a try with the latest beta? https://github.com/ValveSoftware/steamvr_unity_plugin/releases

I tested the last beta (v2.8.3) and indeed I faced no issue, furthermore the issue https://github.com/ValveSoftware/steamvr_unity_plugin/issues/1107 seems to be solved.
For anyone working in linux that is facing some problems, you can try following my steps:
My setup:

Unity 22.3.21f1
Ubuntu 22.04
SteamVR version 2.7.4 (beta)
ALVR v20.8.1

Steps:

Launch ALVR and connect your headset using SteamVR
Open a new 3D Core Project in Unity
Import the steam_vr custom package v2.8.3
Toggle Initialize XR on Startup and OpenVR Loader in Edit -> Project Settings -> XR Plugin Management
Set Stereo Rendering Module to Multipass and Mirror View Mode to Open VR in Edit -> Project Settings -> OpenVR
Switch Graphics API to Vulkan for Linux in Edit ->  Project Settings -> Player -> Rendering
Generate SteamVR Inputs through Window -> SteamVR Input -> Save and generate
Press play and check if the scene enter VR mode
Build app with File -> Build Settings -> Add Open Scenes then select Linux as Target Platform
Launch compiled AppImage

Hey @Hydran00 , how did you manage to even let it run at all? After importing SteamVR (from assets store), it warns me that recommended platform is windows, when I hit play in the editor it crashes

Hi @SinanAkkoyun. I have the same warning obviously, but I ignored it. Are you able to enter play mode without connecting the headset? Did you follow each of my steps in the previous comment? I need more details to be able to help you! 😀

Hi @Hydran00 , acutally you already helped me out, I just had a problem with switching to Vulkan but now it works 😊 Thank you!


static_dynamicCRT build still has dynamic link dependency for SDL2

Built what was basically the example project with the following dub.sdl on Linux Mint 20.3 (x64):

dub.sdl
name "imgui"
description "A minimal D application."
authors "<my name>"
copyright "Copyright © 2022, <my name>"
license "proprietary"
dependency "bindbc-loader" version="~>1.0.1"
dependency "bindbc-opengl" version="~>1.0.2"
dependency "bindbc-sdl" version="~>1.1.3"
dependency "bindbc-imgui" path="libs/bindbc-imgui"
subConfigurations "bindbc-imgui" "static_dynamicCRT"
subConfiguration "bindbc-sdl" "static"
versions "GL_33" "USE_SDL2" "USE_GL3"

This is the output of ldd imgui:

ldd output
	linux-vdso.so.1 (0x00007ffd4aa72000)
	libSDL2-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libSDL2-2.0.so.0 (0x00007f7309d78000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7309d72000)
	libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007f7309cb3000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7309a99000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7309a76000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7309927000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f730991b000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7309900000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f730970e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f730a167000)
	libasound.so.2 => /usr/lib/x86_64-linux-gnu/libasound.so.2 (0x00007f7309613000)
	libpulse.so.0 => /usr/lib/x86_64-linux-gnu/libpulse.so.0 (0x00007f73095be000)
	libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f730947f000)
	libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f7309468000)
	libXcursor.so.1 => /usr/lib/x86_64-linux-gnu/libXcursor.so.1 (0x00007f730945b000)
	libXinerama.so.1 => /usr/lib/x86_64-linux-gnu/libXinerama.so.1 (0x00007f7309456000)
	libXi.so.6 => /usr/lib/x86_64-linux-gnu/libXi.so.6 (0x00007f7309444000)
	libXrandr.so.2 => /usr/lib/x86_64-linux-gnu/libXrandr.so.2 (0x00007f7309437000)
	libXss.so.1 => /usr/lib/x86_64-linux-gnu/libXss.so.1 (0x00007f7309432000)
	libXxf86vm.so.1 => /usr/lib/x86_64-linux-gnu/libXxf86vm.so.1 (0x00007f7309429000)
	libwayland-egl.so.1 => /usr/lib/x86_64-linux-gnu/libwayland-egl.so.1 (0x00007f7309424000)
	libwayland-client.so.0 => /usr/lib/x86_64-linux-gnu/libwayland-client.so.0 (0x00007f7309413000)
	libwayland-cursor.so.0 => /usr/lib/x86_64-linux-gnu/libwayland-cursor.so.0 (0x00007f7309408000)
	libxkbcommon.so.0 => /usr/lib/x86_64-linux-gnu/libxkbcommon.so.0 (0x00007f73093c6000)
	libpng16.so.16 => /usr/lib/x86_64-linux-gnu/libpng16.so.16 (0x00007f730938c000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7309370000)
	libpulsecommon-13.99.so => /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-13.99.so (0x00007f73092ee000)
	libdbus-1.so.3 => /lib/x86_64-linux-gnu/libdbus-1.so.3 (0x00007f730929d000)
	libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f7309273000)
	libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007f7309069000)
	libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007f730905e000)
	libffi.so.7 => /usr/lib/x86_64-linux-gnu/libffi.so.7 (0x00007f7309052000)
	libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f7308fa3000)
	libwrap.so.0 => /usr/lib/x86_64-linux-gnu/libwrap.so.0 (0x00007f7308f97000)
	libsndfile.so.1 => /usr/lib/x86_64-linux-gnu/libsndfile.so.1 (0x00007f7308f19000)
	libasyncns.so.0 => /usr/lib/x86_64-linux-gnu/libasyncns.so.0 (0x00007f7308d11000)
	libapparmor.so.1 => /usr/lib/x86_64-linux-gnu/libapparmor.so.1 (0x00007f7308cfc000)
	libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f7308cf6000)
	libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f7308cee000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f7308cc5000)
	liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f7308ca2000)
	libgcrypt.so.20 => /usr/lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f7308b84000)
	libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007f7308b67000)
	libFLAC.so.8 => /usr/lib/x86_64-linux-gnu/libFLAC.so.8 (0x00007f7308b29000)
	libogg.so.0 => /usr/lib/x86_64-linux-gnu/libogg.so.0 (0x00007f7308b1c000)
	libvorbis.so.0 => /usr/lib/x86_64-linux-gnu/libvorbis.so.0 (0x00007f7308aee000)
	libvorbisenc.so.2 => /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2 (0x00007f7308a41000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f7308a25000)
	libbsd.so.0 => /usr/lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f7308a0b000)
	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f73089e8000)

Am I doing something wrong here? Thanks!

That's intentional, but I was/am planning to eventually fix this.
The idea of the static_* configs is for cimgui to be statically linked. As it's not provided by distros (or well configured for our needs in Inochi, and we had issues with dynamically linked libraries on Linux.
I believe the default config for SDL on bindbc-sdl is dynamically linked SDL (correct me if I'm wrong), so it's easier for us to match that with our C++ dependencies.
We are planning to switch everything but the C/C++ runtime libraries to be statically linked, but I'm not sure how that will change our configs (do we turn this into a triplet that includes the permutations of SDL/Freetype?)

I see. It seems desirable to be as statically-linked as possible! This is a dub question more than anything else - is my line subConfiguration "bindbc-sdl" "static" in my project's dub.sdl actually doing anything? Or would I have to edit this library's too?

Haha, that's the opposite of my expectation for most Linux programmers. I was under the impression most folks want their deps dynamically linked so their package managers handle them.
As for the sub configuration, here's where we start getting into headaches and part of why it's easier to link dynamically against our non-cimgui dependencies. When you specify that you want bindbc-sdl to be static, that will indeed do what you expect, specifically for those bindings. But bindbc-imgui needs to compile cimgui and imgui (the C++ dependencies) itself. That process doesn't know about the subconfiguration you've used for a different D dependency and uses whatever you've configured for bindbc-imgui, which currently doesn't provide any option other than dynamically linking.
Making sure the C++ libraries we depend on are the same as the C++ libraries other D libraries depend on are the same is tricky and not super clear how to do for every project. (Which I'm guessing is why most of the other D binding libraries probably default to linking against packaged stuff.)
That said, on Linux specifically, if we provided a config for "linking against SDL statically" you'd just need to pass that config and it'd be easy for all involved. We just don't have that right now. The first step was going to be compiling our own SDL and Freetype on Linux and figuring out how to make our AppImage happy with it, and then switch to statically linking.

Haha, that's the opposite of my expectation for most Linux programmers. I was under the impression most folks want their deps dynamically linked so their package managers handle them.

That's usually true, but I myself am super tired of the status quo of choosing between package manager hell or some heavy-handed packaging layer like AppImage, Flatpak, etc. Ideally, I'd like to learn how to make executables that I can copy between computers and have work (a tall order for modern Linux). Glibc and OpenGL as I understand probably have to stay dynamic for reasons... Though hopefully everything else should still be fine.

I hear you. I'll try to figure something out soon. The easiest thing would be something similar to what we did with cimgui in the dub.sdl
Would you also want us to build our own copy of SDL? That's what we do on Windows and Mac, but I'm not sure how we could make that work on Linux as an optional thing without adding even more subconfigs. (Beyond what we'd already need to add configs for statically linking to freetype and SDL.

Wait, maybe I should generate our dub.sdl 🤯

SDL2 is probably not a good idea to statically link either since it's meant to be dynamically linked and be able to automatically switch to newer versions of the library on load.
Each distro should be seen as a separate operating system and it's your choice as a developer to decide which of those you want to support. Alternatively you should go back to the POSIX roots of shipping software & their dependencies in /opt, with rpath set to load from your /opt directory or just give up and accept AppImage.
This is over-complicating bindbc-imgui for honestly quite bad reasons, I feel we should even reduce the amount of configurations we have currently, as some make very little sense in the grand scheme of things. (eg. static-static) Unless we want to turn this in to a mix of a bindbc, sdl and freetype binding.

SDL2 is probably not a good idea to statically link either since it's meant to be dynamically linked and be able to automatically switch to newer versions of the library on load.

According to this, it seems like it's certainly a valid strategy (even if not one they recommend).

Each distro should be seen as a separate operating system
Don't you think this is kind of insane though?? These "different OSs" with the same kernel and most likely a lot of the same base software on it (perhaps different versions) are unable to share executables without some costly abstraction, just because we didn't want to statically link anything? This divide is an issue between releases of distros too... The releases page for mGBA gets me every time for how ridiculous the multiple Ubuntu downloads look.

In any case, I'm not convinced the reasons are bad, but I do understand that this may increase complexity on your part. Thanks for maintaining these bindings at all - good luck at whatever you guys choose to do.

Don't you think this is kind of insane though?? These "different OSs" with the same kernel and most likely a lot of the same base software on it (perhaps different versions) are unable to share executables without some costly abstraction, just because we didn't want to statically link anything? This divide is an issue between releases of distros too... The releases page for mGBA gets me every time for how ridiculous the multiple Ubuntu downloads look.
In any case, I'm not convinced the reasons are bad, but I do understand that this may increase complexity on your part. Thanks for maintaining these bindings at all - good luck at whatever you guys choose to do.

I don't, they are simply separate systems. And the issue isn't that things aren't statically linked, but that we treat all software installed by the package manager as system software, instead of separating system and user applications sensibly, giving each application the dependencies they need (like on Windows or macOS) through rpath and /opt, which is a POSIX path specifically meant to house software that isn't system software, but which is very under-utilized.


Alias for compose

Make compose command line available with edocker commands wehnelt docker-compose.yaml is present

This issue was moved to pamtrak06/edjanger#14