added link to maclinux article

Updated install article to make it clear that it was about installing on Windows and linked to article for installing on macOS and Linux.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

azureps-cmdlets-docs/ResourceManager/docs-conceptual/4.3.1/TOC.md
:white_check_mark:Succeeded

azureps-cmdlets-docs/ResourceManager/docs-conceptual/TOC.md
:white_check_mark:Succeeded

azureps-cmdlets-docs/ResourceManager/docs-conceptual/install-azurerm-ps.md
:white_check_mark:Succeeded

azureps-cmdlets-docs/ResourceManager/docs-conceptual/install-azurermps-maclinux.md
:white_check_mark:Succeeded

azureps-cmdlets-docs/ResourceManager/docs-conceptual/other-install.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

@tarameyer #sign-off


storcon: skip draining shard if it's secondary is lagging too much

Problem
Migrations of tenant shards with cold secondaries are holding up drains in during production deployments.
Summary of changes
If a secondary locations is lagging by more than 256MiB (configurable, but that's the default), then skip
cutting it over to the secondary as part of the node drain.
Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] If it is a core feature, I have added thorough tests.
[ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
[ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

[ ] Do not forget to reformat commit message to not include the above checklist

There's a key design decision here: when we can't to a high quality live migration quickly, do we prefer to:
A) leave the attached location where it is
B) to cut it over anyway with a less-than-warm cache
This PR does A: I haven't thought about it deeply but would like to make sure we have a solid rationale.
Whether we prefer A or B, I think we need a way to tell live_migrate to give up more quickly than usual if it is doing a migration during a drain-for-restart: today, we have TOTAL_DOWNLOAD_TIMEOUT after which we cross out fingers and go ahead to cut over anyway.  We need this because even with the pre-check on pending downloads added in this PR, we can still end up hitting a download that takes a long time (because the pre-check is based on the most recent heatmap the secondary location saw)
TLDR: this PR isn't wrong, and covers the case of shards that are egregiously far behind on downloads (e.g. right after a shard split happens), but I think we need to invest a bit more time to figure out our total story

There's a key design decision here: when we can't to a high quality live migration quickly, do we prefer to: A) leave the attached location where it is B) to cut it over anyway with a less-than-warm cache

Indeed. To my mind a hybrid approach is the desired state:

If the secondary is lagging by a lot (more than 256MiB or 512MiB?) I'd say A is preferable since there's a big chance QOS will suffer more if we migrate. Restarts should be quick if the pageserver is not loaded (i.e. has drained most shards).
If the secondary is lagging by a solid amount (more than 64MiB), then B is preferable since we can soak up the cost of hydrating on the secondary.
If the secondary is lagging by a bit (less than 64MiB), then perhaps it makes sense to warm it up before cutting over

This PR implements the first bullet point.

Whether we prefer A or B, I think we need a way to tell live_migrate to give up more quickly than usual if it is doing a migration during a drain-for-restart: today, we have TOTAL_DOWNLOAD_TIMEOUT after which we cross out fingers and go ahead to cut over anyway. We need this because even with the pre-check on pending downloads added in this PR, we can still end up hitting a download that takes a long time (because the pre-check is based on the most recent heatmap the secondary location saw)

Yes, this makes sense. Will add.


SSM agent

Environment:

AWS Region: eu-north-1
Instance Type(s): t3.micro / t3.small
EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.7
Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.14
AMI Version: amazon-eks-node-1.14-v20190927 (ami-0c1dc1b58d2c9db01)
Kernel (e.g. uname -a): Linux ip-10-20-102-207.eu-north-1.compute.internal 4.14.146-119.123.amzn2.x86_64 #1 SMP Mon Sep 23 16:58:43 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Release information (run cat /etc/eks/release on a node):

cat: /etc/eks/release: No such file or directory

Would it make sence to include SSM agent by default? While these type of instances are more like cattle, it could be handy to be able to connect from time to time through Systems Manager Session. I want to avoid running ssh at the nodes

user data is not accessible when using node groups,  you can not specify AMI or anything when using managed node groups.

Seems like there should be two versions of the EKS AMI, one with the SSM agent enabled and one without. Seems like one additional jenkins job to make everyone happy.

@slimm609 Amazon Elastic Kubernetes Service (EKS) now supports EC2 Launch Templates and custom AMIs for managed node groups, which allows specifying user data.
See: https://aws.amazon.com/blogs/containers/introducing-launch-template-and-custom-ami-support-in-amazon-eks-managed-node-groups/

Hi all,
We've released the new EKS Optimized Amazon Linux AMI with the SSM agent pre-installed and enabled by default.
https://github.com/awslabs/amazon-eks-ami/releases/tag/v20210621


Failed to auth with captcha

Hi
Login with capthca failed wit
1:32PM INF Creating auth client
1:32PM INF Starting authentication process password=<my password> username=<my email>
1:32PM INF Got main page
1:32PM INF Got CSRF token token=2d39ec46711bd0c0e4559f64b3bc914a353ccfe66fe1d673ef76005425c1eaa6
1:32PM INF Got auth0 URL url=https://auth0.openai.com/authorize?client_id=<my client id>&scope=openid%20email%20profile%20offline_access%20model.request%20model.read%20organization.read&response_type=code&redirect_uri=https%3A%2F%2Fchat.openai.com%2Fapi%2Fauth%2Fcallback%2Fauth0&audience=https%3A%2F%2Fapi.openai.com%2Fv1&prompt=login&state=vTLekskUYyPyk4W27jNfoJTqncB2u7ZULlfrYmf2uc4&code_challenge=GZOSJW_18H1RthKYYmJx3bYi7V0TuF1LkPAssn49Jtg&code_challenge_method=S256
1:32PM INF Captcha detected
1:32PM INF Got auth0 authorization hasCaptcha=true
Captcha answer: rHgtQb
1:32PM INF Username sent
1:32PM INF Password sent
1:32PM INF Session resumed
{"level":"error","error":"authCallback: couldn't find token","time":"2022-12-11T13:32:55+08:00","message":"Failed to finish auth"}

The nexData is {"props":{"pageProps":{}},"page":"/auth/error","query":{},"buildId":"ShICingOrjj6GgETAxwpI","nextExport":true,"autoExport":true,"isFallback":false,"scriptLoader":[]}
OS: Linux x86_64
Commit: 7f01406196f55a30e2472656e8e1c69b7cc0c7b4

It looks like a network issue, when I change my proxy from HK to US, it woks.
Thank you for your greak work!


--harmony flag not needed for Node 0.12.0+

This line in node-harmonize says that the --harmony flag isn't needed for Node v0.12.0+.
Yet I am being told by Jest v0.3.0:
Error: Please run node with the --harmony flag!

Duplicate of #244.

Whoops, thanks!


Dynamic imports not splitting into different chunk for lazy loading

Version
3.0.0-beta.15
Reproduction link
https://github.com/pranavjindal999/beta15-repro
Steps to reproduce

Create a new Vue project using Vue Cli 3 (any version)
Make a dynamic import anywhere in the code. (using import('module-name') )
Build the project with Vue CLI-3.0.0.beta 12 or above

What is expected?
Dynamically imported module should split into a separate file and is not included in vendor bundle.
What is actually happening?
Dynamically imported module is included in vendor bundle and NO separate file is made.

This worked fine with Vue CLI 3.0.0-beta.11 and down.

Thanks for the quick fix 👍

Having this same issue in Vue CLI 4.3.1, suggestions?

Having this same issue in Vue CLI 4.3.1, suggestions?


Add Getting Started Notebook

The DCO is a pointless check. It maybe had some reasoning when it was introduced ~15 years ago, but security on github has made it a redundant 'feature'.
/soapbox
This version is a Colab friendly notebook that will help users get started with caikit-nlp. (That has the correct magic words in the commit message :roll_eyes: )
What does Colab have to do with it?
A user can click run in Colab, requisition a T4 GPU for free from google, and run the example.

For any reviewer that would rather look at the notebook rendering instead of the json rendering convenience link
https://github.com/rawkintrevo/caikit-nlp/blob/101-b/examples/Caikit_Getting_Started.ipynb


CS0103: Velocity doesn't exit in the current context

Godot version:
3.2.3 mono
OS/device including version:
Windows 10
Issue description:
I was following the step by step guide to making your first game in the engine (https://docs.godotengine.org/en/stable/getting_started/step_by_step/your_first_game.html) when I noticed visual studio 2019 was flagging a lot of errors regarding the usage of the term velocity. That has prevented me from successfully building the project. Here's my code so far for reference;
//Namespaces go here
using Godot;
using System;
using System.Collections;

public class Player : Area2D
{
    //Varibles
    [Export]
    public int speed = 400;

    private Vector2 Screensize;

    public override void _Ready()
    {
        Screensize = GetViewport().Size;
    }

    public override void _Process(float delta)
    {
       if (Input.IsActionPressed("ui_right"))
        {
            velocity.x += 1;
        }
        if (Input.IsActionPressed("ui_left"))
        {
            velocity.x -= 1;
        }
        if (Input.IsActionPressed("ui_up"))
        {
            velocity.y -= 1;
        }
        if (Input.IsActionPressed("ui_down"))
        {
            velocity.y += 1;
        }

        var animatedSprite = GetNode<AnimatedSprite>("AnimatedSprite");

        if (velocity.length() > 0)
        {
            velocity = velocity.Normalized() * speed;
            animatedSprite.Play();
        }
        else
        {
            animatedSprite.Stop();
        }

        Position += velocity * delta;
        Position = new Vector2(
            x: Mathf.Clamp(Position.x, 0, Screensize.y),
            y: Mathf.Clamp(Position.y, 0, Screensize.y)
         );
    }
}

I'm fairly certain that I spelt velocity correctly, but you never know. Should also note that I'm using a custom script template if that helps.

You did not define the velocity variable
{
    var velocity = new Vector2(); // <------ **** You do not have this line ****

    if (Input.IsActionPressed("ui_right"))
    {
        velocity.x += 1;
    }

    if (Input.IsActionPressed("ui_left"))
    {
        velocity.x -= 1;
    }

    if (Input.IsActionPressed("ui_down"))
    {
        velocity.y += 1;
    }

    if (Input.IsActionPressed("ui_up"))
    {
        velocity.y -= 1;
    }

    var animatedSprite = GetNode<AnimatedSprite>("AnimatedSprite");

    if (velocity.Length() > 0)
    {
        velocity = velocity.Normalized() * Speed;
        animatedSprite.Play();
    }
    else
    {
        animatedSprite.Stop();
    }
}

Oh, and here I was thinking this thing was built into the Godot namespace. Thank you!

Closing per @EricEzaM's comment.


Ignore existing prebackup deployments

Summary
This should ignore AlreadyExists errors when creating prebackup deployments.
Fixes #246
Checklist

[x] Keep pull requests small so they can be easily reviewed.
[x] Categorize the PR by setting a good title and adding one of the labels:
bug, enhancement, documentation, change, breaking,
as they show up in the changelog
[x] Link this PR to related issues.

Note that, as long as prebackup deployments exist, they won't be updated. Should we add a prebackup controller that updates existing deployments upon changes?

Should we add a prebackup controller that updates existing deployments upon changes?

Afaik in pre 1.0 of K8up this wasn't available?
It might make sense to have that in a future K8up release, yes.

Afaik in pre 1.0 of K8up this wasn't available?

I don't think so, but in https://github.com/vshn/k8up/issues/246#issuecomment-752009165 I understood we should be addressing this?

@Kidswiss I'm getting lots of these after applying the fix (K8s 1.18)
2020-12-30T14:24:02.043Z	INFO	controllers.Backup	adding job to the queue	{"backup": "default/backupjob-1609338240"}
2020-12-30T14:24:02.044Z	INFO	controllers.Backup	starting pre backup pod	{"backup": "default/backupjob-1609338240", "namespace": "default", "name": "mysqldump"}
2020-12-30T14:24:02.046Z	INFO	controllers.Backup	starting pre backup pod	{"backup": "default/backupjob-1609338240", "namespace": "default", "name": "mysqldump"}
2020-12-30T14:24:02.051Z	INFO	controllers.Backup	unexpected event during deployment watch 	{"backup": "default/backupjob-1609338240", "name": "mysqldump", "event type": "ADDED", "namespace": "default"}
2020-12-30T14:24:02.052Z	INFO	controllers.Backup	unexpected event during deployment watch 	{"backup": "default/backupjob-1609338240", "name": "mysqldump", "event type": "ADDED", "namespace": "default"}
2020-12-30T14:24:02.080Z	INFO	controllers.Backup	starting pre backup pod	{"backup": "default/backupjob-1609338240", "namespace": "default", "name": "mysqldump"}
2020-12-30T14:24:02.085Z	INFO	controllers.Backup	unexpected event during deployment watch 	{"backup": "default/backupjob-1609338240", "name": "mysqldump", "event type": "ADDED", "namespace": "default"}
2020-12-30T14:24:02.112Z	INFO	controllers.Backup	starting pre backup pod	{"backup": "default/backupjob-1609338240", "namespace": "default", "name": "mysqldump"}
2020-12-30T14:24:02.119Z	INFO	controllers.Backup	unexpected event during deployment watch 	{"backup": "default/backupjob-1609338240", "name": "mysqldump", "event type": "ADDED", "namespace": "default"}
2020-12-30T14:24:02.145Z	INFO	controllers.Backup	starting pre backup pod	{"backup": "default/backupjob-1609338240", "namespace": "default", "name": "mysqldump"}
2020-12-30T14:24:02.151Z	INFO	controllers.Backup	unexpected event during deployment watch 	{"backup": "default/backupjob-1609338240", "name": "mysqldump", "event type": "ADDED", "namespace": "default"}

Is that expected/ok? (Note: The backup jobs are all failing, since the they try to start the default "myprojec/restic" image)
To reproduce:
make e2e_test
k apply -f config/samples/k8up_v1alpha1_prebackuppod.yaml
k apply -f config/samples/k8up_v1alpha1_schedule.yaml

@ccremer I'd say we can change the code a bit so that the ADDED events aren't unexpected anymore. I haven't seen them before.
https://github.com/vshn/k8up/blob/master/prebackup/prebackup.go#L199

Note that, as long as prebackup deployments exist, they won't be updated. Should we add a prebackup controller that updates existing deployments upon changes?

I'd suggest to move discussions about improvements for the pre-backuppod stuff here: https://github.com/vshn/k8up/issues/137
I'm not so happy with the current implementation. It's more or less the same logic as legacy K8up uses and has a few flaws. For example if two pods are running at the same time in the same namespace and start pre-backuppods, the backup that finishes first will kill the deployments and could interrupt the second backup.

Regarding failing codeclimate check:
I have tested the fix with e2e testing tools locally, which isn't covered in the coverage.
I propose to skip a pseudo-test (a test just to increase coverage) if we think to overhaul prebackups anyway.


[upgrade] fix 3616 another operation is in progress

  the fleet-agent takes time to apply changes to downstream objects
  restart of fleet-agent may cause data inconsistency
  add workaroud of helm rollback to sovle the issue

IMPORTANT: Please do not create a Pull Request without creating an issue first.
Problem:

  the `fleet-agent` takes time to apply changes to downstream objects
  `upgrade_os` restarts fleet-agent forcely and causes inconsistency

Solution:

before upgrade_os, wait all bundles to be ready
Related Issue:
https://github.com/harvester/harvester/issues/3616
candidate PR: https://github.com/harvester/harvester/pull/3641
Test plan:
manual test:
(1) kubectl edit managedchart -n fleet-local rancher-monitoring to set a new resource limit for grafana POD
(2) check, when the bundle is in WaitApplied(1) [Cluster fleet-local/local], and after 10 seconds, kill the fleet-agent process
(3) easily, the bundle is in status ErrApplied(1) [Cluster fleet-local/local: another operation (install/upgrade/rollback) is in progress]

(4) run test shell script in one machine, with kubeconfig file and helm
gh-issue-3616-independent-run-script.txt
the log looks like:
gh-issue-3616-helm-rollback-test-result.log
(5) the bunlde is rolled-back
system test:
build-iso, run single-node upgrade, from v1.1.1 to master-head

single-node harvester cluster, upgrade from v1.1.1 to master-head
reference validation: https://github.com/harvester/harvester/issues/3616#issuecomment-1466104172

@w13915984028 I use this approach in my first implementation of https://github.com/harvester/harvester/pull/3641 but found an issue:

Sometimes the monitoring stack pods (alertmanager, prometheus, grafana) will become "creating". (Thus rancher-monitoring chart will not become ready and will block wait_for_fleet_bundles ). I don't know the reason yet but seems to be related to the RKE2 upgrade as well, losing APIs or something.

That's the reason I move the fleet-agent restore code to the controller (after reboot).

Observed one error in local test:
convert_nodenetwork_to_vlanconfig -> detect_upgrade, which fails due to API server is not ready/busy,
will also add enhancement
+ convert_nodenetwork_to_vlanconfig
+ detect_upgrade
++ kubectl get upgrades.harvesterhci.io hvst-upgrade-zv5zr -n harvester-system -o yaml
The connection to the server <IP_ADDRESS>:443 was refused - did you specify the right host or port?
+ upgrade_obj=
[Error] Running "upgrade_node.sh single-node-upgrade"
---loop again---

With more tests, we found another issue:
In apply-manifests, the upgrade of fleet-agent itself, may also cause another operation is in progress.
cc @bk201, @guangbochen
NAME                           READY   STATUS    RESTARTS      AGE
fleet-agent-86dc84bdb7-pvw8r   1/1     Running   2 (16m ago)   3d3h
...
...
Tue Mar 14 15:15:29 UTC 2023
---get bundle&pod information
NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local                             0/1                       WaitApplied(1) [Cluster fleet-local/local]
fleet-local   local-managed-system-agent                    0/1                       WaitApplied(1) [Cluster fleet-local/local]
fleet-local   mcc-harvester                                 1/1
fleet-local   mcc-harvester-crd                             1/1
fleet-local   mcc-local-managed-system-upgrade-controller   0/1                       WaitApplied(1) [Cluster fleet-local/local]
fleet-local   mcc-rancher-logging                           0/1                       WaitApplied(1) [Cluster fleet-local/local]
fleet-local   mcc-rancher-logging-crd                       1/1
fleet-local   mcc-rancher-monitoring                        1/1
fleet-local   mcc-rancher-monitoring-crd                    1/1
NAME                           READY   STATUS              RESTARTS      AGE
fleet-agent-6ffcdc7f94-7w4wj   0/1     ContainerCreating   0             2s
fleet-agent-86dc84bdb7-pvw8r   1/1     Running             2 (16m ago)   3d3h

cattle-system               apply-hvst-upgrade-n6hwn-prepare-on-harv31-with-d7f377ddf-7lfmv   0/1     Completed           0               8m17s
cattle-system               system-upgrade-controller-79fc9c84b7-gskwl                        1/1     Running             2 (16m ago)     3d3h
harvester-system            hvst-upgrade-n6hwn-apply-manifests-df2pg                          1/1     Running             0               62s
harvester-system            virt-launcher-upgrade-repo-hvst-upgrade-n6hwn-cg69s               1/1     Running             0               9m15s
---check pod to fetch log
pod not find yet
---loop again---

Tue Mar 14 15:15:29 UTC 2023

Tue Mar 14 15:15:39 UTC 2023
---get bundle&pod information
NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local                             0/1                       ErrApplied(1) [Cluster fleet-local/local: another operation (install/upgrade/rollback) is in progress]
fleet-local   local-managed-system-agent                    1/1
fleet-local   mcc-harvester                                 1/1
fleet-local   mcc-harvester-crd                             1/1
fleet-local   mcc-local-managed-system-upgrade-controller   1/1
fleet-local   mcc-rancher-logging                           1/1
fleet-local   mcc-rancher-logging-crd                       1/1
fleet-local   mcc-rancher-monitoring                        1/1
fleet-local   mcc-rancher-monitoring-crd                    1/1
NAME                           READY   STATUS    RESTARTS   AGE
fleet-agent-6ffcdc7f94-7w4wj   1/1     Running   0          13s

fleet-agent itself has this error now, and the fleet-agent POD is just replaced

Run 4 more times of upgrade test, all are successful.

A general question: is there any chance to wait for all the bundles to settle down in upgrade_manifest.sh? This way, we might be able to improve the overall upgrade stability for both single-node and multi-node scenarios.

fleet-agent detects tons of related objects dynamically, there is no clear law that when a managedchart/bundle will be in non-ready and a new update will be started.
the wait in upgrade_manifest.sh means at that moment, it was OK, but does not mean later it will still be OK

e.g.  one of the upgrade log shows:
from  upgrade_rke2 -> convert_nodenetwork_to_vlanconfig, the bundle changes from ready to non-ready

upgrade_rke2
check bundle 22

NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local                             1/1
fleet-local   local-managed-system-agent                    1/1
fleet-local   mcc-harvester                                 1/1
fleet-local   mcc-harvester-crd                             1/1
fleet-local   mcc-local-managed-system-upgrade-controller   1/1
fleet-local   mcc-rancher-logging                           1/1
fleet-local   mcc-rancher-logging-crd                       1/1
fleet-local   mcc-rancher-monitoring                        1/1
fleet-local   mcc-rancher-monitoring-crd                    1/1

Waiting for RKE2 to be upgraded...
NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local                             0/1                       NotReady(1) [Cluster fleet-local/local]; deployment.apps cattle-fleet-local-system/fleet-agent [progressing] Deployment does not have minimum availability., Available: 0/1
fleet-local   local-managed-system-agent                    1/1
fleet-local   mcc-harvester                                 0/1                       NotReady(1) [Cluster fleet-local/local]; daemonset.apps harvester-system/harvester-network-controller [progressing] Available: 0/1; daemonset.apps harvester-system/harvester-node-manager [progressing] Available: 0/1; daemonset.apps harvester-system/kube-vip [progressing] Available: 0/1; deployment.apps harvester-system/harvester-load-balancer [progressing] Deployment does not have minimum availability., Available: 0/1
fleet-local   mcc-harvester-crd                             1/1
fleet-local   mcc-local-managed-system-upgrade-controller   0/1                       NotReady(1) [Cluster fleet-local/local]; deployment.apps cattle-system/system-upgrade-controller [progressing] Deployment does not have minimum availability., Available: 0/1
fleet-local   mcc-rancher-logging                           0/1                       NotReady(1) [Cluster fleet-local/local]; daemonset.apps cattle-logging-system/rancher-logging-rke2-journald-aggregator [progressing] Available: 0/1; deployment.apps cattle-logging-system/rancher-logging [progressing] Deployment does not have minimum availability., Available: 0/1
fleet-local   mcc-rancher-logging-crd                       1/1
fleet-local   mcc-rancher-monitoring                        0/1                       NotReady(1) [Cluster fleet-local/local]; apiservice.apiregistration.k8s.io v1beta1.custom.metrics.k8s.io [progressing] endpoints for service/rancher-monitoring-prometheus-adapter in "cattle-monitoring-system" have no addresses with port name "", Resource is current; daemonset.apps cattle-monitoring-system/rancher-monitoring-prometheus-node-exporter [progressing] Available: 0/1; deployment.apps cattle-monitoring-system/rancher-monitoring-kube-state-metrics [progressing] Deployment does not have minimum availability., Available: 0/1; deployment.apps cattle-monitoring-system/rancher-monitoring-operator [progressing] Deployment does not have minimum availability., Available: 0/1
fleet-local   mcc-rancher-monitoring-crd                    1/1
+ kubectl get pods -n cattle-fleet-local-system -o wide
NAME                           READY   STATUS    RESTARTS   AGE     IP          NODE     NOMINATED NODE   READINESS GATES
fleet-agent-6ffcdc7f94-g9fkm   1/1     Running   0          4m52s   <IP_ADDRESS>   harv31   <none>           <none>

convert_nodenetwork_to_vlanconfig

Another tricky is, fleet-agent is upgrading itself, uses a new deployment to replace current, and its POD is replaced
We observed: fleet-agent is upgrading a bunch of bundles, including itself, and there are 2 fleet-agent PODs running (there is leader selection )
it is possbile that the 30seconds of (terminationGracePeriodSeconds: 30) termination is not enough, the old fleet-agent has no chance to revert the mark in helm releases, wich causes the another operation is in progress.
Tue Mar 14 14:39:30 UTC 2023
---get bundle&pod information
NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local                             0/1                       WaitApplied(1) [Cluster fleet-local/local]
fleet-local   local-managed-system-agent                    0/1                       WaitApplied(1) [Cluster fleet-local/local]
fleet-local   mcc-harvester                                 1/1
fleet-local   mcc-harvester-crd                             1/1
fleet-local   mcc-local-managed-system-upgrade-controller   0/1                       WaitApplied(1) [Cluster fleet-local/local]
fleet-local   mcc-rancher-logging                           0/1                       WaitApplied(1) [Cluster fleet-local/local]
fleet-local   mcc-rancher-logging-crd                       1/1
fleet-local   mcc-rancher-monitoring                        1/1
fleet-local   mcc-rancher-monitoring-crd                    1/1

** two fleet-agent PODs, due to `type: RollingUpdate`

NAME                           READY   STATUS        RESTARTS      AGE
fleet-agent-6ffcdc7f94-6p44n   1/1     Running       0             4s
fleet-agent-86dc84bdb7-pvw8r   1/1     Terminating   3 (16m ago)   3d2h

In one single-node cluster upgrade, the fleet-agent itself is upgraded 7 times (from the helm history), I did not dive too much of this, but looks not so good
rancher@wang:~$ helm history fleet-agent-local -n cattle-fleet-local-system
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/rancher/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure.  This is insecure. Location: /home/rancher/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/rancher/.kube/config
REVISION	UPDATED                 	STATUS    	CHART                	APP VERSION	DESCRIPTION
1       	Sat Mar 11 12:13:22 2023	superseded	fleet-100.1.0+up0.4.0	0.4.0      	Install complete
2       	Sat Mar 11 12:17:29 2023	superseded	fleet-100.1.0+up0.4.0	0.4.0      	Upgrade complete

3       	Tue Mar 14 20:21:26 2023	superseded	fleet-100.1.0+up0.4.0	0.4.0      	Upgrade complete
4       	Tue Mar 14 20:21:38 2023	deployed  	fleet-100.1.0+up0.4.0	0.4.0      	Upgrade complete

To now, the futher workaround will be in this direction:  detect the bundle conflict error and helm rollback it
--
could fleet-agent use more robust label against helm releases, thus new fleet-agent can take over the ongoing upgrade or use helm rollback to recover self, given the previous label is from this branch of fleet-agent?
the current implementaion simply label the release, to act as a lock, but seems fragile
https://github.com/rancher/fleet/issues/637#issuecomment-1469787719

To verify the helm rollback workaround, we reproduce the issue by:
1. kubect edit managedchart -n fleet-local mcc-rancher-monitoring
2. wait the bundle to be `WaitApplied`
3. kill the fleetagent process

get the error: ErrApplied(1) [Cluster fleet-local/local: another operation (install/upgrade/rollback) is in progress]

@bk201 @starbops please help validate the latest commit when you are free, thanks.

Unfortunately, the current solution may still not enough:
Did some single-node upgrade tests, found:
In a successful upgrade, after system rebooting, some bundle is still in another operation (install/upgrade/rollback) is in progress
harv31:/home/rancher # kk get upgrade.harvesterhci.io -A
NAMESPACE          NAME                 AGE
harvester-system   hvst-upgrade-dchkw   26m
..
    generateName: hvst-upgrade-
    generation: 13
    labels:
      harvesterhci.io/latestUpgrade: "true"
      harvesterhci.io/upgradeState: Succeeded
    - lastUpdateTime: "2023-03-16T10:16:12Z"
      status: "True"
      type: SystemServicesUpgraded
    - lastUpdateTime: "2023-03-16T10:24:33Z"
      status: "True"
      type: NodesUpgraded

cattle-system               apply-hvst-upgrade-dchkw-prepare-on-harv31-with-7a87877d5-sn9k6   0/1     Completed   0                22m
cattle-system               system-upgrade-controller-79fc9c84b7-gskwl                        1/1     Running     3 (6m ago)       4d22h
harvester-system            hvst-upgrade-dchkw-apply-manifests-x4llx                          0/1     Completed   0                14m
harvester-system            hvst-upgrade-dchkw-single-node-upgrade-harv31-zvv78               0/1     Completed   0                11m

harv31:/home/rancher # kk get bundle -A
NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local                             1/1
..
fleet-local   mcc-rancher-logging                           1/1
fleet-local   mcc-rancher-logging-crd                       0/1                       ErrApplied(1) [Cluster fleet-local/local: another operation (install/upgrade/rollback) is in progress]

harv31:/home/rancher # kk get pods -A
NAMESPACE                   NAME                                                              READY   STATUS      RESTARTS        AGE
cattle-fleet-local-system   fleet-agent-6ffcdc7f94-5fw5l                                      1/1     Running     0               4m9s
cattle-fleet-system         fleet-controller-5d546cc5b8-zbjwv                                 1/1     Running     2 (7m13s ago)   14m
..
cattle-logging-system       harvester-default-event-tailer-0                                  1/1     Running     3 (7m12s ago)   4d22h

The possible reason is:
  wait_for_fleet_bundles
  # scale down fleet-agent, scale up will be done in upgrade controller
  echo "scale down fleet-agent"
  kubectl scale --replicas=0 deployment/fleet-agent -n cattle-fleet-local-system
  kubectl rollout status deployment fleet-agent -n cattle-fleet-local-system

-> to now, all bundles are ready, and fleet-agent is scaled down
-> But in some steps follwing, the fleet-agent is scaled up during upgrade, and it starts to apply change for some bundle, and the POD is further killed/exit itself due to api-server interruption, it stucks in middle
  # Upgarde RKE2
  upgrade_rke2

  wait_rke2_upgrade
  clean_rke2_archives
  upgrade_os

As long as we did enough times of test, we can encounter the issue in/after upgrade.
If fleet-agent cannot recover itself, it is hard to path somewhere.  Or we need a job to watch this all the time and try to roll back...

Test log at 2023.03.17 12:00 (CEMT), the log shows wait for fleet bundle and scale down fleet-agent, meets our expectation
@bk201 @guangbochen
manifest upgrade:
addon.harvesterhci.io/vm-import-controller patched
addon.harvesterhci.io/pcidevices-controller patched

Fri Mar 17 11:32:28 UTC 2023
NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local                             1/1
fleet-local   local-managed-system-agent                    1/1
fleet-local   mcc-harvester                                 1/1
fleet-local   mcc-harvester-crd                             1/1
fleet-local   mcc-local-managed-system-upgrade-controller   1/1
fleet-local   mcc-rancher-logging                           1/1
fleet-local   mcc-rancher-logging-crd                       1/1
fleet-local   mcc-rancher-monitoring                        1/1
fleet-local   mcc-rancher-monitoring-crd                    1/1
all bundles are ready

node upgrade:
Fri Mar 17 11:33:15 UTC 2023
+ kubectl get bundle.fleet.cattle.io -A
NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local                             1/1
fleet-local   local-managed-system-agent                    1/1
fleet-local   mcc-harvester                                 1/1
fleet-local   mcc-harvester-crd                             1/1
fleet-local   mcc-local-managed-system-upgrade-controller   1/1
fleet-local   mcc-rancher-logging                           1/1
fleet-local   mcc-rancher-logging-crd                       1/1
fleet-local   mcc-rancher-monitoring                        1/1
fleet-local   mcc-rancher-monitoring-crd                    1/1
++ kubectl get bundle.fleet.cattle.io -A -o 'jsonpath={.items[*].status.conditions[?(@.type=="Ready")].status}'
+ bsr='True True True True True True True True True'
+ '[' 0 '!=' 0 ']'
+ [[ True True True True True True True True True == *\F\a\l\s\e* ]]
+ echo 'all bundles are ready'
+ return
all bundles are ready
+ echo 'scale down fleet-agent'
+ kubectl scale --replicas=0 deployment/fleet-agent -n cattle-fleet-local-system
scale down fleet-agent
deployment.apps/fleet-agent scaled
+ kubectl rollout status deployment fleet-agent -n cattle-fleet-local-system
deployment "fleet-agent" successfully rolled out
+ upgrade_rke2


Atlas of the Aging Mouse Brain, 'mature neuron' and 'immature neuron' needed?

Hello CL Team,
Atlas of the Aging Mouse Brain brings up some new concepts/terms that are currently missing in CL https://singlecell.broadinstitute.org/single_cell/study/SCP263/aging-mouse-brain#study-summary
adult neurogenesis with concept of 'mature' and 'immature' neurons sounds emerging..?
https://www.pennmedicine.org/news/news-releases/2022/july/discovery-of-molecular-signatures-of-immature-neurons-in-the-human-brain
Thank you for your help here!
Anne

Hi Anne,
Thank you for your ticket, I will get back to you as soon as possible to incorporate these new concepts in CL.
Alida

@JABelfiore  please review:
Given the following literature research (see list of papers below), the concept of immature neurons should be included in the ontology as a cell classification as these cells remain in this state for a prolonged time in the tissue and therefore can be identified.
PMID: 31096632
PMID: 37833544
PMID: 32116519
https://molecularbrain.biomedcentral.com/articles/10.1186/s13041-019-0522-8
https://pubmed.ncbi.nlm.nih.gov/21609825/
The following NTRs will be added:
cell label: immature neuron
cell definition:
A neuron in a central nervous system that is not committed to a differentiated fate, that it is not been newly derived from neurogenesis, and that does not integrate within any circuit.
database_cross_reference:

PMID: 37833544
PMID: 32116519
PMID: 31096632

comments:
It is hypothesised that immature neurons can be utilised as a neurogenic reserve and that they can be involved in circuit plasticity.
database_cross_reference:
PMID: 32116519
has related synonym:
IN

PMID: 32116519

has related synonym:
nng-IN
PMID: 31096632
parent term:
central nervous system neuron
uberon term:
central nervous system
cell label: cortical immature neuron
definition:
An immature neuron of a cerebral cortex.
database_cross_reference:

PMID: 37833544

has related synonym:
cIN
database_cross_reference:

PMID: 37833544

parent term: immature neuron
uberon term: cerebral cortex

I would not add a new term just because of its anatomic location, unless there is some evidence that it has a different molecular or functional property in this location.

Dear @scheuerm, thank you for your comment, I will have a look regarding the cortical immature neuron, but will keep the immature neuron NTR


Luzerne County, PA dropped cities

As of 9/2/22 Luzerne County, PA is missing cities (see history)

The current source:
https://gis.luzernecounty.org/arcgis/rest/services/PublicMap/MapServer/8
is no longer accessible. There's a parcel viewer here:
https://luzernegis.maps.arcgis.com/apps/webappviewer/index.html?id=80d1e82579f4431490f0596af9c379d0
that pulls in this address layer:
https://luzernegis.maps.arcgis.com/apps/webappviewer/index.html?id=80d1e82579f4431490f0596af9c379d0
and this parcel layer:
https://services7.arcgis.com/hzCSL69lejmynMEC/arcgis/rest/services/Parcels_032223_PUBLIC/FeatureServer/0


Add helper, filter, and tags to prevent s390x ipv6 tests from running

Description
ipv6 is disabled on the builder hosts for s390x hosts due to connection stability errors in the environment.

add a platform helper to flag when RUBY_PLATFORM contains s390x
add a filter tag :not_supported_on_s390x
add tags to the ipv6 tests of concern

internal adhoc build to validate
Related Issue

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Chore (non-breaking change that does not add functionality or fix an issue)

Checklist:

[ ] I have read the CONTRIBUTING document.
[ ] I have run the pre-merge tests locally and they pass.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] If Gemfile.lock has changed, I have used --conservative to do it and included the full output in the Description above.
[ ] All new and existing tests passed.
[X] All commits have been signed-off for the Developer Certificate of Origin.

superseded by #14433 changes


Convert from TRK to JSON to create bundles.json data

For each subject:
{bundleID_1: n_1 by 100 by 3 array,
bundleID_2: n_2 by 100 by 3 array,
...}

This might be something for https://github.com/yeatmanlab/pyAFQ


Testcafe can't establish connection with browser when running in CI

Are you requesting a feature or reporting a bug?
Bug
What is the current behavior?
The test running can't connect to browser when executed in node (dockerized with chrome installed from apt, extended from ¡ node:8.11.1 ).
ERROR Unable to establish one or more of the specified browser connections. This can be caused by network issues or remote device failure.
What is the expected behavior?
Test expected to be functional same as locally
How would you reproduce the current behavior (if this is a bug)?
By running test suite in bash docker container.
Specify your

operating system: Debian
testcafe version: 0.21.1
node.js version: 8.11.1

This is my DockerFile configuration
`FROM node:8.11.1 as base
WORKDIR /opt/app
USER root
RUN apt-get update -y &&
apt-get install -y
libseccomp2 python-openssl
gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 \
libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 \
libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 \
libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libappindicator3-1 libnss3 lsb-release xdg-utils wget &&
rm -rf /var/lib/apt/lists/*
RUN wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb &&
dpkg -i google-chrome*.deb
FROM base as build
ARG ENV=$ENV
COPY ./ ./
RUN npm install
RUN npm run build
FROM base
ARG ENV=$ENV
COPY .npmrc package*.json yarn.* newrelic.js process.yml start.sh ./
COPY --from=build /opt/app/build ./build
RUN chmod -R ugo+rw /opt/app
RUN chmod +x start.sh
RUN npm install -g<EMAIL_ADDRESS>RUN npm install -g testcafe
RUN npm install --production
EXPOSE 5000
ENTRYPOINT ["./start.sh"]`

I'll try with the flag "--no-sandbox" like chromium but not working

HI @AndreyBelym i have a js for exec my test with child process "exec"
testcafe path:/path/to/my/sh ./mytest.js
and my .sh like:
#!/bin/sh Xvfb :19 -screen 0 "1024x768x16" >/dev/null 2>&1 & export DISPLAY=:19 firefox
and throw... ERROR Unable to establish one or more of the specified, i changed my docker file:
`FROM alpine:edge
WORKDIR /opt/app
ARG ENV=$ENV
RUN apk --no-cache --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ add
python make g++ nodejs nodejs-npm chromium firefox xwininfo xvfb dbus eudev ttf-freefont fluxbox bash`
thanks for your answer

Hi @davidarce
Was this issue fixed and closed? If so can you please let me know what was done to fix it?

Hi @melwynjensen
Yes, it was fixed, the solution was in the start.sh, this is my start.sh
dbus-daemon --session --fork     Xvfb :1 -screen 0 "1280x720x24" >/dev/null 2>&1 &     export DISPLAY=:1.0     fluxbox >/dev/null 2>&1 &     exec testcafe "$@"

Hi @davidarce
Thanks. But in my case am using headless chromium and firefox but get the same error which you have mentioned not really sure how to fix that.

Hi everyone,
I am facing the same issue when trying to run my tests in Jenkins CI using firefox browser.
Here is the log from Jenkins job:
Switched to a new branch 'dev'
Branch dev set up to track remote branch dev from origin.
<EMAIL_ADDRESS>/mnt/data/jenkins/jobs/e2e-tests/workspace/tests
├──<EMAIL_ADDRESS>└──<EMAIL_ADDRESS>
npm WARN<EMAIL_ADDRESS>No description
npm WARN<EMAIL_ADDRESS>No repository field.
Using locally installed version of TestCafe.
ERROR Unable to establish one or more of the specified browser connections. This can be caused by network issues or remote device failure.

Type "testcafe -h" for help.
Build step 'Execute shell' marked build as failure
Finished: FAILURE

UPDATE ^^
Sorry for spam here ;)
Just found the older answer for this issue from @AndreyBelym  and it solved the error.
Hope this helps:
https://github.com/DevExpress/testcafe/issues/1133#issuecomment-353849065


Add RLWE encoding attributes

Adds three more encodings:

PolyCoefficientEncoding, for CGGI's bootstrapping rotation polynomial encoding
PolyEvaluationEnvoding, for standard BGV/BFV encodings
PolyRoundedEmbeddingEncoding, for CKKS's standard encoding

cc @b5li just wanted to CC you on the work Jeremy's doing here on encodings for LWE/RLWE schemes

Note the website build is breaking due to https://github.com/llvm/llvm-project/commit/d339d8fede356f0acfad6d6c1bb960269f2467a9#commitcomment-128747641
Added the custom shortcode needed to this PR.


Implement workaround for sharing views in terraform configuration

Implement the role grants needed to share views in our default terraform configuration, cf the conversation in #273. Copying the probable way forward here (option 1)

Okay, so after some more conversation, I'm more-or-less convinced that the same solution as above is the least-bad option.
The basic problem is that functional vs access roles is an abstraction, and a leaky one. This is a case where we cannot maintain the purity of access roles. In order for a view to work, the view owner must have read access to the underlying table. If that underlying table exists in a different database or schema than the view owner can read, then the view will fail. This is despite the fact that view users only need to have SELECT access on the view, rather than the underlying table. This is described in more detail here. It's an annoying limitation, but appears to be expected behavior.
The problem we run into is that the owner of a view is the relevant access role for the database/schema (i.e., the read/write/control role). But if the view references a table in another database/schema, it will fail because, by design, the access role only has access to its own object. There appears to be no way around this, except for the following two compromises:

Grant TRANSFORM_READ to ANALYTICS_READWRITECONTROL. This gives the view owner read on the underlying db. Note that ANALYTICS_READ, and thus the dashboarding users, would still not have access. Though it does break the purity of access roles.
Create a new functional role for dashboarding users that has read access to both ANALYTICS and TRANSFORM. If there is any pre-aggregated sensitive data in TRANSFORM, then the dashboard users would theoretically have access.

Of these two options, I prefer one. This is just a place where the access vs functional role abstraction leaks.

This should be completed before we need to spin up an account for HFA

Hi @ian-r-rose, Thank for providing the background information. I agree with option #1 too.
I will go ahead an update ANALYTICS_READWRITECONTROL to grant TRANSFORM_READ for development and production environments

Change has been applied to both development and production environments successfully. Hence closing the issue


when we will have a start doc?

What I do:

git clone Flutter4GitLab
modify const.dart

KEY_ACCESS_TOKEN = 'xxxxxxx';
KEY_HOST = '<IP_ADDRESS>'; // <IP_ADDRESS> is also not work

flutter run

What is problem:
console is show (This is taking an unexpectedly long time.) for a long time.
in a word:
I think you need give a start doc

Thanks for your advice.
KEY_ACCESS_TOKEN and KEY_HOST are keys for store value in local storage. (NSUserDefaults on iOS and SharedPreferences on Android). You should not modify them.
To config your gitlab token and host, just running this app and you will see config entrance in home page.

@huang12zheng Doc Updated -> https://github.com/stefanJi/Flutter4GitLab#usage

thanks


The entity number exceed

When I used de.tudarmstadt.ukp.wikipedia.datamachine-1.0.0-jar-with-dependencies to parse chinese wiki dumps, it reported that the entity number exceeded 50000000. Could anyone solve this problem?

This is a known issue - see here:
https://github.com/dkpro/dkpro-jwpl/issues/144


add: postcss logo

Not sure if Compiler is the proper category.
See https://github.com/postcss/brand/tree/master for logo source and license

Thanks 🚀🚀

I further reduced the number precision in SVGOMG to reduce the size of the wordmark version.
Checks are passing now and visually its still fine.

Thanks 🚀


Data Mesh Implementation with dbt Cloud

Hi @iamontheinet can you please review this on Monday? This is for MD235 during Snowflake Summit so we need to get this in. Thanks 🙏

@dpguthrie Thanks for creating this QS. Can you pl incorporate the following:

Rename the first step to "Overview" step and include "Prerequisites", "What You'll Learn" and "What You'll Build" as part of the Overview step. So basically then the "About this guide" step goes away
Rename the last step to "Conclusion and Resources" and include links to resources.

Refer to this QS for layout and format -- https://quickstarts.snowflake.com/guide/getting_started_with_dataengineering_ml_using_snowpark_python/index.html#0

Hey @iamontheinet, we're ready for a re-review. Can you please do this as soon as you can? We need this merged in by Friday for Snowflake Summit event Monday. Thanks 🙏

@dpguthrie @boxysean Thanks. I don't see "What You'll Learn" and "What You'll Build" as part of the Overview step and the layout of Conclusion and Resources isn't the same as https://quickstarts.snowflake.com/guide/getting_started_with_dataengineering_ml_using_snowpark_python/index.html#0
Is this too much of a lift for Summit?

Hey @iamontheinet thanks for the feedback. I went back and added the sections in the template. Thanks for the guidance.
Can this now please be merged so we identify any issues today before Monday's session? If you have any other feedback you would like in for longevity on quickstarts.snowflake.com, happy to work on it post-Summit.
Thank you. 🙏


Implement genetic algo for fine tuning strategy parameters

Take a look genetic-js
It is also present on npm.
Let's discuss here

hurr hurr hurr, i'm waaaay ahead of you. Unfortunately it's in python but i'm getting some good results #299

@arpheno How are you using your python strat with a JS based system? Does your python algo run theny ou take the outputted data over to zenbot?

Yes.
On Fri, Jun 16, 2017 at 6:57 PM Jordan<EMAIL_ADDRESS>wrote:

@arpheno https://github.com/arpheno How are you using your python strat
with a JS based system? Does your python algo run theny ou take the
outputted data over to zenbot?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/carlos8f/zenbot/issues/298#issuecomment-309078942,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABBzGh2FMd3a-oRbJMK2MYmQ0I2UUyeaks5sErQHgaJpZM4N8Xyg
.

@arpheno Great work, I'll take a look. Have you seen the algo that I have linked?

I've setting up your work. I'm still waiting for backfilling..
Some suggestion, can you make it agnostic in environment and selector? With a conf file? I've see that there is much stuff hardcoded, like specific commands for docker. And in zen/tests/test_integration.py this hardcoded selector main('gdax.BTC-ETH',120)

Finally i've succeeded in run your genetics.. After a fight with the new f-strings feature in python 3.6 and after.. I'm go to run some simulations and we will see

I will be glad to hear how it went. Currently 4 people are running it
including you and me.
On Sun, Jun 18, 2017 at 11:49 PM gelotus<EMAIL_ADDRESS>wrote:

Finally i've succeeded in run your genetics.. After a fight with the new
f-strings feature in python 3.6 and after.. I'm go to run some simulations
and we will see
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/carlos8f/zenbot/issues/298#issuecomment-309305251,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABBzGhTrr80VtSOnyldDRc19d8gmKWcxks5sFZtxgaJpZM4N8Xyg
.

@arpheno sorry, i'm not expert in python, but seems to be a bug in evaluate_zen() that not pass the right values. If I print fitness var I obtain always [-100, -100]. Please take a look

Ok, with your fix all seems to work right.
Let's show some results:
Generation 181 {'avg': 1.1251920040357279, 'std': 6.1372615421965255, 'min': -15.320000000000269, 'max': 5.287425556773063, 'len': 20}
Current Hall of Fame:
../zenbot.sh sim gdax.BTC-GBP --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Randal Pitts
../zenbot.sh sim gdax.BTC-USD --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Stephanie Houchins
../zenbot.sh sim gdax.BTC-USD --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Patrick Smith
../zenbot.sh sim gdax.BTC-GBP --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Thomas Tomlin
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 James Alvarez
../zenbot.sh sim gdax.BTC-GBP --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Cody Harvey
../zenbot.sh sim gdax.BTC-GBP --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Jason Dixon
../zenbot.sh sim gdax.BTC-GBP --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Lynn Hetrick
../zenbot.sh sim gdax.BTC-GBP --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Tamika Vega
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Jill Ogrady
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Patrica Espinoza
../zenbot.sh sim gdax.BTC-GBP --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Harold Richardson
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Shirley Harten
../zenbot.sh sim gdax.BTC-USD --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.5633 --period 95m  [5.62, 6.55] 5.287425556773063 Dustin Wright
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 James Mckinney

4 Elites will survive, they're currently the strongest:
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Randal Pitts
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Lynn Hetrick
../zenbot.sh sim gdax.BTC-USD --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Tamika Vega
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Stephanie Houchins
8 Specialists will survive, they're the best in their domain:
../zenbot.sh sim gdax.BTC-USD --strategy sar --avg_slippage_pct 0.4 --min_periods 10 --sar_af 0.0064 --sar_max_af 0.0778 --period 131m  [-7.91, 13.4] -7.910000000556141 Celina Belzer
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0072 --sar_max_af 0.2336 --period 94m  [6.5, 0.06] 0.058404866221893226 Scott Stancil
../zenbot.sh sim gdax.BTC-USD --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0072 --sar_max_af 0.2336 --period 94m  [6.5, -0.21] -0.21121792215630394 Felix Roy
../zenbot.sh sim gdax.BTC-USD --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0072 --sar_max_af 0.2336 --period 94m  [6.5, 0.06] 0.058404866221893226 John Cook
../zenbot.sh sim gdax.BTC-USD --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0072 --sar_max_af 0.2336 --period 94m  [6.5, -0.56] -0.5608584095551045 Harriet Martin
../zenbot.sh sim gdax.BTC-USD --strategy sar --avg_slippage_pct 0.4 --min_periods 4 --sar_af 0.0078 --sar_max_af 0.2336 --period 78m  [-8.53, 13.19] -8.530000000369082 Anita Farnsworth
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 4 --sar_af 0.0091 --sar_max_af 0.2336 --period 78m  [-15.32, 13.63] -15.320000000000269 Jo Cleveland
../zenbot.sh sim gdax.BTC-GBP --strategy sar --avg_slippage_pct 0.4 --min_periods 4 --sar_af 0.0078 --sar_max_af 0.2336 --period 78m  [-8.53, 13.19] -8.530000000369082 Juanita Hurst
Some other have fought their way through:
../zenbot.sh sim gdax.BTC-GBP --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Lynn Hetrick
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Randal Pitts
../zenbot.sh sim gdax.BTC-USD --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Lynn Hetrick
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Lynn Hetrick
../zenbot.sh sim gdax.BTC-EUR --strategy sar --avg_slippage_pct 0.4 --min_periods 13 --sar_af 0.0091 --sar_max_af 0.2336 --period 95m  [5.62, 6.55] 5.287425556773063 Lynn Hetrick
...

With long simulations the simulations directory grows fast in size, needs a fix. In the hall of fame all the population have the same parameters.

But.. (philosophical thinking) it works on static data. What happens if the strong elites dies after n generation and the datas are refreshed with an auto backfill over the same n generations?
I think that if one keep the sim running in the same time that the trading bot working we will have a fresh and plus or minus the right parameters for actual period. There be also possible to create an automated bot refresh procedure that pass the new parameters automatically.
What do you think?

hey swozny what's happen in your repo?
Anyway I've tried to merge some of my ideas in your work. Environment configuration, selectors configuration and autobackfilling. The draw function is broken but i have create a little 'if' that point in conf for turning on or off. I've also setting minimum limit in parameters.
Now i'm working in linux env, need some good guy that try this in docker, because i'm the king of the ignorants on this env.
Will be a good thing to append a "date-time from to" to the hall of fame for improve analisys of results with autobackfill turned on.
If someone want to try this is my repo:
https://github.com/gelotus/zenbot
Take a look in conf.py,
command syntax varied a little bit:
for manual backfilling <zenbot_dir>$ fab autobackfill_local:"<selector>,<days>"
for starting sim
<zenbot_dir>/zen $ python3.6 -m scoop main.py <selector> <days> [<population>][<strategy>] [backfill]
Selectors are in conf, days are the days of trade, polulation in the number of live individuals after a generation change, min 10 default 10, strategy is the same of zenbot, default trend_ema, if backfill is present, it will turn on autobackfill timer during sim for updating the trade history.
Note that in my repo the docker files are absent. Pick those in carlos repo

I tried to merge your PR and it deleted the dockerfiles. I got a lot of
complaints from other people when I merged it, so I had to revert :(
On Thu, Jun 29, 2017 at 12:11 AM gelotus<EMAIL_ADDRESS>wrote:

hey swozny what's happen in your repo?
Anyway I've tried to merge some of my ideas in your work. Environment
configuration, selectors configuration and autobackfilling. The draw
function is broken but i have create a little 'if' that point in conf for
turning on or off. I've also setting minimum limit in parameters.
Now i'm working in linux env, need some good guy that try this in docker,
because i'm the king of the ignorants on this env.
Will be a good thing to append a "date-time from to" to the hall of fame
for improve analisys of results with autobackfill turned on.
If someone want to try this is my repo:
https://github.com/gelotus/zenbot
Take a look in conf.py,
command syntax varied a little bit:
for manual backfilling <zenbot_dir>$ fab
autobackfill_local:","
for starting sim
<zenbot_dir>/zen $ python3.6 -m scoop main.py
[][] [backfill]
Selectors are in conf, days are the days of trade, polulation in the
number of live individuals after a generation change, min 10 default 10,
strategy is the same of zenbot, default trend_ema, if backfill is present,
it will turn on autobackfill timer during sim for updating the trade
history.
Note that in my repo the docker files are absent. Pick those in carlos repo
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/carlos8f/zenbot/issues/298#issuecomment-311806903,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABBzGv84k3scHIEKmRZwiSKM6N-bHdPyks5sIs-JgaJpZM4N8Xyg
.

ok, let's try to implement a windows+docker env and test with it..

I think I'll get to work on this on the weekend :)
I'm absolutely amazed that someone is building on my work, not gonna let it
go to waste.
On Thu, Jun 29, 2017 at 9:13 PM gelotus<EMAIL_ADDRESS>wrote:

ok, let's try to implement a windows+docker env and test with it..
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/carlos8f/zenbot/issues/298#issuecomment-312065191,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABBzGo4P6BMB1gmgH-DupmY24cGNmCorks5sI_GegaJpZM4N8Xyg
.

Just droppin my 2 cents: would be cool if trading would be performed by agent with best score, while in the background there is still evolution going on data of past day or so (quickly iterate) and if found better agent, use that. (While keeping backups in case of unpredicted disasters)

I think you're very courageous if you're not double checking the strategy
the optimizer comes up with before throwing it at real money.
On Thu, Jun 29, 2017 at 11:02 PM Maximus-CZ<EMAIL_ADDRESS>wrote:

Just droppin my 2 cents: would be cool if trading would be performed by
agent with best score, while in the background there is still evolution
going on data of past day or so (quickly iterate) and if found better
agent, use that. (While keeping backups in case of unpredicted disasters)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/carlos8f/zenbot/issues/298#issuecomment-312103202,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABBzGqFtuWJh5i04MVIt0Wl29exXNHcUks5sJA2PgaJpZM4N8Xyg
.

Well before agent would trade with real money, it would first need to run trough simulations to verify its profitability, If I had 2 agents with simulated +10% and +30% ROI respectively, I wouldnt really see a problem switching to better agent at runtime.
On top of this, having Zenbot run several parallel agents in papertrading mode can also provide indication of which agent would be best choice for given market/day.

A little question.. Who uses docker? And on windows or on linux? I've see that the last docker for windows works only on windows 10, both stable and edge versions. For windows 7-8-8.1 there is docker-toolbox. That's very fragmented, on linux i don't see the convenience of using docker.. But i don't know, need an advice

@arpheno have you vanished? :) Take a look in my repo, docker was reinserted.

I get a weird container error:
sudo docker-compose exec server bash
ERROR: Container 4b0e2a54d74d7757ef3886f6128dfc1070521c45928d8f780c6f7c5df4339c6b is restarting, wait until the container is running

Hey Guy, Just want to say thanks for all your efforts to implement genetic algorithms into zenbot. It has been the first introduction to the topic for me and i think im hooked :+1:
For the last few weeks I have been trying to add covariance matrix adaptation evolution strategy, which i believe will automatically adjust cx and mut probabilities to help with convergence and to prevent getting stuck on local maximums for more complex strategies.
I think this should be possible and relatively easy using the tools from the python deap package a la:
http://deap.readthedocs.io/en/master/examples/cmaes.html
https://github.com/DEAP/deap/blob/a5d77d1e30f7674948bb6a2c4381fa4e247ac89d/examples/es/fctmin.py
Unfortunately this is my first introduction to both python and evolution strategies, and to be honest, i'm a bit out of my depth.  @gelotus, @arpheo I was wondering if you guys, (or anyone else) would be able to have a look (im sad to see development seems to have stopped on this fork). I think this would be very beneficial to zenbot development and for me personally very rewarding =)
import numpy

from functools import partial
from timer import ThreadTimer
from deap import algorithms, base, benchmarks, cma, creator, tools
from deap.tools import cxTwoPoint, mutGaussian
from scoop import shared
from blessings import Terminal
from subprocess import run, DEVNULL
from conf import indpb, sigma, partitions, selectors, path, bkfint
from evaluation import evaluate_zen, Andividual
from evolution import evolve

term = Terminal()

creator.create("FitnessMax", base.Fitness, weights=(1.0,))
creator.create("Individual", list, fitness=creator.FitnessMax)

toolbox = base.Toolbox()

N=30

def main(instrument, days, popsize, zstrategy='trend_ema', nobf=None):
    print(term.blue("Starting evolution...."))
    evaluate = partial(evaluate_zen, days=days)
    toolbox.register("evaluate", evaluate_zen, days=days)
    print(term.blue("Evaluating ") + term.green(str(popsize)) + term.blue(" individuals over ")
    + term.green(str(days)) + term.blue(" days in ") + term.green(str(partitions)) + term.blue(" partitions."))
    Andividual.path = path
    Andividual.instruments = selectors[instrument]
    Andividual.mate = cxTwoPoint
    Andividual.mutate = partial(mutGaussian, mu=0, sigma=sigma, indpb=indpb)
    Andividual.strategy = zstrategy
    print(term.blue("Mating function is ") + term.green(str(Andividual.mate)))
    print(term.blue("Mutating function is ") + term.green(str(Andividual.mutate)))

    numpy.random.seed(128)

    strategy = cma.Strategy(centroid=[5.0]*N, sigma=5.0, lambda_=20*N)
    toolbox.register("generate", strategy.generate, creator.Individual)
    toolbox.register("update", strategy.update)

    hof = tools.HallOfFame(1)
    stats = tools.Statistics(lambda ind: ind.fitness.values)
    stats.register("avg", numpy.mean)
    stats.register("std", numpy.std)
    stats.register("min", numpy.min)
    stats.register("max", numpy.max)

    algorithms.eaGenerateUpdate(toolbox, ngen=250, stats=stats, halloffame=hof)

    res = evolve(evaluate, Andividual, popsize)
    return res, hof[0].fitness.values[0]```

I too am interested in learning more about live ML/genetic algo integration for realtime trade-bot strategy optimization, but my knowledge here is limited at the moment.  I'm more than happy to start digging in myself, and probably will, but am also curious if @gelotus and/or @arpheno have any news regarding their efforts 🔥

Anybody else get stuck at
Currently 1 valid individuals          | xxxxxxxxx Currently 1 valid individuals          | xxxxxxxxx Currently 1 valid individuals          | xxxxxxxxx Currently 1 valid individuals
using RSI or other strategies?

Has anybody encountered this problem? Im using a custom strategy and the only option it has is a --period option. It works fine for regular simulations and trading. I'm getting this error:
Generation 0 {'avg': -21.975665246121547, 'std': 4.6351407503136413, 'min': -27.740000000369083, 'max': -11.259453225628276, 'len': 10} It's breeding season, we're expecting new members of the tribe... Traceback (most recent call last):   File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main     "__main__", mod_spec)   File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code     exec(code, run_globals)   File "/usr/local/lib/python3.6/site-packages/scoop/bootstrap/__main__.py", line 302, in <module>     b.main()   File "/usr/local/lib/python3.6/site-packages/scoop/bootstrap/__main__.py", line 92, in main     self.run()   File "/usr/local/lib/python3.6/site-packages/scoop/bootstrap/__main__.py", line 290, in run     futures_startup()   File "/usr/local/lib/python3.6/site-packages/scoop/bootstrap/__main__.py", line 271, in futures_startup     run_name="__main__"   File "/usr/local/lib/python3.6/site-packages/scoop/futures.py", line 64, in _startup     result = _controller.switch(rootFuture, *args, **kargs)   File "/usr/local/lib/python3.6/site-packages/scoop/_control.py", line 253, in runController     raise future.exceptionValue   File "/usr/local/lib/python3.6/site-packages/scoop/_control.py", line 127, in runFuture     future.resultValue = future.callable(*future.args, **future.kargs)   File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path     pkg_name=pkg_name, script_name=fname)   File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code     mod_name, mod_spec, pkg_name, script_name)   File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code     exec(code, run_globals)   File "main.py", line 68, in <module>     res = main(INSTRUMENT, TOTAL_DAYS, popsize, strategy)   File "main.py", line 26, in main     res = evolve(evaluate, Andividual, popsize)   File "/app/zen/evolution/__init__.py", line 18, in evolve     return algorithm(cls, popsize, futures.map, evaluate, select, breed, mutate, stats, history, hof)   File "/app/zen/evolution/core.py", line 22, in algorithm     offspring = breed(population)   File "/app/zen/evolution/core.py", line 43, in breed     child1, child2 = parent1 + parent2   File "/app/zen/evolution/individual_base.py", line 36, in __add__     child1, child2 = self.__class__.mate(*couple)   File "/usr/local/lib/python3.6/site-packages/deap/tools/crossover.py", line 44, in cxTwoPoint     cxpoint2 = random.randint(1, size - 1)   File "/usr/local/lib/python3.6/random.py", line 220, in randint     return self.randrange(a, b+1)   **File "/usr/local/lib/python3.6/random.py", line 198, in randrange     raise ValueError("empty range for randrange() (%d,%d, %d)" % (istart, istop, width)) ValueError: empty range for randrange() (1,1, 0)**

The randrange() function is being called on a range with no depth (1,1), hence the error.
Looking up the trace, we see the function is calling randrange(a, b+1).  On mobile right now so debugging is limited, but I would check to see what variable "b" is supposed to represent, and figure out why it's being passed through as 0

@jaketblank
I can't find the random.py file in my computer. I'm running on Docker. Where would I be able to read random.py? Also I've been able to use zen on other strategies like trend_ema so I don't think that zen itself is broken.

The problem isn't with the randrange() function - if it was there would be a major problem with all python code that uses it :)
(That file should be located in "/usr/local/lib/python3.6/" on your system, just for reference)
I'd start looking in "individual_base.py" and look at the "mate()" method.  See, then track forwards to see where values are being generated/sent to randrange()

@jaketblank
random.randint(1, size - 1)
self.randrange(a, b+1)
randrange(1,1, 0)

that means b = 0 and if b = size - 1 then size = 1
From crossover.py:
size = min(len(ind1), len(ind2))
It's the minimum length of the two individuals being crossed over
Wouldn't that mean that at least one of the lengths of the individuals being crossed over is 1?
I'm not sure what that means but I think the problem is that the strategy I'm using only has one option: period length. I think this makes the length of the individuals too similar, but I'm not sure. I think a strategy with one option would be better optimized using the backtester script. Thanks though.

I get a strange error:
#python3.6 -m scoop main.py BTC-CUR 10 10 macd
When genetic_algo starts it prints:
Starting evolution....
Evaluating 10 individuals over 10 days in 2 partitions.
Mating function is <function cxTwoPoint at 0x7f5ee6defa60>
Mutating function is functools.partial(<function mutGaussian at 0x7f5ee1ae0c80>, mu=0, sigma=20, indpb=0.3)
Sampling an initial valid population, this may take a while...
 The DB was correctly backfilled.

Should I worry about this thing:
/usr/local/lib/python3.6/dist-packages/deap/tools/_hypervolume/pyhv.py:33: ImportWarning: Falling back to the python version of hypervolume module. Expect this to be very slow.
"module. Expect this to be very slow.", ImportWarning)

@albabosh I'm having the same output of you
xxxxxxxxxxCurrently 1 valid individuals for each cycle.

Can this be closed?

For me it's ok
Il 06 mar 2018 10:13 PM, "LoneWolf345"<EMAIL_ADDRESS>ha
scritto:

Can this be closed?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/DeviaVir/zenbot/issues/298#issuecomment-370930228,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA9DL5I96Va_CsxDMxZyyaYjLvE8UZsrks5tbvwMgaJpZM4N8Xyg
.


Cell unions are empty

This is the behavior right now:
# Alternatively, use an operator to create union:
# TODO: Not working: magic method result is empty!
>>> cells_in_datasets_union = (
...     ex_client.query('dataset', 'cell', [dataset_a])
...     | ex_client.query('dataset', 'cell', [dataset_b])
... )
>>> len(cells_in_datasets_union)
0

Under the hood, this is just doing:
    def __or__(self, other_set):
        new_handle = self.client.set_union(self.handle, other_set.handle, self.output_type)
        ...

and the corresponding set_intersection works, so I think the problem may be on the backend.

Fixed


Add TracerLocalSparsityDetector

Closes #72.

Codecov Report
Attention: Patch coverage is 85.71429% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 66.08%. Comparing base (c49dc32) to head (50139d1).

Files
Patch %
Lines

src/adtypes.jl
85.71%
2 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #81      +/-   ##
==========================================
- Coverage   76.03%   66.08%   -9.96%
==========================================
  Files          14       13       -1
  Lines         676      625      -51
==========================================
- Hits          514      413     -101
- Misses        162      212      +50

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


service pidfile handling is probably broken and insecure

tl;dr
Killing a service via pidfile should use the command_user privileges.
Background
I was looking around in the documentation on writing openrc services due to a discussion about a znc service dying where someone mentioned "i bet its bc openrc wants to run it as root". I figured this is wrong, because surely openrc allows marking a service as needing to be started as non-root.
It turns out that openrc does support this: https://github.com/OpenRC/openrc/blob/master/service-script-guide.md#dont-write-your-own-startstop-functions
You're supposed to set command_user="user:group".
However, it is later on dis-recommended and warned against as being a security vulnerability to start services as non-root: https://github.com/OpenRC/openrc/blob/master/service-script-guide.md#pid-files-should-be-writable-only-by-root

PID files must be writable only by root, which means additionally that they must live in a root-owned directory. This directory is normally /run under Linux and /var/run under other operating systems.
Some daemons run as an unprivileged user account, and create their PID files (as the unprivileged user) in a path like /var/run/foo/foo.pid. That can usually be exploited by the unprivileged user to kill root processes, since when a service is stopped, root usually sends a SIGTERM to the contents of the PID file (which are controlled by the unprivileged user). The main warning sign for that problem is using checkpath to set ownership on the directory containing the PID file. For example,

Why is this a problem?
What does this even mean? It's bad to start services as an unprivileged user that cannot do arbitrary things to root, because the service manager grants absolute trust to the pidfile and allow it to DoS any process it likes, so instead we grant absolute trust to the daemon itself and believe in its pinky promise to drop privileges later?
This documentation appears to have been written by the author of http://michael.orlitzky.com/cves/cve-2017-18284.xhtml which I was linked to as supporting rationale for this documentation warning.
IMO this is totally wrong and indicative of an underlying flaw in openrc itself. If daemons are insecure because they write a pidfile as non-root, owned by non-root, that root then kills...
... the solution is not to start all daemons as root, the solution is to perform killing of that pid after dropping privileges to the command_user.

/cc @thesamesam @orlitzky

... the solution is not to start all daemons as root, the solution is to perform killing of that pid after dropping privileges to the command_user.

I don't think anyone ever said that the solution is to start all daemons as root. There are essentially two types of daemons:

Traditional UNIX daemons that are supposed to be started as root, that write their own PID files, and then fork/background themselves.
"Simple" daemons that are basically just regular programs that run in the foreground. The service manager (OpenRC, systemd) is supposed to push them into the background and handle the PID file.

The mistake is launching the first type as a non-root user and then letting it create a PID file in an unprivileged location. OpenRC doesn't necessarily even know what user such a daemon will run as. For example, the user account is often specified in the daemon's configuration file, e.g.
# grep User /etc/clamd.conf
User clamav

and arbitrary configuration formats aren't parseable in POSIX shell. That design isn't spectacular but it makes sense when you consider the context -- it had to work with a very dumb service manager like sysvinit that doesn't maintain any state between start/stop. And it still works pretty well so long as you don't mistake it for the second type.
There's no problem with the second type because OpenRC handles the PID file (securely) for you. And almost all daemons are of the second type (or provide a compatible option) today because it's what works best with systemd.
To summarize, the PID handling is secure so long as the service script writer chooses the best way to launch the daemon. There are insane corner cases with goofball software obviously but in general you probably just want to launch everything in the foreground and let OpenRC background it these days.
As a side note, it sounds unbelievable, but AFAIK there is no POSIX way to drop privileges. Unrelated to PID files, there's a race condition in checkpath that would benefit greatly from one.

there is no POSIX way to drop privileges

in the shell
of course the C daemons know how to do it

services started by start-stop-daemon get stopped by it, in c code, so we can (and very likely should) drop privileges before sending stop signals
even if "properly" set up services won't have a problem, there's no real reason to leave the door open to mistakes. services that start as root and drop privs themselves will still have root kill them, but now services that don't will have a sane safety guard, makes the process less foot-gunny


swal is not defined

The Menu and Location links are supposed to be Sweet Alerts that aren't working. I keep getting this error: swal is not defined.
Web Page
https://masterminding.github.io/smoothie-queen/
Github
https://github.com/masterminding/smoothie-queen/tree/gh-pages

"Failed to load resource: the server responded with a status of 404 ()"
fix your code first to point to the correct location of the script...


Not Updated Since VNet Peering

In a situation where the VNet that managed instance is in is peered with a VNet that has a GW subnet the creation and of the primary, secondary , and connection GW  needs to be omitted. This article should be updated to reflect. VNet peering

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 03a0dc36-e3d5-3a28-c9f9-9e514150d7a4
Version Independent ID: 5b19f443-599f-e367-3566-f43be87dcf78
Content: Tutorial: Add SQL Managed Instance to a failover group - Azure SQL Managed Instance
Content Source: articles/azure-sql/managed-instance/failover-group-add-instance-tutorial.md
Service: sql-managed-instance
Sub-service: high-availability
GitHub Login: @BustosMSFT
Microsoft Alias: robustos

Thanks for the feedback! We are currently investigating and will update you shortly.

@hhitchler Thank you for the feedback.
This is being assigned to the content author to evaluate and update the documentation as appropriate.

Hi @hhitchler , thanks for contacting Microsoft! We've updated the content to bring attention to VNet peering, and may revisit rewriting the steps to include VNet peering sometime in the future.  I'll go ahead and close out the Git Issue for the time being, but please feel free to comment should you need anything further.
Thanks again for helping improve our documentation, and I hope you have a wonderful rest of your day!
Masha from the SQL Docs team
#please-close
#reassign:zoran-rilak-msft


Align developer doc with feature from pull request #1295

Align developer doc with feature from pull request #1295 "Volumio pull feature for branch and forked repository"

Thanks!


regression Make checkpoint/clean thread trigger more frequently

When run p0, we want some checkpoint/clean thread in FE work more frequently

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run p0 10

run p0 10

run buildall


Declare exported variable types

I want to declare the types of each property in the exports object of a module.
This is the only way I've found works:
// $FlowFixMe
declare var exports: {
  foo: () => Promise<void>,
}

exports.foo = () => Promise.resolve()

Is there a recommended approach for doing this?

You may find some insight in the example we have here: https://github.com/Dash-OS/flow-by-example/tree/master/examples/typing-imports

@bradennapier Are you talking about this file? I'm looking to declare the property types of the exports object, where the function implementation exists in the same file and may have multiple signatures.
This works:
module.exports = {
  foo(): Promise<void> {
    // ...
  }
}

But I would prefer to use the provided exports object.
It seems I can't override its default type of Object though.

You specifically want to do this within a flowlib and not with exported .flow.js files?  It works perfectly for me if declared in the files themselves but perhaps I am still misunderstanding what you want to do.

What you want is to just use declare function multiple times.  Those are provided as exports.
/* @flow */
declare module 'import-test' {
  declare function foo(): Promise<void>;
  declare function foo(a: number): number;
}

/* @flow */

const obj = require('import-test');

const { foo } = obj;

const result = foo(1);

(result: number);

const result2 = foo();

(result2: Promise<void>);

I should clarify I'm working with an internal module, but your advice works well for others that come across this issue. Thanks for investigating! 👍

If outside of a flowlib then you dont want to use declare like you are as that is globally declaring the type which is obviously going to be a problem.

Not always true. The following declaration is what I'm looking for:
declare module.exports: {
  foo: () => Promise<void>,
}

But it doesn't seem to support function overloading:
// The second `foo` takes precedence.
declare module.exports: {
  foo: () => Promise<void>,
  foo: (a: number) => number,
}

// Same here.
declare module.exports: {
  foo(): Promise<void>;
  foo(a: number): number;
}

You will still get an error, but something like this does work... the error is probably similar to the one you ran into.
However it does type things properly in the other files.

const foo = (a?: number): Promise<number> | number => {
  if (typeof a === 'number') {
    return Promise.resolve(2);
  }
  return 2;
};

// $FlowIgnore
module.exports = ({
  foo,
}: {
  foo: ((a: number) => Promise<number>) & (() => number),
  // foo(a: string): Promise<number>,
});

const obj = require('./mod');

const { foo } = obj;

const result = foo(1);

// $Works
(result: Promise<number>);

const result2 = foo();

// $Works
(result2: number);

@bradennapier Awesome, thank you for the help!


[Queues] Using TextXMLEncodePolicy/TextXMLDecodePolicy by default is confusing

The default policy for QueueClient is to use TextXMLEncodePolicy and TextXMLDecodePolicy when encoding and decoding text from the wire.  The rationale here is that the wire format for the storage APIs is XML, and the documentation mentions that the message content needs to be XML Encoded.
However, it appears that the underlying XML library that we use to construct the actual request is already encoding the text, which means we end up double XML encoding data.  This means if you send a message to a queue and that contains data that has to be XML escaped, the message will be "corrupted" as it flows across the wire.  If you look at the message in the portal, you will note that characters like <, >, and & have been escaped.  In addition, if you receive the message in Java, .NET or JavaScript, you'll get back expected content.  Since Python also double decodes by default, if you receive back with Python, you'll get the expected behavior.
Consider the following program:
from azure.identity import SharedTokenCacheCredential
from azure.storage.queue import QueueClient, NoEncodePolicy, NoDecodePolicy, TextXMLEncodePolicy

cred = SharedTokenCacheCredential("matell@microsoft.com");
defaultClient = QueueClient("https://matellsa.queue.core.windows.net/", "test-queue", cred)
noEncodeClient = QueueClient("https://matellsa.queue.core.windows.net/", "test-queue", cred, message_encode_policy=NoEncodePolicy(), message_decode_policy=NoDecodePolicy())

defaultClient.enqueue_message("<!-- A comment --> Also there is some stuff here & there")
noEncodeClient.enqueue_message("<!-- A comment --> Also there is some stuff here & there")

After running, the Azure Portal shows this double escaping:

In addition, if you look at the raw response from the service, when you do an operation like Peek, you'll see in the XML the double escaping has happened:
<?xml version="1.0" encoding="utf-8"?>
<QueueMessagesList>
  <QueueMessage>
    <MessageId>9fb5debe-5ca7-484e-bd51-8dc1b5e6b987</MessageId>
    <InsertionTime>Fri, 18 Oct 2019 20:39:31 GMT</InsertionTime>
    <ExpirationTime>Fri, 25 Oct 2019 20:39:31 GMT</ExpirationTime>
    <DequeueCount>0</DequeueCount>
    <MessageText>&lt;!-- A comment --&gt; Also there is some stuff here &amp; there</MessageText>
  </QueueMessage>
  <QueueMessage>
    <MessageId>e8e3f235-7f51-49d4-8130-2dc422940a98</MessageId>
    <InsertionTime>Fri, 18 Oct 2019 20:39:32 GMT</InsertionTime>
    <ExpirationTime>Fri, 25 Oct 2019 20:39:32 GMT</ExpirationTime>
    <DequeueCount>0</DequeueCount>
    <MessageText><!-- A comment --> Also there is some stuff here & there</MessageText>
  </QueueMessage>
</QueueMessagesList>

Given that the underlying rest framework also does the escaping, I think we should remove TextXMLEncodePolicy and TextXMLDecodePolicy and make the default policy be NoEncode/NoDecode (the Base64 policy can remain for folks that want to pass binary data)

Interestingly enough, it looks like this bug was there in the previous version of azure.storage.queues as well.


DDAbstractLogger with Swift

I needed to implement my own logger to integrate with Logentries but the only way I was able to do it was by overriding the onInternalLoggerQueue method:
import Foundation
import LogEntries
import CocoaLumberjack

class SCLogEntriesLogger: DDAbstractLogger {

    override func logMessage(logMessage: DDLogMessage!) {
        var message = logMessage.message

        if ((logFormatter) != nil) {
            message = logFormatter.formatLogMessage(logMessage)
        }

        if (message != nil) {
            LELog.sharedInstance().log(message)
        }
    }

    override var onInternalLoggerQueue: Bool {
        get {
            return false
        }
    }
}

On Objective-C we would use the _logFormatter variable to not cause the deadlock on NSAssert(![self isOnInternalLoggerQueue], @"MUST access ivar directly, NOT via self.* syntax."); but on Swift language there is no way to access the internal variable because there is no such thing.
My questions are:

Will this approach cause any side effect?
There is any other way to bypass the NSAssert on Swift?
Should we change the CL library to fix this issue (maybe a specific DDAbstractLoggerfor Swift)?

Probably the Assert is no longer pertinent as more people start using Swift.

After more testing this solution doesn't work because of how the logFormatter  getter method is designed to sync the formatter to be applied on correct serial queue position.
- (id <DDLogFormatter>)logFormatter {
    // [...]
    // IMPORTANT NOTE:
    //
    // Methods within the DDLogger implementation MUST access the formatter ivar directly.
    // [...]

    NSAssert(![self isOnGlobalLoggingQueue], @"Core architecture requirement failure");
    NSAssert(![self isOnInternalLoggerQueue], @"MUST access ivar directly, NOT via self.* syntax.");

    dispatch_queue_t globalLoggingQueue = [DDLog loggingQueue];

    __block id <DDLogFormatter> result;

    dispatch_sync(globalLoggingQueue, ^{
        dispatch_sync(_loggerQueue, ^{
            result = _logFormatter;
        });
    });

    return result;
}

I didn't figure it out why the return result; is never called but it is related with the dispatch_sync calls. Any thoughts?

FYI, I had the exact same problem and worked around it this way ;)
override func logMessage(logMessage: DDLogMessage!) {
    var message = logMessage.message

    let ivar = class_getInstanceVariable(object_getClass(self), "_logFormatter")
    if let formatter = object_getIvar(self, ivar) as? DDLogFormatter {
        message = formatter.formatLogMessage(logMessage)
    }

    logentries.log(message)
}

But I'd also appreciate a better long term solution for this.

I also ran into this issue and it's kind of bothering. @mackuba's solution sounds interesting. I solved it by creating another logFormatter property which I can access whenever I want to:
private var internalLogFormatter: DDLogFormatter?

override internal var logFormatter: DDLogFormatter! {
    set {
        super.logFormatter = newValue
        internalLogFormatter = newValue
    }
    get {
        return super.logFormatter
    }
}

override func logMessage(logMessage: DDLogMessage!) {
    var messageText = logMessage.message
    if let logFormatter = internalLogFormatter {
        messageText = logFormatter.formatLogMessage(logMessage)
    }

    // do something with messageText
}

But those are all hacky solutions and we need to rethink this sync architecture, for all the Swift users.

Both @bpoplauschi and @mackuba are nice workarounds and I agree that there should be a better architecture with Swift in mind but, I think for now, these "solutions" should be made available on documentation.

We are experiencing the same issue when backgrounding, then foregrounding the app. Namely, the app freezes once foregrounded, and looks like a race condition on access to _loggerQueue. I have implemented all the solutions presented above, but none are addressing this issue.

I think we don't need this anymore :)


Support for configuring default minifiers for CSS and Javascript bundles...

....
Use the fluent configuration syntax to configure default minifies when
configuring bundles.
For instance to use YUI as the default minifier for both CSS and JS:
Bundle.ConfigureDefaults()
.UseYuiForCssMinification()
.UseYuiForJsMinification();

This is cool.  I wonder if its worth making the methods generic, like UseMinifierForCss() or something?  This would be a bit more flexible if people wanted to use other minifiers, and a bit less code to maintain
.

Added generic methods but left the existing methods as I think it makes the
available options more discoverable for 1st time users.
Let me know what you think.
On 25 February 2012 22:34, Alex Ullrich <
<EMAIL_ADDRESS>
wrote:

This is cool.  I wonder if its worth making the methods generic, like
UseMinifierForCss() or something?  This would be a bit more
flexible if people wanted to use other minifiers, and a bit less code to
maintain
.

Reply to this email directly or view it on GitHub:
https://github.com/jetheredge/SquishIt/pull/159#issuecomment-4176830

Looks great, thanks.  Could see this being very handy in scenarios where you've got more than a few bundles to create.


Fix iOS build "ImagePickerManager.h:32:43: No type or protocol named 'RNImagePickerSpec'"

Thanks for submitting a PR! Please read these instructions carefully:

[x] Explain the motivation for making this change.
[x] Provide a test plan demonstrating that the code is solid.
[x] Match the code formatting of the rest of the codebase.
[x] Target the main branch, NOT a "stable" branch.

Motivation (required)
I can't build RN app with NEW_ARCH enabled
Test Plan (required)
npx react-native init Rn73
cd Rn73
yarn add react-native-image-picker
NO_FLIPPER=1 RCT_NEW_ARCH_ENABLED=1 npx pod-install
yarn ios
# ^^^ ImagePickerManager.h:32:43: No type or protocol named 'RNImagePickerSpec'

cc @Johan-dutoit

cc @Johan-dutoit

cc @Johan-dutoit

Fixed here https://github.com/react-native-image-picker/react-native-image-picker/pull/2252


Q: val() get/set units

Hi !
Trying to get/set value of a slider,
slider:val() returns value in expected format,
and slider:val( new_val ) requires new_val to be set in incrementation steps ?
Do I miss something there ?
Thanks !

Here is the formula to set from an arbitrary val value:
    slider:val(  (val- slider.min) / slider.inc )

Hi X-Raym, sorry I missed this when you posted it. That's correct, and I regret leaving it that way. I actually thought I had gone through all of them at some point and standardized the :val methods to use the same inputs and outputs. I guess not. :(
Unfortunately, this is probably not fixable since it would break any existing scripts that use Slider:val and Reaper/Reapack don't have any dependency management to support multiple versions of Scythe.
If and when I make it back to this project, though, I'll add the correct syntax to the docs.

@jalovatt Thanks for your feedback,
and no problem once we figure out the workaround, but sure having it written on the doc would prevent trials and errors 📦 I
a fix could involve having a new val method, or add some kind of parameter to tell "incrementation" or "value" mode.
(Long time no see, so I take advantage of this post to show you the focus system I made for scythe (tab navigation between elements, offset with arrow key...) Also succeeded flexible slider width :P https://forum.cockos.com/showpost.php?p=2358702&postcount=762... Hope you fine ! Cheers !)


Update release tests

Description

Updates the docs release tests so that they match with the docs site changes.
Related issues/PRs

Todos

[ ] Tests
[ ] Documentation
[ ] Release note

Release Note

None required

Yup, tested it on 3.16 and it passed.


fix: Avoid casting settings before env var expansion

This fixes a regression in #7232. In that PR logic was added to cast settings within the setting store. This was needed because some settings were not retrieved through a settings service in order to avoid infinite regression (i.e. the get method in a setting service cannot generally call itself).
A consequence of those changes was that now we are casting setting values before env var expansion has occurred. To address the regression, this PR makes the casting logic from #7232 disabled by default, and it is only enabled when accessing settings from within the get methods of a setting service (where we cannot rely on the regular setting casting).
A cleaner approach would be preferable, but this should be sufficient to address the regression while preserving the bug fix that #7232 implemented.
Closes #7293

The new test may be flaky: https://github.com/meltano/meltano/actions/runs/4131188538/jobs/7138602617
I have yet to reproduce a failure like this.


cmd/compile: misleading error for multiple value function call in contexts that do accept multiple values

playground
I can use a function's multiple returns as arguments to another function, but cannot use them as parameters in an initialization.
If it is intended, then the compiling error multiple-value ... in single-value context is misleading, because an array or struct initialization can indeed take multiple values.

It's intended.  Changed this bug to be for a better compiler error message.