OpenEBS storage policy for DevOps

A Brief Introduction
This issue talks about putting policy into the hands of operators & hence offering finest level of granularity. This issue talks about implementation of OpenEBS storage policy via a generic MayaPolicy.
NOTE: This is key to implement #976
High Level Design
It is assumed that DevOps operator will have various Kubernetes configurations, deployments, policies stored in some version control system e.g. Github. These files have the required configurations to deploy applications that suit a particular workload, environment, team, etc.
Further, lets assume below folder structure inside the github repo of (a particular team/operator)
- k8s/
   |__ k8s/app/percona/
   |__ k8s/storage/openebs/

Low Level Design

A MayaPolicy is defined as a set of Kubernetes ConfigMap yamls.

To make things clear, let us assume below folder structure
- k8s/
   |__ k8s/storage/openebs/
      |__ policy-db.yaml
      |__ controller-policy.yaml
      |__ replica-policy.yaml
      |__ service-policy.yaml
      |__ volume-conf-policy.yaml
      |__ storage-pool-policy.yaml
      |__ README.md

A team is free to choose a folder structure & naming that suits their requirement
NOTE: Above files are just for reference purposes. There will be more files to suit a team's requirements.
In above folder structure, policy-db.yaml is the MayaPolicy

It refers to various MayaPolicySpecs

Each MayaPolicySpec is composed of properties like:

deployRef, hostPathRef & configMapRef

NOTE: More details can be understood as you read through the example of how OpenEBS defines its storage policies via MayaPolicy

Using MayaPolicy

A DevOps persona installs the MayaPolicy & corresponding MayaPolicySpecs using kubectl
An application can make use of a single MayaPolicy only.
However, a MayaPolicy can be referred to by multiple applications.
The relation between application & policy is N:1
Maya operator needs to be triggered to use MayaPolicy
Trigger as a K8s Initializer

openebs.io/initializer: name_of_configmap_that_represents_the_MayaPolicy

Trigger as a K8s Finalizer

openebs.io/finalizer: name_of_configmap_that_represents_the_MayaPolicy

Trigger as a storage policy

openebs.io/storage-policy: name_of_configmap_that_represents_the_MayaPolicy
This has to be one of the parameters in any K8s StorageClass
This StorageClass can use any external provisioners including openebs provisioner
However, the provisioner should set below as an initializer to the PV that it creates

openebs.io/initializer: name_of_configmap_that_represents_the_MayaPolicy

OpenEBS defines its storage policies via MayaPolicy

In the case of OpenEBS:

One or more persistent volumes can make use of a maya policy.
A MayaPolicy can be referred to in StorageClass.

A MayaPolicy will be executed by a logic known as maya operator.
This logic will get executed when OpenEBS provisioner tries to create a PersistentVolume.
A DevOps operator can design & install many variants of OpenEBS storage policies (i.e. MayaPolicies).

MayaPolicy definition that caters to OpenEBS storage policy

The policy is defined as a Kubernetes ConfigMap
NOTE:

This needs to be installed before being used

apiVersion: v1
kind: ConfigMap
metadata:
  name: policy-db
  annotations:
    openebs.io/policy-operator: maya
data:
  apiVersion: apps/v1beta1
  kind: MayaPolicy
  MayaPolicySpecs:
    - deployRef: replica-policy
      hostPathRef: storage-pool-policy
      configMapRef: volume-conf-policy
    - deployRef: controller-policy
    - deployRef: service-policy

MayaPolicySpec definition that caters to OpenEBS volume's host path

The policy is defined as a Kubernetes ConfigMap
NOTE:

This needs to be installed before being used

apiVersion: v1
kind: ConfigMap
metadata:
  name: storage-pool-policy
data:
  namespace: default
  apiVersion: openebs.io/v1
  kind: StoragePool
  action: get
  name: default

MayaPolicySpec definition that caters to OpenEBS volume's config

The policy is defined as a Kubernetes ConfigMap
NOTE:

This needs to be installed before being used

apiVersion: v1
kind: ConfigMap
metadata:
  name: volume-conf-policy
data:
  namespace: default
  apiVersion: v1
  kind: ConfigMap
  action: put
  yaml: |
    apiVersion: {{.APIVersion}}
    kind: {{.Kind}}
    metadata:
      name: {{.Owner}}
    data: |
      pools:
      - type: src
        create: no
        name: spool
        cachefile: /tmp/cstor/spool.cache
        pooltype:
        diskpaths: /tmp/cstor/sdisk.img
      datasets:
      - type: src
        id: 0
        create: no
        name: svol
        parent: spool
        volblocksize: 4096
        size:<PHONE_NUMBER>0
        readonly: off
        sync: always
        copies: 1
        logbias: latency
        compression: on

MayaPolicySpec definition that caters to OpenEBS volume controller

The policy is defined as a Kubernetes ConfigMap
NOTE:

This needs to be installed before being used
The controller referred here has no relation to Kubernetes controller

This controller is the openebs volume's controller

apiVersion: v1
kind: ConfigMap
metadata:
  name: controller-policy
data:
  namespace: default
  apiVersion: apps/v1beta2
  kind: Deployment
  action: put
  yaml: |
    apiVersion: {{.APIVersion}}
    kind: {{.Kind}}
    metadata:
      annotations:
        openebs.io/volume-type: cstor
      labels:
        openebs.io/controller: cstor-controller
        openebs.io/pv: {{.Owner}}
      name: {{.Owner}}-ctrl
      namespace: {{.Namespace}}
    spec:
      replicas: {{.Replicas}}
      selector:
        matchLabels:
          monitoring: volume_exporter_prometheus
          openebs.io/controller: cstor-controller
          openebs.io/pv: {{.Owner}}
      template:
        metadata:
          labels:
            monitoring: volume_exporter_prometheus
            openebs.io/controller: cstor-controller
            openebs.io/pv: {{.Owner}}
        spec:
          containers:
          - args:
            - controller
            - --frontend
            - gotgt
            - --clusterIP
            - {{.ServiceIP}}
            - {{.Owner}}
            command:
            - launch
            image: openebs/cstor:0.0.1
            name: {{.Owner}}-ctrl-con
            ports:
            - containerPort: 3260
              protocol: TCP
            - containerPort: 9501
              protocol: TCP
          - args:
            - -c=http://<IP_ADDRESS>:9501
            command:
            - maya-volume-exporter
            image: openebs/m-exporter:0.0.2
            name: maya-volume-exporter
            ports:
            - containerPort: 9500
              protocol: TCP
          tolerations:
          - effect: NoExecute
            key: node.alpha.kubernetes.io/notReady
            operator: Exists
            tolerationSeconds: 0
          - effect: NoExecute
            key: node.alpha.kubernetes.io/unreachable
            operator: Exists
            tolerationSeconds: 0

MayaPolicySpec definition that caters to OpenEBS volume replica

The policy is defined as a Kubernetes ConfigMap
NOTE:

This needs to be installed before being used
The controller referred here has no relation to Kubernetes ReplicaSet

This controller is the openebs volume's replica

apiVersion: v1
kind: ConfigMap
metadata:
  name: replica-policy
data:
  namespace: default
  apiVersion: apps/v1beta2
  kind: Deployment
  action: put
  yaml: |
    apiVersion: {{.APIVersion}}
    kind: {{.Kind}}
    metadata:
      annotations:
        openebs.io/storage-pool: {{.StoragePoolName}}
        openebs.io/volume-type: cstor
      labels:
        openebs.io/replica: cstor-replica
        openebs.io/volume-type: cstor
        openebs.io/pv: {{.Owner}}
      name: {{.Owner}}-rep
    spec:
      replicas: {{.Replicas}}
      selector:
        matchLabels:
          openebs.io/replica: cstor-replica
          openebs.io/pv: {{.Owner}}
      template:
        metadata:
          creationTimestamp: null
          labels:
            openebs.io/replica: cstor-replica
            openebs.io/pv: {{.Owner}}
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchLabels:
                    openebs.io/replica: cstor-replica
                    openebs.io/pv: {{.Owner}}
                topologyKey: kubernetes.io/hostname
          containers:
          - args:
            - replica
            - --frontendIP
            - {{.ServiceIP}}
            - --size
            - {{.Capacity}}
            - /openebs
            command:
            - launch
            image: openebs/cstor:0.0.1
            name: {{.Owner}}-rep-con
            ports:
            - containerPort: 9502
              protocol: TCP
            - containerPort: 9503
              protocol: TCP
            - containerPort: 9504
              protocol: TCP
            volumeMounts:
            - name: openebs
              mountPath: /openebs
            - name: volume-config
              mountPath: /etc/config
          tolerations:
          - effect: NoExecute
            key: node.alpha.kubernetes.io/notReady
            operator: Exists
          - effect: NoExecute
            key: node.alpha.kubernetes.io/unreachable
            operator: Exists
          volumes:
          - name: openebs
            hostPath:
              path: {{.HostPath}}/{{.Owner}}
          - name: volume-config
            configMap:
              name: {{.Owner}}

MayaPolicySpec definition that caters to OpenEBS volume service

The policy is defined as a Kubernetes ConfigMap
NOTE:

This needs to be installed before being used

apiVersion: v1
kind: ConfigMap
metadata:
  name: service-policy
data:
  namespace: default
  apiVersion: v1
  kind: Service
  action: put
  yaml: |
    apiVersion: {{.APIVersion}}
    kind: {{.Kind}}
    metadata:
      labels:
        openebs.io/controller-service: cstor-controller-service
        openebs.io/volume-type: cstor
        openebs.io/pv: {{.Owner}}
      name: {{.Owner}}-ctrl-svc
      namespace: {{.Namespace}}
    spec:
      ports:
      - name: iscsi
        port: 3260
        protocol: TCP
        targetPort: 3260
      - name: api
        port: 9501
        protocol: TCP
        targetPort: 9501
      selector:
        openebs.io/controller: cstor-controller
        openebs.io/pv: {{.Owner}}

Appendix

The fields defined inside the MayaPolicySpec i.e. data property of K8s ConfigMap are understood by operator i.e. maya.

These custom fields are:

namespace (K8s namespace where get/put operation will take place)
apiVersion (the api version of K8s Kind; enables maya operator to invoke necessary K8s http client)
kind (the kind of K8s object; enables maya operator to invoke necessary K8s http client)
action (get/put operation)
name (used for get actions)
yaml (used for put actions)

Miscellaneous

Hand editing these ConfigMaps are time taking & error prone. OpenEBS will provide doc on using various tools that will aid the DevOps in these efforts.
Keep the number of ConfigMaps minimal & hence avoid burdening etcd

YAML Specifications
StorageClass specification

Watch out for openebs.io/storage-policy. This is understood by maya operator.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-percona
  annotations:
    openebs.io/provisioner: provisioner-iscsi
    provisioner.openebs.io/version: 0.6.0
provisioner: openebs.io/provisioner-iscsi
parameters:
  openebs.io/storage-policy: "my-stor-policy"

StoragePolicy specification

Maya exposes StoragePolicy as a Kubernetes CustomResource

apiVersion: openebs.io/0.6.0
kind: MayaStoragePolicy
metadata:
  name: my-stor-policy
  annotations:
    # Below is updated by the maya operator once this CR is installed
    openebs.io/referredTemplate:
spec:
  # name of the template
  baseTemplate:
  policies:
    type: ReplicaCount
      value: 2
    type: ReplicaImage
      value: openebs.io/jiva:0.6.0
    type: IsMonitoring
      value: On
    type: StoragePool
      refer: my-host-dir
    type: Quota
      yaml:
    type: QOS
      yaml:
    type: Snapshot
      yaml:
status:
  phase:
  message:
  reason:
  conditions:
    - type:
      status:
      lastProbeTime:
      lastTransitionTime:
      reason:
      message:

YAML Specifications -- Low Level
MayaRun Specification

This specification understood by maya-runner operator
This is expressed as a K8s ConfigMap
MayaRun is the declarative code that maya uses to provision openebs volume in K8s
MayaRun is a specification that executes various MayaTemplate specs
Below is a sample MayaRun template:

Notice how this is referred from the StorageClass as openebs.io/storage-policy

apiVersion: v1
kind: ConfigMap
metadata:
  name: maya-run-tpl-0.6.0
  labels:
    openebs.io/kind: MayaRun
    mayarun.openebs.io/version: 0.6.0
  annotations:
    openebs.io/operator: maya-runner
    operator.openebs.io/version: 0.6.0
data:
  yaml: |
    inputs:
      - namespace: storageguy
      - storage-engine-type: cstor
      - owner: abc-123-123
      - replica-image: openebs/jiva:0.5.0
      - replica-count: 2
      - controller-image: openebs/jiva:0.5.0
      - monitoring-image: openebs/vol-monitor:0.1.0
      - volume-mon-lbl-key: openebs.io/monitoring
      - volume-mon-lbl-val: vol-mon
      - capacity: 20G
      - storage-pool-path: /var/openebs
    imports:
      - template: storage-service-tpl-0.6.0
      - template: storage-config-tpl-0.6.0
      - template: storage-controller-tpl-0.6.0
      - template: storage-replica-tpl-0.6.0

MayaTemplate Specification

MayaTemplate is expressed as a K8s ConfigMap
MayaTemplate can embed any K8s Kind
This template specification is used by MayaRun

In other words, MayaRun composes a set of MayaTemplate(s)

OpenEBS Service As MayaTemplate

Below is a sample MayaTemplate spec that represents an openebs service

apiVersion: v1
kind: ConfigMap
metadata:
  name: storage-service-tpl-0.6.0
  labels:
    openebs.io/kind: MayaTemplate
    mayatemplate.openebs.io/version: 0.6.0
  annotations:
    openebs.io/operator: maya-runner
    operator.openebs.io/version: 0.6.0
    mayatemplate.openebs.io/meta: |
      namespace: {{inputs “namespace”}}
      apiVersion: v1
      kind: Service
      action: put
data:
  yaml: |
    apiVersion: {{.APIVersion}}
    Kind: {{.Kind}}
    metadata:
      labels:
        openebs.io/controller-service: {{inputs “storage-engine-type”}}-controller-svc
        openebs.io/storage-engine-type: {{inputs “storage-engine-type”}}
        openebs.io/pv: {{.Owner}}
      name: {{.Owner}}-ctrl-svc
      namespace: {{.Namespace}}
    spec:
      ports:
      - name: iscsi
        port: 3260
        protocol: TCP
        targetPort: 3260
      - name: api
        port: 9501
        protocol: TCP
        targetPort: 9501
      selector:
        openebs.io/controller: {{inputs “storage-engine-type”}}-controller
        openebs.io/pv: {{.Owner}}

OpenEBS Config As MayaTemplate

Below is a sample MayaTemplate spec that represents an openebs config

apiVersion: v1
kind: ConfigMap
metadata:
  name: storage-config-tpl-0.6.0
  labels:
    openebs.io/kind: MayaTemplate
    mayatemplate.openebs.io/version: 0.6.0
  annotations:
    openebs.io/operator: maya-runner
    operator.openebs.io/version: 0.6.0
    mayatemplate.openebs.io/meta: |
      namespace: {{inputs “namespace”}}
      apiVersion: v1
      kind: ConfigMap
      action: put
data:
  yaml: |
    apiVersion: {{.APIVersion}}
    Kind: {{.Kind}}
    metadata:
      labels:
        openebs.io/storage-config: {{inputs “storage-engine-type”}}-config
        openebs.io/storage-engine-type: {{inputs “storage-engine-type”}}
        openebs.io/pv: {{.Owner}}
      name: {{.Owner}}-config
      namespace: {{.Namespace}}
    data: |
      pools:
      - type: src
        create: no
        name: spool
        cachefile: /tmp/cstor/spool.cache
        pooltype:
        diskpaths: /tmp/cstor/sdisk.img
      datasets:
      - type: src
        id: 0
        create: no
        name: svol
        parent: spool
        volblocksize: 4096
        size:<PHONE_NUMBER>0
        readonly: off
        sync: always
        copies: 1
        logbias: latency
        compression: on

OpenEBS Controller As MayaTemplate

Below is a sample MayaTemplate spec that represents an openebs controller

apiVersion: v1
kind: ConfigMap
metadata:
  name: storage-controller-tpl-0.6.0
  labels:
    openebs.io/kind: MayaTemplate
    mayatemplate.openebs.io/version: 0.6.0
  annotations:
    openebs.io/operator: maya-runner
    operator.openebs.io/version: 0.6.0
    mayatemplate.openebs.io/meta: |
      namespace: {{inputs “namespace”}}
      apiVersion: v1
      kind: Deployment
      action: put
data:
  yaml: |
    apiVersion: {{.APIVersion}}
    Kind: {{.Kind}}
    metadata:
      labels:
        openebs.io/controller: {{inputs “storage-engine-type”}}-controller
        openebs.io/storage-engine-type: {{inputs “storage-engine-type”}}
        openebs.io/pv: {{.Owner}}
      name: {{.Owner}}-ctrl
      namespace: {{.Namespace}}
    spec:
      replicas: 1
      selector:
        matchLabels:
          {{inputs “volume-mon-lbl-key”}}: {{inputs “volume-mon-lbl-val”}}
          openebs.io/controller: {{inputs “storage-engine-type”}}-controller
          openebs.io/pv: {{.Owner}}
      template:
        metadata:
          labels:
            {{inputs “volume-mon-lbl-key”}}: {{inputs “volume-mon-lbl-val”}}
            openebs.io/controller: {{inputs “storage-engine-type”}}-controller
            openebs.io/pv: {{.Owner}}
        spec:
          containers:
          - args:
            - controller
            - --frontend
            - gotgt
            - --clusterIP
            - {{stores “service-ip”}}
            - {{.Owner}}
            command:
            - launch
            image: {{inputs “controller-image”}}
            name: {{.Owner}}-ctrl-con
            ports:
            - containerPort: 3260
              protocol: TCP
            - containerPort: 9501
              protocol: TCP
          - args:
            - -c=http://<IP_ADDRESS>:9501
            command:
            - maya-volume-exporter
            image: {{inputs “monitoring-image”}}
            name: maya-volume-exporter
            ports:
            - containerPort: 9500
              protocol: TCP
          tolerations:
          - effect: NoExecute
            key: node.alpha.kubernetes.io/notReady
            operator: Exists
            tolerationSeconds: 0
          - effect: NoExecute
            key: node.alpha.kubernetes.io/unreachable
            operator: Exists
            tolerationSeconds: 0

OpenEBS Replica As MayaTemplate

Below is a sample MayaTemplate spec that represents an openebs replica

apiVersion: v1
kind: ConfigMap
metadata:
  name: storage-replica-tpl-0.6.0
  labels:
    openebs.io/kind: MayaTemplate
    mayatemplate.openebs.io/version: 0.6.0
  annotations:
    openebs.io/operator: maya-runner
    operator.openebs.io/version: 0.6.0
    mayatemplate.openebs.io/meta: |
      namespace: {{inputs “namespace”}}
      apiVersion: v1
      kind: Deployment
      action: put
data:
  yaml: |
    apiVersion: {{.APIVersion}}
    Kind: {{.Kind}}
    metadata:
      labels:
        openebs.io/replica: {{inputs “storage-engine-type”}}-replica
        openebs.io/storage-engine-type: {{inputs “storage-engine-type”}}
        openebs.io/pv: {{.Owner}}
      name: {{.Owner}}-rep
      namespace: {{.Namespace}}
    spec:
      replicas: {{inputs “replica-count”}}
      selector:
        matchLabels:
          openebs.io/replica: {{inputs “storage-engine-type”}}-replica
          openebs.io/pv: {{.Owner}}
      template:
        metadata:
          labels:
            openebs.io/replica: {{inputs “storage-engine-type”}}-replica
            openebs.io/pv: {{.Owner}}
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchLabels:
                    openebs.io/replica: {{inputs “storage-engine-type”}}-replica
                    openebs.io/pv: {{.Owner}}
                topologyKey: kubernetes.io/hostname
          containers:
          - args:
            - replica
            - --frontendIP
            - {{stores “service-ip”}}
            - --size
            - {{inputs “capacity”}}
            - /openebs
            command:
            - launch
            image: {{inputs “replica-image”}}
            name: {{.Owner}}-rep-con
            ports:
            - containerPort: 9502
              protocol: TCP
            - containerPort: 9503
              protocol: TCP
            - containerPort: 9504
              protocol: TCP
            volumeMounts:
            - name: openebs
              mountPath: /openebs
            - name: volume-config
              mountPath: /etc/config
          tolerations:
          - effect: NoExecute
            key: node.alpha.kubernetes.io/notReady
            operator: Exists
          - effect: NoExecute
            key: node.alpha.kubernetes.io/unreachable
            operator: Exists
          volumes:
          - name: openebs
            hostPath:
              path: {{inputs “storage-pool-path”}}/{{.Owner}}
          - name: volume-config
            configMap:
              name: {{.Owner}}

////////////////////////////////////////
//  pkg/types/
////////////////////////////////////////

// MayaPolicyType is a typed representation of various
// maya policies
type MayaPolicyType string

const (
  ReplicaCountMPT MayaPolicyType = "ReplicaCount"

  ReplicaImageMPT MayaPolicyType = "ReplicaImage"

  IsMonitoringMPT MayaPolicyType = "IsMonitoring"

  StoragePoolMPT MayaPolicyType = "StoragePool"

  QuotaMPT MayaPolicyType = "Quota"

  SnapshotMPT MayaPolicyType = "Snapshot"
)

// Policy represents a generic policy understood by maya
// It can compose of a value or refer to a resource or
// even compose its yaml representation
type Policy struct {

  Type MayaPolicyType

  Value string

  Refer string

  Yaml string
}

// Verification comprises of properties that represents the
// verification results of an entity
type Verification struct {
  Type string

  Status string

  LastProbeTime string

  LastTransitioTime string

  Reason string

  Message string
}

// StoragePolicy represents a StoragePolicy specification
type StoragePolicy struct {

  metav1.TypeMeta   `json:",inline"`
  metav1.ObjectMeta `json:"metadata,omitempty"`

  Spec StoragePolicySpec `json:"spec"`

  Status StoragePolicyStatus `json:"status"`
}

// StoragePolicySpec refers to a RunTemplate and an array
// of policies
type StoragePolicySpec struct {
  RunTemplate string

  Policies []Policy
}

// StoragePolicyStatus holds the overall status of StoragePolicy
// as well verification results of individual policy
type StoragePolicyStatus struct {
  Phase string

  Message string

  Reason string

  Verifications []Verification
}

////////////////////////////////////////
//  pkg/maya/maya.go
////////////////////////////////////////

// TemplateInfo composes various properties that provides
// information about a template
type TemplateInfo struct {
  // Kind of the template contents
  Kind
  // APIVersion of the template contents
  APIVersion
  // Namespace of the template contents
  Namespace
  // Action to be invoked on the template contents
  Action RunAction
}

// TemplateInfoFilter is a typed function that flags
// if the provided template suits the invoking function's
// requirement
func TemplateInfoFilter func(TemplateInfo) bool

// OpenEBSKind represents various kinds of openebs objects
type OpenEBSKind string

const (
  MayaRunOEK OpenEBSKind = "MayaRun"

  MayaTemplateOEK OpenEBSKind = "MayaTemplate"
)

// TemplateSelector helps in selecting or identifying
// required template
type TemplateSelector struct {
  Kind OpenEBSKind

  Select TemplateInfoFilter

  // Path is the nested path found in the selected
  // template
  Path ...string
}

type MayaRunner struct {
}

func (m MayaRunner) Run() {}

func (m MayaRunner) Verify() {}

type RunTemplate struct {
  Inputs      map[string]string
  Stores      map[string]string
  Imports    []MayaTemplate
  Verifications []MayaVerification
}

type MayaTemplate struct {
  TemplateInfo
}

func (m MayaTemplate) GenerateDeployment() MayaDeployment {}

type K8sOperation {
  Verb Verb // add, update, get, delete, watch, inform, list, event

  Namespace string

  IsAllNamespaces bool

  Config  K8sConfig
}

////////////////////////////////////////
//     pkg/maya/policy.go
////////////////////////////////////////

// PolicyOpType represents the kind of policy action
type PolicyOpType string

const (
  // OnPOT indicates activation of the policy
  OnPOT PolicyOpType = "On"
  // OffPOT indicates de-activation of the policy
  OffPOT PolicyOpType = "Off"
  // CustomPOT indicates a custom policy logic
  CustomPOT PolicyOpType = "Custom"
  // InputsPOT indicates the policy has to populate
  // the inputs placeholder
  InputsPOT PolicyOpType = "Inputs"
  // StoresPOT indicates the policy has to populate
  // the stores placeholder
  StoresPOT PolicyOpType = "Stores"
)

// PolicyVerifyFn verifies the enforcement of any policy
type PolicyVerifyFn func() bool

// PolicyValueFn provides the policy value to be enforced
type PolicyValueFn func() interface{}

// PolicyOperation provides details of the
// operation that a specific policy object would
// execute
type PolicyOperation struct {
  OpType PolicyOpType

  Operate PolicyOperateFn

  Verify PolicyVerifyFn
}

// PolicyEnforcer holds the contract methods implemented
// by all maya policies
type PolicyEnforcer interface {
  // Enforce implements the logic to enforce this policy
  Enforce() error
  // Verify implements the logic to verify the enforcement
  // of this policy. This is typically a post enforcement
  // logic.
  Verify() Verification
}

func NewPolicyEnforcer(policy Policy) (PolicyEnforcer, error) {
  if policy.Type == ReplicaCountMPT {
    return NewReplicaCountPolicy(policy), nil
  } else if policy.Type == ReplicaImageMPT {
    return NewReplicaImagePolicy(policy), nil
  }

  return nil, fmt.Errorf("Invalid policy '%v'", policy)
}

////////////////////////////////////////
//    pkg/maya/storage_policy.go
////////////////////////////////////////

type ReplicaCountStoragePolicy struct {
  Policy

  Template TemplateSelector

  Operation PolicyOperation
}

type ReplicaImageStoragePolicy struct {
  Policy
}

type IsMonitoringStoragePolicy struct {
  Policy
}

type StoragePoolStoragePolicy struct {
  Policy
}

type QuotaStoragePolicy struct {
  Policy
}

type SnapshotStoragePolicy struct {
  Policy
}

func (s StoragePolicy) Enforce() {
  // get all the storage policies
  // invoke Enforce() methods of all its policies
  // invoke MayaRun.Run()
}

func (s StoragePolicy) Verify() {
  // make use of Verify() method of all its policies
}


$touched returns true for disabled fields

$touched method of ng-messages for form field returns true even if form field is disabled..

Isn't this desirable behaviour? Many valid forms would not validate if a disabled field is pristine by default and there's no way to touch it, e.g. if you're disabling a 'save' button when the form is pristine.

What if input field is diabled and user clicks on it, validation message will be shown as validation condition sayd this field is required..

Can you please show an example of this? You mean, if a user blurs a disabled field, $touched is set to true?

Actually I am playing with angular material chips and this field is
dependent upon selection of some other field, so I have diabled this chips
field untill the related value is selected. In this case if the chips field
is in diabled state and user clicks on this disabled field, ng-messaged
"required" message is being shown.
I hope you got my point..
Rayon Dabre.
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>Cell: +919823608959
On Mon, Aug 3, 2015 at 10:21 PM, Martin Staffa<EMAIL_ADDRESS>wrote:

Can you please show an example of this? You mean, if a user blurs a
disabled field, $touched is set to true?
—
Reply to this email directly or view it on GitHub
https://github.com/angular/angular.js/issues/12454#issuecomment-127329029
.

@rayon1990 I don't really understand how this can happen. $touched is set on the control after it has been blurred, not when it is in focus. Can you please provide a small demo of this problem?

@rayon1990 I'm gonna close this issue, since we don't have a reproduce scenario. Please reopen when you can provide a demo of the problem.


npm run build error

Boilerplate version: 3.3.3
Webpack version: 2.2.0-rc.7
offline-plugin version: 4.5.5
    ERROR in ./~/offline-plugin/lib/misc/sw-loader.js?json=%7B%22data_var_name%22%3A%22__wpo%22%2C%22loaders%22%3A%5B%5D%2C%22cacheMaps%22%3A%5B%5D%7D!./~/offline-plugin/empty-entry.js
    Module not found: Error: Can't resolve 'exports' in '/Users/xxx/project/node_modules/offline-plugin'
    BREAKING CHANGE: It's no longer allowed to omit the '-loader' suffix when using loaders.
                     You need to specify 'exports-loader' instead of 'exports'.
     @ ./~/offline-plugin/lib/misc/sw-loader.js?json=%7B%22data_var_name%22%3A%22__wpo%22%2C%22loaders%22%3A%5B%5D%2C%22cacheMaps%22%3A%5B%5D%7D!./~/offline-plugin/empty-entry.js 1:0-42

I could not reproduce this on my machine.
npm run setup
npm run build
:white_check_mark: Successful
npm run clean
npm run build
:white_check_mark: Successful
OS: Ubuntu 14.04
RBP: 3.3.3
Can you specify repro steps?

@18601673727 I ran into something similar. My solution was to rm -rf node_modules the 'yarn'.
If you don't have Yarn package manager installed you should, but npm install should work too.

Thanks @gihrig but doesn't work for me, I regenerated the project in 3.4.0 and everything goes well.

@18601673727 What was the change in 3.4.0 that solved your issue? I'm not in a position to replace the entire project.

Add this to your webpack.base.babel to fix on earlier versions:
new webpack.ProvidePlugin({
      fetch: 'exports-loader?self.fetch!whatwg-fetch',
    });

@designbyadrian sorry for the delay but the change list is huge and I've already switched for a long time.


Translation of phrases with links

Currently, all the phrases with links eg "Data Review Tool" are translated as separated strings in transifex. We should consider a way to have them in the same paragraph where they are mentioned, instead of in separated strings. E.g. in https://github.com/open-contracting/spoonbill-web/blob/83ef61cfc33eb36359c393c12b75d3f81db8e589/frontend/src/views/UploadFile/index.vue#L41-L54
we should consider using the same translate tag for the complete paragraph

The issue was fixed. Please check.

Hmm I still see the strings as separate strings in transifex and in the code

https://github.com/open-contracting/spoonbill-web/blob/f0c99526c0608832bbd6e3731d357f52c1f8d9ca/frontend/src/views/UploadFile/index.vue#L50-L64

But the text on UI is translated?

I think we can leave this as a low-priority cleanup task, which OCP can do at some time.


Fix HBase Model for Traffic

Diego's worry is that "in the traffic table, as Hbase partitions the tables based on rowkey and column family ... [since]  there is only one column family, and each rowkey will have associated millions of cells, eventually."
It's true. If you added one piece of "data" every minute, for each unique "data", we would have ~520,000 columns added per year.
Hence, I propose adding at least the year to the row key.
Right now the row key is geohashFROM_geohashTO. I propose geohashFROM_geohashTO_YYYY where YYYY is the year of the data.

Done as of 9316603ecc3b4729be98beb6bd1f71cbc7f49ca1


Feature Requests

Great work is fantastic!
so here a few more feature requests:

API
option to change base path for models

Added API support in https://github.com/rupeshs/fastsdcpu/releases/tag/v1.0.0-beta.32


fn deploy -v is confusing

Verbose is confusing/broken - it seems there are two flags - one on "fn " one on "fn build" and only the former works
Expectation
 fn  deploy  --app test   --verbose
<verbose output>

actual :
fn  deploy  --app test   --verbose
<terse output>

what actually works
fn --verbose   deploy  --app test

I'd rather verbose works in both positions and  in the same way

+1 -- i brought this up somewhere... thought it was an issue already. @rdallman pointed out that it's a 3rd party lib that needs modified/fixed to support this.

our cli package is doing a global flag set for verbose mode and due to the shortcomings of the go flag package which it uses (iirc) forces this to be directly after the binary arg. changing cli packages isn't very fun, we should probably just deal with the hassle of not using a global flag and having to litter the -v flag over commands that want it

Note, if this does change, then the test harness also needs to be modified following https://github.com/fnproject/cli/pull/305 as that is using the global flag.


Add error messages for the spooky insertions

Objective
Sometimes, the unwraps in entity_mut could fail here, if the entity was despawned before this command was applied.
The simplest case involves two command buffers:
use bevy::prelude::*;
fn b(mut commands1: Commands, mut commands2: Commands) {
    let id = commands2.spawn().insert_bundle(()).id();
    commands1.entity(id).despawn();
}
fn main() {
    App::build().add_system(b.system()).run();
}

However, a more complicated version arises in the case of ambiguity:
use std::time::Duration;

use bevy::{app::ScheduleRunnerPlugin, prelude::*};
use rand::Rng;

fn cleanup(mut e: ResMut<Option<Entity>>) {
    *e = None;
}

fn sleep_randomly() {
    let mut rng = rand::thread_rng();
    std::thread::sleep(Duration::from_millis(rng.gen_range(0..50)));
}

fn spawn(mut commands: Commands, mut e: ResMut<Option<Entity>>) {
    *e = Some(commands.spawn().insert_bundle(()).id());
}

fn despawn(mut commands: Commands, e: Res<Option<Entity>>) {
    let mut rng = rand::thread_rng();
    std::thread::sleep(Duration::from_millis(rng.gen_range(0..50)));
    if let Some(e) = *e {
        commands.entity(e).despawn();
    }
}

fn main() {
    App::build()
        .add_system(cleanup.system().label("cleanup"))
        .add_system(sleep_randomly.system().label("before_despawn"))
        .add_system(despawn.system().after("cleanup").after("before_despawn"))
        .add_system(sleep_randomly.system().label("before_spawn"))
        .add_system(spawn.system().after("cleanup").after("before_spawn"))
        .insert_resource(None::<Entity>)
        .add_plugin(ScheduleRunnerPlugin::default())
        .run();
}

In the cases where this example crashes, it's because despawn was ordered before spawn in the topological ordering of systems (which determines when buffers are applied). However, despawn actually ran after spawn, because these systems are ambiguous, so the jiggles in the sleeping time triggered a case where this works.
Solution

Give a better error message

This should be added for despawn as well, as the entity can fail to exist in the same way.


Bump github.com/moby/buildkit from 0.11.6 to 0.12.5

Cherry-picking from https://github.com/buildpacks-community/kpack/pull/1565

@tomkennedy513 should I backport those changes to 0.12.x as well?

sure if you dont mind


Doc: Clean documentation for WAN

Change Summary

Move keys outside of PREVIEW except for HA in schema
Move WAN preview input variables to main input variables
Move WAN preview how-to to
Massage documentation for eos_designs

Component(s) name
arista.avd.eos_designs
Proposed changes
Cf summary
How to test
Nice doc
Checklist
User Checklist

[ ] Discuss Flow tracking wording as it is rendered only on WAN routers today

Reviwer Checklist

[ ] eos_designs Input Variables new inputs

Repository Checklist

[x] My code has been rebased from devel before I start
[x] I have read the CONTRIBUTING document.
[x] My change requires a change to the documentation and documentation have been updated accordingly.
[x] I have updated molecule CI testing accordingly. (check the box if not applicable)

Blocker remove wan_edge and wan_transit

Blocker: ansible different inventories


Wireframe

Untitled presentation.pdf

Title
Hungry But Broke
Frugal Feast
Budget Bites
My $1 Menu
Frugal Foodie
Description
We want to plan our meals for the week and know where to go for the ingredients nearby. We will have a main page to filter and input ingredients, returning meal ideas. Second page will be a local storage saved of the meals we choose from the first page. The user can see the closest markets to buy the ingredients.
User Story
as a health consicous indivudual / broke college student
I want to plan my meals
so that I can be more frugal
Spoonacular
Google Maps
Tasks
Consider user interaction
Familiarize with accessing the API's
Establish branches and Clean repository file structure
Create issues/Kanban board
Consider modals/local storage/readme
HTML Framework of the layout
CSS framework for the design
Writing the functions to return basic objects from the API's


Horizontal Tabs

When i code for horizontal tab with data-expand="fs" the default tab is showing i.e collapsed. not the next one. please give a solution

Check your Metro 4 version

check your Metro 4 version, create pen on codepen.io with your problem code. create a screenshot for your browser with open developer console.

/*

Metro 4 Components Library v4.2.14 build 691 (https://metroui.org.ua)
Copyright 2018 Sergey Pimenov
Licensed under MIT
*/

Can I know when can we get the stable build of Metro 4

Update to latest

Stable always in master branch, currently 4.2.17

Thank you, Its due to version problem. now working!

Can you give some js code examples to trigger tab event for horizontal tabs.


Data attribute module option passed to the error summary overrides the default attribute value

What
The data attributes module attribute passed to the error summary component overrides the default attribute of govuk-error-summary.
The passed option should be appended to the attribute value.
<%= render "govuk_publishing_components/components/error_summary", {
  title: "Message to alert the user to a problem goes here",
  description: "Optional description of the errors and how to correct them",
  data_attributes: {
    module: "auto-track-event"
    tracking: "GTM-123AB"
  },
} %>

<div class="gem-c-error-summary govuk-error-summary" data-module="govuk-error-summary auto-track-event" data-tracking="GTM-123AB" id="error-summary-73a1ff37" data-govuk-error-summary-module-started="true">

Why
The component cannot be initialised and keyboard focus not moved to the error summary on page load (see error summary - design system guidance).
Anything else

I've recently come across a simliar issue when adding the keyboard shim for link buttons
This may affect other components which have a default data attribute module and which are initialised with JavaScript.

Am hoping that this change https://github.com/alphagov/govuk_publishing_components/pull/3171 will help to resolve this problem.


Symbol error

In the DFI Wallet, an icon with an arrow pointing upwards is used to sell the asset. The icon with the arrow pointing down, is used for adding.
According to this logic, the arrows must be exchanged in the Liquidity Wallet. Add liquidity decreases the asset balance in the DFI Wallet and increases the asset balance in the Liquidity Wallet. So, the symbols in the Liquidity Wallet should be rotated.

Hi @RobinTorque , thanks for your feedback. As I understood, for eg:
• User has BTC wallet, a DFI wallet and a DFI-BTC wallet
• When add liquidity, DFI wallet & BTC wallet should display outgoing symbol
• And DFI-BTC wallet should display incoming symbol
Currently, app displays outgoing symbol for for all add liquidity tx.
Feedback well noted, will convey to team.

/triage accepted


threads: add validation for shared calls

The shared-everything-threads proposal prevents shared contexts (e.g., in a shared function) from accessing unshared objects. This change adds validation to prevent shared-to-unshared calls.

cc: @alexcrichton, I have another branch that adds shared to FuncType but that doesn't seem quite right — no other composite type carries that along with it (but it would make things look nicer!).

Hm how about making CompositeType an enum again? Basically move the shared: bool field into each of the enum arms. Then have something fn shared(&self) -> bool on CompositeType which accesses each variant?

Hm how about making CompositeType an enum again? Basically move the shared: bool field into each of the enum arms. Then have something fn shared(&self) -> bool on CompositeType which accesses each variant?

Well, (1) that complicates parsing because we would have to pass down shared-ness into the variant parsing and (2) I think there's some argument to be made to keeping the wasm-tools AST close to the spec AST.

To come back to this Andrew and I talked bit on Zulip about this and concluded the way to go might be to refactor WasmResources to only be able to return &CompositeType for a certain index. Existing func/struct/array accessors would all be removed. Then in the operator validator there would be helpers that would downcast this further to &FuncType and would auto-check for shared-ness which would avoid the need to thread around a boolean and make sure everything is checked everywhere.

In working on this, I was able to reuse the *_type_at helpers to add shareability checks but this only works when we retrieve by sub-type index; for some kinds of calls, we actually return the type by function index. If we want to get rid of the weird signature on WasmModuleResources::type_of_function, we might be able to just remove that entirely, call WasmModuleResources::type_id_of_function and use that ID with func_type_at (does that work?).

Ah yeah I see, could that perhaps return just the u32 type index of where the function's type is located in the module? (or CoreTypeId if that works, either way)


Is there any quick way to save generated Documents and reload it?

Sometimes it is quick expensive to crawl all the URLs, is it possible to save the Documents and reload it later?
For example:
loader = GitbookLoader("https://docs.gitbook.com")
page_data = loader.load()

Then save the page_data to gitbook.json, could be in the format of
[
    {
        "page_content": "...",
        "metadata": {"source": "...", "title": "..."}
    }
]

Next time when I want to resplit the document or rebuild embeddlings, I can do:
documents = JsonLoader("gitbook.json")

It would be great if both the Save function and JsonLoader can be developed.

There's a JSON loader in the JS version of the library. Would also really like one for this one!

Here is a simple solution

from langchain.document_loaders import PyPDFLoader

loader = PyPDFLoader(file_path='file.pdf')

docs = loader.load()

print(len(docs))

########################################

from  langchain.schema import Document
import json
from typing import Iterable

def save_docs_to_jsonl(array:Iterable[Document], file_path:str)->None:
    with open(file_path, 'w') as jsonl_file:
        for doc in array:
            jsonl_file.write(doc.json() + '\n')

def load_docs_from_jsonl(file_path)->Iterable[Document]:
    array = []
    with open(file_path, 'r') as jsonl_file:
        for line in jsonl_file:
            data = json.loads(line)
            obj = Document(**data)
            array.append(obj)
    return array

save_docs_to_jsonl(docs,'data.jsonl')

docs2=load_docs_from_jsonl(,'data.jsonl'))
print(len(docs2))


Update io.tailscale.ipn.macos.json

The option to hide network devices was not working, I opened a ticket with Tailscale and we found the issue to be a syntax error. The policy key was "Hidden network devices" where it should be "HiddenNetworkDevices" as specified here: https://tailscale.com/kb/1315/mdm-keys#hide-network-devices

Please make this update upstream. See the readme for details.

Looks like the changes from the source have not been converted to JSON.
https://github.com/ProfileCreator/ProfileManifests/commits/master/Manifests/ManagedPreferencesApplications/io.tailscale.ipn.macos.plist
On the Mirror we have the version from 6 months ago.

Thanks for reporting. Looks like that's due to #25, which I'll look into soon.


NOAA NCDC global summary of the day GSOD for rainfall in iotools

NOAA NCDC database of GSOD data has daily rainfall that is useful for determining soling rates based on Addie Kimber's dynamic soiling model. There used to be an ftp site, but I didn't look too hard for it.
Is there a chance that some one from SunPower, like @cedricleroy or @anomam , could dig through the pvlife github repo and extract the gsod code and add it to iotools - or post it to a gist so that someone else can work it into shape for pvlib.iotools?
thanks!

Nit - that's  NOAA database, not NREL.  The link is correct.

If it's already useful for PV modelers that's fine with me. I am not picky about what goes into iotools but I don't want it to be a dumping ground for random weather data. Unless I missed something GSOD does not have irradiance or aerosol data, so it might be better placed in a package like Unidata's Siphon

I only used it for daily rainfall in the soiling model. If there's a better source of rainfall data, then this probably isn't needed. If it's not useful for pvlib, I can open an issue for it at Siphon, if it's not already there.

I think the noaa ncdc isd also has global rainfall

@kandersolar With the addition of #1767, is the GSOD dataset mentioned here still desirable? I'm unsure if they retrieve data from the same source.
Link to GSOD: https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.ncdc:C00516

I browsed through the list of available stations for Denmark as an example and saw only partial overlap with what ACIS makes available, so I think #1767 does not make this issue redundant.


feat: remove bloxy building step

Removes the bloxy building step since my fixes are finally merged 👍🏿
Requires https://github.com/Visualizememe/bloxy/pull/188 to be published to npm

Published my own fork to https://www.npmjs.com/package/@guidojw/bloxy and changed package.json to use that.
If bloxy ever gets updates pushed to npm, the version could be changed back.


gnome3.gpaste: Add GPaste service

Motivation for this change
This makes GPaste finally work.
Things done

[x] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] Linux

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Do I need to run Gnome to test it?

@Mic92 I think the module should work even outside of GNOME, setting the sessionPath should only be necessary for the GNOME Shell extension.

I suppose the gnome icon theme or something else is still missing:
systemctl status --user org.gnome.GPaste.Ui
● org.gnome.GPaste.Ui.service - GPaste user interface
   Loaded: loaded (/nix/store/z2xq8jmpjd13h6d8lkzhdrjc2vcrzgcj-gpaste-3.24.2/etc/systemd/user/org.gnome.GPaste.Ui.service; static; vendor preset: enabled)
   Active: active (running) since Sun 2017-09-03 11:06:58 BST; 41s ago
 Main PID: 19451 (.gpaste-ui-wrap)
   CGroup<EMAIL_ADDRESS>           └─19451 /nix/store/z2xq8jmpjd13h6d8lkzhdrjc2vcrzgcj-gpaste-3.24.2/libexec/gpaste/gpaste-ui --gapplication-service
Sep 03 11:06:58 turingmachine systemd[9834]: Starting GPaste user interface...
Sep 03 11:06:58 turingmachine systemd[9834]: Started GPaste user interface.
Sep 03 11:06:58 turingmachine .gpaste-ui-wrap[19451]: Error retrieving accessibility bus address: org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not pr
Sep 03 11:06:58 turingmachine .gpaste-ui-wrap[19451]: Could not find the icon 'edit-paste'. The 'hicolor' theme
                                                      was not found either, perhaps you need to install it.
                                                      You can get a copy from:
                                                              http://icon-theme.freedesktop.org/releases

@Mic92 That would be adwaita-icon-theme.

also in 17.09 now.


Nuke using apple API key

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I read the Contribution Guidelines
[x] I read docs.fastlane.tools
[x] I searched for existing GitHub issues

Issue Description
I'm trying to rune match nuke using the Apple API key as auth method but the command is failing with:
[!] The request could not be completed because:
	Authentication credentials are missing or invalid. - Provide a properly configured and signed bearer token, and make sure that it has not expired. Learn more about Generating Tokens for API Requests https://developer.apple.com/go/?id=api-generating-tokens

here the log:
$ fastlane match nuke distribution
[✔] 🚀
[12:57:23]: Successfully loaded '/Users/[redacted]/Projects/[redacted]/fastlane/Matchfile' 📄

+-----------------------+--------------------------------------------------------------------------+
|                           Detected Values from './fastlane/Matchfile'                            |
+-----------------------+--------------------------------------------------------------------------+
| git_url               |<EMAIL_ADDRESS>      |
| git_branch            | [redacted]                                                                   |
| clone_branch_directly | true                                                                     |
| app_identifier        | ["com.[redacted], "com.[redacted]] |
| username              |<EMAIL_ADDRESS>                                                  |
| force                 | false                                                                    |
| api_key_path          | fastlane/[redacted].json                                                 |
| team_id               | [redacted]                                                               |
+-----------------------+--------------------------------------------------------------------------+

[12:57:24]: Creating authorization token for App Store Connect API
[12:57:24]: Cloning remote git repo...
[12:57:24]: Checking out branch [redacted]...
[12:57:24]: Enter the passphrase that should be used to encrypt/decrypt your certificates
[12:57:24]: This passphrase is specific per repository and will be stored in your local keychain
[12:57:24]: Make sure to remember the password, as you'll need it when you run match on a different machine
[12:57:24]: Passphrase for Match storage: ********************************
[12:57:29]: Type passphrase again: ********************************
security: SecKeychainAddInternetPassword <NULL>: The specified item already exists in the keychain.
[12:57:30]: 🔓  Successfully decrypted certificates repo

+--------------------------------+-------------------------------------------------------------------+
|                                   Summary for match nuke 2.173.0                                   |
+--------------------------------+-------------------------------------------------------------------+
| type                           | development                                                       |
| readonly                       | false                                                             |
| generate_apple_certs           | true                                                              |
| skip_provisioning_profiles     | false                                                             |
| api_key_path                   | fastlane/[redacted].json                                          |
| username                       |<EMAIL_ADDRESS>                                           |
| team_id                        | [redacted]                                                        |
| storage_mode                   | git                                                               |
| git_url                        |[redacted]|
| git_branch                     | [redacted]                                                            |
| shallow_clone                  | false                                                             |
| clone_branch_directly          | true                                                              |
| keychain_name                  | login.keychain                                                    |
| force                          | false                                                             |
| force_for_new_devices          | false                                                             |
| skip_confirmation              | false                                                             |
| skip_docs                      | false                                                             |
| platform                       | ios                                                               |
| derive_catalyst_app_identifier | false                                                             |
| fail_on_name_taken             | false                                                             |
| skip_certificate_matching      | false                                                             |
| skip_set_partition_list        | false                                                             |
| verbose                        | false                                                             |
+--------------------------------+-------------------------------------------------------------------+

[12:57:30]: Fetching certificates and profiles...

Looking for related GitHub issues on fastlane/fastlane...

[!] The request could not be completed because:
	Authentication credentials are missing or invalid. - Provide a properly configured and signed bearer token, and make sure that it has not expired. Learn more about Generating Tokens for API Requests https://developer.apple.com/go/?id=api-generating-tokens

my matchfile
<EMAIL_ADDRESS>git_branch("[redacted]")
clone_branch_directly(true)
app_identifier(["[redacted]"])
<EMAIL_ADDRESS># Your Apple Developer Portal username
force(false)
api_key_path("fastlane/[redacted].json")
team_id("xxxx")

The json file used in api_key_path:
{
  "key_id": "xxx",
  "issuer_id": "6053b7fe-xxxx-165aa6465141",
  "key": "-----BEGIN PRIVATE KEY-----\nMIGTAgEAMBMGBxxxxxQgCNDO6IE7jAkjkp6q\nGT1IMLhyh4c8QW7ZY3Le/auJyD+gxxx8y\n9VcA0u1o\n-----END PRIVATE KEY-----",
  "in_house": false
}

The API key has just been created and has Admin privileges.
NOTE 1: When I created the JSON file I copied the key from the .p8 certificate generated by apple and I replaced every newline with a \n
NOTE 2: I successfully used the same .p8 for authentication in the fastfile:
api_key = app_store_connect_api_key(key_id: "xxx",
                                            issuer_id: "69a6de74-xxxc7c11a4d1",
                                            key_filepath: "./fastlane/AuthKey_xxxx.p8",
                                            in_house: false)

Is there any way to use directly the app_store_connect_api_key in the matchfile? or alternatively to run nuke as command in the Fastfile?

Have your tried setting SPACESHIP_SKIP_2FA_UPGRADE=1?
So you'd call nuke from the command line like:
SPACESHIP_SKIP_2FA_UPGRADE=1 fastlane match nuke distribution

SPACESHIP_SKIP_2FA_UPGRADE  is the first thing I've tried but in the fastfile only, I'll give it a go from command line, thanks!

Strange, it worked for me. Version 2.179.0

Hi @federicocappelli
Added support to run nuke using API key in this PR: https://github.com/fastlane/fastlane/pull/18387 released in v2.179.0
Now, You can run match_nuke inside your Fastfile very easily...
  private_lane :nuke_example_using_api_key do
     api_key = app_store_connect_api_key(
          key_id: "D8xxxxxx23",
          issuer_id: "2xxxx0bbf-xxxx-458c-xxxx-3d80xxxxx07f",
          key_filepath: "D8xxxxx23.p8",
          in_house: true
     )

    match_nuke(
      api_key: api_key,
      type: "distribution",
      readonly: false
   )
end

Please let me know if have any question and problem to use match_nuke also now I think we can close this issue! 😇

A friendly ping to @max-ott to close this issue as we already have this feature live now since v2.179.0 (PR: #18387 )

great news! thank you!


synth_nexus with -abc9: CCU2 cells get the low order CIN connected to constant 1'b0

Version
Yosys 0.36+42 (git sha1 70d35314d)
On which OS did this happen?
Linux
Reproduction Steps
Unzip the archive; run 'make'.   The example adds two 4b values together to get a 5b result.
Expected Behavior
If you have nextpnr-nexus in your path, it will complete successfully.
If you don't, you'll get an error, but you can examine the Yosys output netlist and see a CCU2 cell with its CIN port unconnected.
Actual Behavior
If you have nextpnr-nexus in your path, it will issue an error either:
ERROR: Cell type 'CCU2' instantiated as 'x0_OB_O_I_CCU2_S0' is not supported by this device.

or if you have a very up-to-date version, you'll get the improved error:
ERROR: CCU2 'b0_IB_I_O_CCU2_A0_1_CIN_CCU2_COUT' CIN net 'b0_IB_I_O_CCU2_A0_C0' driven by non-CCU2 cell 'b0_IB_I_O_CCU2_A0_C0_VLO_Z'.

If you don't have nextpnr-nexus, you can examine the output netlist (top.json or top_out.v) and see that the CIN port of one of the CCU2 cells  is driven by a VLO cell.
(background: https://github.com/YosysHQ/nextpnr/issues/1261)
FYI @gatecat

abc9-ccu2.zip

I should have mentioned that the synth_nexus - nextpnr-nexus flow had been working for quite a while, presumably with the desired behavior that the low-order CIN is left undriven after synth_nexus, but then stopped working at some point, and I didn't diagnose it for far too long.
Going back to older versions of Yosys, I found that the behavior changed between versions 0.29 and 0.30, specifically at commit ec8d7b1c6:
commit ec8d7b1c6857474792c9ccaa149cd0318876a237
Date:   Sun May 21 18:20:18 2023 -0700

    abc9_ops -prep_hier to unmap entire module


Is there documentation available for capabilities/objects/dataPoint/properties/defaultColor ?

I am creating a custom visualization, and the only reference in the wiki for color settings I found is the Objects (Formatting Objects) section.
I have copied and pasted the dataPoint definition from the Funnel Chart into my chart, but I do not see the Data Colors UI when importing my visual into the Power BI client.

I am missing something, and I cannot figure it out what it is.
Additionally, I would like to understand what other formatting options are available to integrate into my visualization.
Thank you,
Gilberto.

You also need to implement the enumerateObjectInstances function and return those objects with instance values as shown [here] (https://github.com/Microsoft/PowerBI-visuals/wiki/Hello-IVisual) in step five.
Alternatively you can look at funnel charts implementation of enumerate. The reason for needing both is we use this mechanism to allow you to perform data validation on property values (say a 1000 was the max number you can input) or hide certain properties in some scenarios (by not returning them).

Implementing enumerateObjectInstances worked perfectly.
Thank you!


Missing access to BETA tenant

Describe the request
I would like to request to add the connectivity for the beta tenant, allowing me to test and provide feedback on the new functionalities.
Thanks

@Timberval Can you please be more specific on what you mean by "add the connectivity for the beta tenant"?
Please refer to the module READ me page here: https://github.com/zscaler/terraform-azurerm-zpa-app-connector-modules?tab=readme-ov-file#zscaler-requirements
If you still have questions, please open a case with Zscaler support, for further assistance.
Zscaler DevRel.


Add look-back support for MergeRollupTask

Adding look-back support for MergeRollupTask, which aims to handle:

pushing historical data
late stream arrivals

The look-back mode can be enabled by adding the following config, which will be applied to all merge levels. If it's enabled, MergeRollupTask will ignore the watermark, and schedule tasks for any bucket having unmerged segments. A new metric mergeRollupTaskNumBucketsToProcess.<tableName>.<mergeLevel> will be published .
"task": {
      "taskTypeConfigsMap": {
        "MergeRollupTask": {
          "mode": "lookBack",
          "100days.mergeType": "concat",
          "100days.bucketTimePeriod": "100d",
          "100days.bufferTimePeriod": "1d"
        }
      }
    }

Codecov Report

Merging #10387 (41849b1) into master (5d0089a) will decrease coverage by 54.14%.
The diff coverage is 28.68%.

@@              Coverage Diff              @@
##             master   #10387       +/-   ##
=============================================
- Coverage     67.87%   13.74%   -54.14%
+ Complexity     5742      231     -5511
=============================================
  Files          1521     1982      +461
  Lines         80305   108038    +27733
  Branches      12826    16523     +3697
=============================================
- Hits          54506    14848    -39658
- Misses        21957    91997    +70040
+ Partials       3842     1193     -2649

Flag
Coverage Δ

unittests1
?

unittests2
13.74% <28.68%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../org/apache/pinot/core/common/MinionConstants.java
0.00% <ø> (ø)

...on/tasks/mergerollup/MergeRollupTaskGenerator.java
63.45% <28.68%> (ø)

...src/main/java/org/apache/pinot/sql/FilterKind.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ain/java/org/apache/pinot/core/data/table/Key.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...in/java/org/apache/pinot/spi/utils/BytesUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ain/java/org/apache/pinot/common/CustomObject.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...n/java/org/apache/pinot/core/data/table/Table.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/core/data/table/Record.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/core/util/GroupByUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/BaseRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 1798 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@jtao15
Can we name both the new mode and the old mode? For example StartingFromEarliest and StartingFromBreakingPoint? And we default to StartingFromBreakingPoint.
This way, it's easy for users to understand. The name loopBack seems not easy to follow.

@jtao15
Can we name both the new mode and the old mode? For example StartingFromEarliest and StartingFromBreakingPoint? And we default to StartingFromBreakingPoint.
This way, it's easy for users to understand. The name loopBack seems not easy to follow.

@zhtaoxiang How about processAll and processFromWatermark since we are mentioning the watermark in Apache Pinot docs already?

@jtao15
Can we name both the new mode and the old mode? For example StartingFromEarliest and StartingFromBreakingPoint? And we default to StartingFromBreakingPoint.
This way, it's easy for users to understand. The name loopBack seems not easy to follow.

@zhtaoxiang How about processAll and processFromWatermark since we are mentioning the watermark in Apache Pinot docs already?

sounds good to me!


ideas

wrap bleak in a class like gopro to simplify user api. home assistant did the same. And adafruit which may be the simplest example.

linux: bleak uses dbus, which is painfully slow, maybe build out a minimal python binding to bluez
https://stackoverflow.com/questions/67178374/light-weight-bluetooth-le-library-in-c

linux: bleak uses dbus, which is painfully slow, maybe build out a minimal python binding to bluez
https://stackoverflow.com/questions/67178374/light-weight-bluetooth-le-library-in-c
https://github.com/gbrault/gattclient
./gattclient -i hci0 -d E8:31:CD:BA:3C:76 -m 256

Doesn't bluez use dbus as well? It did the last time I used it a handful of years ago.

@cmorganBE it does- but apparently there is a way to use a c api directly. it's not actually that bad this was just a very early thought I had.


Outcomes of E-RAFT in Table II

Hello!
I have some questions about the outcomes of E-RAFT trained on DSEC in Table II:

Which weight is used? The one published by E-RAFT or the one trained by yourself?
How is the MVSEC data processed? Is it the same as E-RAFT, or did you make some changes?
Did you use the warm start mentioned in ERAFT during testing?

I am looking forward to your reply! And I would greatly appreciate it if you could share your testing code on ERAFT.

We used the pre-trained model on DSEC (dsec.tar) from E-RAFT repository.
To ensure a fair comparison, we split the MVSEC dataset in the same manner as the code of our paper (https://github.com/danqu130/DCEIFlow/blob/master/utils/datasets/MVSEC.py#L55), which is different from E-RAFT.
We do not turn on warm start. Warm start can slightly improve performance, but the comparison conclusion will not change. For a fair comparison (none of other methods use more temporal data), we only put the unturned ones in the table.

Thanks for your help!


Crash on permission denied

🐛 Bug Report

While allowing location permission it's working fine, but while deny the location permission I am getting  following error.
Dart Error: Unhandled exception:
PlatformException(PERMISSION_DENIED, Access to location data denied, null)
#0      Geolocator._handleInvalidPermissions (package:geolocator/geolocator.dart:197:7)
#1      Geolocator.getCurrentPosition (package:geolocator/geolocator.dart:106:7)

#2      SignupScreenPageState._submitAction (package:locum_mate/Screens/signup.dart:126:10)

#3      SignupScreenPageState.buildWidget.. (package:locum_mate/Screens/signup.dart:261:31)
#4      _InkResponseState._handleTap (package:flutter/src/material/ink_well.dart:507:14)
#5      _InkResponseState.build. (package:flutter/src/material/ink_well.dart:562:30)
#6      GestureRecognizer.invokeCallback (package:flutter/src/gestures/recognizer.dart:102:24)
#7      TapGestureRecognizer._checkUp (package:flutter/src/gestures/tap.dart:242:9)
#8      TapGestureRecognizer.handlePrimaryPointer (package:flutter<…>
Expected behavior
Expect behaviour should give me null response, so that we can proceed further.
Reproduction steps
Import pub in pubspec.yaml
Write following code to any button tap event
try {
Position position = await Geolocator()
.getCurrentPosition(desiredAccuracy: LocationAccuracy.high);
} catch (e) {
print(e);
}
you will have an error.
Configuration
Version: 1.0.1
[✓] Flutter (Channel master, v1.0.1-pre.2, on Mac OS X 10.14 18A384a, locale en-GB)
[✓] Android toolchain - develop for Android devices (Android SDK 28.0.3)
[✓] iOS toolchain - develop for iOS devices (Xcode 10.1)
[✓] Android Studio (version 3.2)
[✓] VS Code (version 1.29.1)
[✓] Connected device (1 available)
• No issues found!
Platform:

[x] :iphone: iOS
[ ] :robot: Android

Have you solved this how to handle this

How to ask permission?

@abdullah4one  You can user permission_handler and ask for permission like this:
import 'package:permission_handler/permission_handler.dart';

Map<PermissionGroup, PermissionStatus> permissions = await PermissionHandler().requestPermissions([PermissionGroup.location]);

Can you please tell how to proceed if I am facing same issue in Android

@harshchan maybe you should create a new issue and explain the problem you are experiencing in detail. This issue is almost 3 years old and a lot has changes since then.
When you create a new issue make sure you mention:

Full stack trace of the error you receive;
The contents of your AndroidManifest.xml file (it should contain the permissions);
A code snippet of the code you are using when the error occurs;
Clear description of steps you take when your error occurs;
Output of the flutter doctor -v command.

I had this problem also and the way to solve this by adding this line of code inside the function that tries to get the location:
LocationPermission permission = await Geolocator.requestPermission();
So the final function will be:
void getLocation() async {
LocationPermission permission = await Geolocator.requestPermission();
Position position = await Geolocator.getCurrentPosition(
desiredAccuracy: LocationAccuracy.low);
// print(position);
}


Add Braintree.Subscription module

It looks like we started on the effort of building a client at the same time! Mine is at https://github.com/radar/braintree_elixir. I'm a noob when it comes to Elixir and I like your code better. So I thought I should contribute to yours and probably "retire" mine.

@radar: I'm glad you found the project, and I appreciate the PR. The integration tests rely on private env vars, so I'm not entirely sure how to get them passing on Travis.

I've removed the debugging stuff now.
WRT to the integration tests, you could setup some encrypted variables on Travis: https://docs.travis-ci.com/user/environment-variables/#Encrypted-Variables.
For this test to pass, you'll need a plan with ID of "starter" in your sandbox account too.

AH, makes sense now. I have set up encrypted variables, it would probably be that the sandbox account doesn't have any plans.

I finally got the encrypted variables into .travis.yml and rebased this branch locally. Thanks!


Setup automatic updates

Something like Shepherd, but for K8S should be running, keeping things up to date.
One candidate is Renovate, but something that actually updates rather than just sending a pull request would be nice.

Flux works like this from the git perspective.

Introduced Flux via #28 and image update via 5196790.


adding parallel rebalancing

fixes #1081
@emelsimsek I documented the GUC but the job infrastructure tables were not in the docs. Do you think it's ok if we just document what we have right now?

Just noticed the issue includes a nice code sample.
We could include the example in the new cluster management section.
ALTER SYSTEM SET citus.max_background_task_executors_per_node = 2;
SELECT pg_reload_conf();

SELECT citus_rebalance_start();

That helps tie it together for people.

fixes #1081
@emelsimsek I documented the GUC but the job infrastructure tables were not in the docs. Do you think it's ok if we just document what we have right now?

I agree that we can leave out the implementation details including the pg_dist_task* tables out of the user facing documentation.

Just noticed the issue includes a nice code sample.
We could include the example in the new cluster management section.
ALTER SYSTEM SET citus.max_background_task_executors_per_node = 2;
SELECT pg_reload_conf();

SELECT citus_rebalance_start();

That helps tie it together for people.

Yes, this could be included where we talk about rebalancing in cluster management.


Fix 3D and PNID linkage

Previously, we would unselect the current model when P&ID chooses a new asset, hence the 3D model would fail to update, this would fix that issue.

[pr-server]
View this change on PR Server

[pr-server]
View this change on PR Server

[pr-server]
View this change on PR Server


Broker triggering throttling being restarted.

I know the logic here hasn't changed, but I realised there is possibly a flaw here.

admin.describeCluster().nodes() returns the currently alive nodes in the cluster.
So the logic here can't account for brokers which are not currently alive.
So a broker with a nearly full disk that restarts will be forgotten about:

Maybe broker 0 got to 99% full and the quota kicks in preventing writes to any brokers in the cluster. What happens if broker 0 is now restarted?
It will disappear from the admin.describeCluster() result (for a time at least), which will release the throttling, allowing appends to the other brokers. Broker 0 might be down for a while if it needed to do log recover on restart.
When broker 0 does rejoin, there is a window before the quotas will get applied again. In the mean time there are appends it needs to replicate, so it fetches and appends and fills its disks.

I think this is all pretty unlikely. In practice it requires a number of things to align, but I think this is possible.

Originally posted by @tombentley in https://github.com/strimzi/kafka-quotas-plugin/pull/37#discussion_r1159571816
And @ robobario https://github.com/strimzi/kafka-quotas-plugin/pull/37#discussion_r1166109990

Yes, we also haven't covered a case mentioned in the proposal yet. If a node drops out of the active set between describeCluster and describeLogDirs calls we could consider this an inconsistency and not act on that data. We'll handle this in another PR.
I'll make an Issue for this problem since it shouldn't block this PR

Moving this here:

We could enable the plugin to operate on stale data for some configurable period. So we retain some cached data about the last state for each node and when calculating the next throttle factor we could combine the data scraped from Kafka with the last cached data. So if we scrape data for nodes [0, 2] and have cached data for [0, 1 ,2, 3] from 1 minute ago, we could proceed with the fresh description of [0,2] and the cached description of [1, 3].
After the configurable duration we could stop considering the cached data.
Could operating on stale data also be a problem? Maybe in some odd case like removing a full broker and replacing it with a new broker with more capacity we might continue blocking writes unneccesarily.

To answer my own old question:
Could operating on stale data also be a problem? I think this is a lesser problem. If they are out of disk space and busily spinning up new instances and decommissioning old nodes I think they could accept some delay in production restarting after the brokers are up. Whereas the other case already identified where we keep filling disks when a full node drops out of the active set temporarily could be more disastrous.

Another case:

Broker 0 is a 99% usages of a value.
Brokers 0-2 stop allowing produce
Broker 0 is removed from the active set
Broker 1 is restarted
Broker 1 observes broker 1-2 and starts producing (it has no state, so on restart is unaware of broker 0)
Same problem as in the original, broker 0 comes up and might fill it's disks replicating from broker 1

This one can't be solved without some external storage of the state, which adds more complexity.

First, we are already working to prevent a disaster situation where users run out of disk and hit all the problems that come with that. So preventing some failures is still a gain even if we can't cover all edge cases.
One idea would be to rename the existing configuration client.quota.callback.static.throttle.factor.validity.duration to client.quota.callback.static.observation.validity.duration and also use this duration to control how long we will use an observation for each node.

In discussion with @tombentley and @SamBarker we have agreed on the above:

we'll support using stale measurements for some time after they are observed, using the existing configuration option.
we will not support cases that require external storage to track the usage of the nodes


TSDoc 1.0 Release Planning/Work

This is a placeholder issue for working on a 1.0 release of TSDoc, partially outlined here: https://tsdoc.org/pages/intro/roadmap/
After talking with @octogonz, he summarized:

As far as blockers for finalizing the spec ("1.0" release), how I see it the really important issues are:

Finish integrating the new declaration reference syntax
Improve the parser to support bold and italics
Improve the parser to support matching of opening/closing HTML elements

These are the breaking changes that would be most disruptive for projects that adopted the old syntax.

Pete and I had a video call to summarize some of the history and challenges of TSDoc syntax and reference implementations. The three items above aren't necessarily ordered; all three items can be independently worked on.
Although I am not an expert in parsers, I'm hoping to find time to work on item 2 for bold and italics in order to familiarize myself with the existing parser. If I can accomplish that, I'd like to take a shot at the new reference syntax.
There is also a stream topic in the Rush Stack Zulip chat room.

Full disclosure, a couple months after making this issue I became an engineering manager and did a lot less development. Then I eventually left that company (which used full-stack TypeScript) and went to work full time with Elixir/Phoenix, so I totally dropped the ball on this. 😅

Happens to the best of us! 😊


Request: add 'Open' to Find in files results

It would be great to add 'Open File' to this menu:

Use case: I want to search for some text in my open folder, but rather than just replace it, I'd like to do some specific work around where the file mentions the text.

Don't you want to left-click the match to see that match in the file?


Confidence intervals are screwed up

Instead of root(N), we divide STD by N. Also, need to double-check the formula for the meta-aggregation: can be screwed up as well.

Closed in pserv.


Launch tests using minimal supported Java version

Now we are using  JDK 11 as minimal version to launch maven build. (so we can use recent version of maven-plugin)
But we produce  JDK 7 (for Leshan 1.x) and JDK 8 (for Leshan 2.x) compatible artifacts thanks to  JVM --release option. (https://github.com/eclipse-leshan/leshan/issues/1412)
But our Junit Tests are now executed with JDK used to launch maven. (so JDK 11)
It could be good to launch tests (at least time to time) with minimal supported java version. (maybe during nightly build ?)
This could be done adding this kind of profile :
<profile>
    <!-- This profile launch tests with minimal supported jdk version using toolchain -->
    <id>useToolchain</id>
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <jdkToolchain>
                        <version>1.${minimal.supported.java.version}</version>
                    </jdkToolchain>
                </configuration>
            </plugin>
        </plugins>
    </build>
</profile>

See :

https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#jdkToolchain
https://maven.apache.org/surefire/maven-surefire-plugin/examples/toolchains.html

Some code about that for 1.x is available in branch toolchain-1.x

This is now done in following PR :

for master : https://github.com/eclipse-leshan/leshan/pull/1562
for 1.x : https://github.com/eclipse-leshan/leshan/pull/1563

(PR are integrated in respective branches but I wait for build success before to close this issue)

Nightly builds and manual test are succeed :white_check_mark:
I think we can close this issue.


[suggestion] type functions that can run type code within current scope (sort of like macros)

Search Terms

Suggestion
f.e., suppose I want to augment JSX types.
Examples
We can write
declare global {
  namespace JSX {
    interface IntrinsicElements {
      'my-component': React.DetailedHTMLProps<MyComponentAttributes, MyComponent>
    }
  }
}

but imagine if we could write:
AddJSXType<'my-component', MyComponentAttributes, MyComponent>

which would be a "type function" call that is essentially like a macro.
The implementation of AddJSXType could be something like
typefunc AddJSXType<N, A, C> =>
  declare global {
    namespace JSX {
      interface IntrinsicElements {
        [k: N]: React.DetailedHTMLProps<A, C>
      }
    }
  }

This would prevent every Custom Element component from having to copy/paste all that boilerplate.
Is there currently another way?
Use Cases
opportunity to make some repetitive boilerplate-heavy things simpler/terser.
Checklist
My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
[x] This wouldn't change the runtime behavior of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, etc.)
[x] This feature would agree with the rest of TypeScript's Design Goals.

This would require a ground-up rearchitecting of how binding and name resolution works, along with at least one additional full tree walk of every source file; I can't possibly imagine we would undertake that kind of effort unless this sort of thing was vastly more common and difficult to do.

It'd be neat if it weren't too complicated.
At the moment the boilerplate for each of my custom elements looks like this:
import * as React from 'react'

interface TimelineAttributes extends React.HTMLAttributes<Timeline> {
  time?: number
}

class Timeline extends Component<TimelineAttributes>(HTMLElement) {
  // ... skipped for brevity ...
}

customElements.define('time-line', Timeline)

declare global {
  interface HTMLElementTagNameMap {
    'time-line': Timeline
  }
}

declare global {
  namespace JSX {
    interface IntrinsicElements {
      'time-line': React.DetailedHTMLProps<TimelineAttributes, Timeline>
    }
  }
}

I wonder if there is a better way.

Suppose the previous example can be simplified to
import * as React from 'react'
import {Component} from './Component'

interface TimelineAttributes extends React.HTMLAttributes<Timeline> {
  time?: number
}

class Timeline extends Component<TimelineAttributes>(HTMLElement) {
  // ... skipped for brevity ...
}

customElements.define('time-line', Timeline)
defineIntrinsicElement<'my-component', MyComponentAttributes, MyComponent>

where defineIntrinsicElement is a "type function" whose implementation is omitted.
Or maybe there's some other possibility than "type function". The main idea is to have some mechanism to make it possible to define types in bulk. The above idea is sort of like a "macro" but fully baked into the type system, intellisense, etc.

The preprocessor idea in https://github.com/microsoft/TypeScript/issues/4691 would help, but it adds much too many possibilities.
What if TS allowed a limited type of preprocessor, call "type function" for example, so that something like
typefunction defineIntrinsicElement<A, B, C> {
  declare global {
    namespace JSX {
      interface IntrinsicElements {
        ${A}: React.DetailedHTMLProps<${B}, ${C}>
      }
    }
  }
}

##defineIntrinsicElement<'my-component', MyComponentAttributes, MyComponent>

would be the same as having written
declare global {
  namespace JSX {
    interface IntrinsicElements {
      'my-component': React.DetailedHTMLProps<MyComponentAttributes, MyComponent>
    }
  }
}

It would have limited functionality, so people won't be inventing new language within TypeScript. For example, a requirement could be that "interpolations" would only be allowed in certain places so that arbitrary names maps to places where arbitrary names would be allowed to appear.

Real world use case: there is a framework that a project I'm working on uses, and it allows to define types dynamically in JavaScript (i.e. a runtime type system). It looks like this:
        openmct.types.addType(
            'foo.timeline.type',
            {
                name: 'Foo Timeline',
                cssClass: 'icon-timeline',
                creatable: true,
                initialize: function (domainObject) {
                    domainObject.composition = [];
                    domainObject.configuration = {
                        activities: {}
                    };
                },
                form: [
                    {
                        name: "Start Time",
                        control: "textfield",
                        cssClass: "l-input-sm",
                        key: "startTime",
                        required: true,
                        property: [
                            "configuration",
                            "startTime"
                        ]
                    },
                    {
                        name: "EndTime",
                        control: "textfield",
                        cssClass: "l-input-sm l-numeric",
                        key: "endTime",
                        required: true,
                        property: [
                            "configuration",
                            "endTime"
                        ]
                    }
                ]
            }
        );

It would be nice if it were possible to have any call to an API like openmct.types.addType also define an ambient interface (perhaps in that example based on the return type of initialize?).
OpenMCT was designed before TypeScript was popular, and is dynamic to the point that trying to figure out what type of object something is involved having to trace it back to one of these definitions, either by debugging in console to follow flow of an object, or by searching for the string name of the type of an object which may also require using the console to first output the string value.
There is a total intellisense blackout in editors like VS Code because these indirection of these runtime types are not statically analyzable like how TypeScript currently is. While writing code one cannot follow the flow of objects or hover on identifiers to see their shape, and will need to manually read the code flow which can require prior knowledge in OpenMCT wiring first, or to inspect objects in debugger.

Should this issue,

https://github.com/microsoft/TypeScript/issues/3136

be marked as Too Complex also?
I think a better name for this feature could be "type-only macros". The other macro feature requests, for example these,

https://github.com/microsoft/TypeScript/issues/4892
https://github.com/microsoft/TypeScript/issues/50815

are too complex indeed, because this wanted to have unlimited macros even for outputting JavaScript code, associating types and methods via new decorator-like macros, etc, etc.
But the idea in this thread is limited to types only.
Another syntax idea is:
type macro defineElement<E, C, A> => {
  declare module 'solid-js' {
    namespace JSX {
      interface IntrinsicElements {
        ${E}: Pick<${C}, ${A}>
      }
    }
  }

  declare global {
    interface HTMLElementTagNameMap {
      ${E}: ${C}
    }
  }
}

It can only be declared in any location where a type can be declared, and it can only be expanded in locations where a type can be declared. The generic-like arguments ("type expression arguments"? I'm not sure the terminology) coud follow some rules like being limited to certain exporession types or locations, not arbitrary syntax.
defineElement<"my-element", MyElementAttributes, MyElement>

Additionally there'd be a rule where by it can be merged with a function, and can serve to combine a JavaScript function call with a type definition. The function call has to happen in a location where a statement could be placed, and would not be allowed in any expression-only location.
type macro defineElement<E, C, A> =>
  // ... same as before ...

function defineElement<E, C, A = 'foo' | 'bar'>(name: E, Class: C) {
  customElements.define(name, Class)
}

// Use it:
defineElement('my-element', MyElement)

The call to defineElement('my-element', MyElement) will expand as if the following was written at the call site:
 // The call still needs to happen, for plain JavaScript
defineElement('my-element', MyElement)

// But it would be as if this were written sibling to the function call:
declare module 'solid-js' {
  namespace JSX {
    interface IntrinsicElements {
      'my-element': Pick<MyElement, 'foo' | 'bar'>
    }
  }
}
declare global {
  interface HTMLElementTagNameMap {
    'my-element': MyElement
  }
}

Then we could write 100 elements without repeating all that boilerplate.


Test

Checklist

[ ] App update process has been followed
[ ] Target branch is develop
[ ] Application version has been bumped

Sorry, accidentally push to wrong repo


Two-Step Verification

Hi. Please add support for Two-Step Verification

Hi Kionlife,
I never thought of that, let me look this weekend at what's involved in implementing that.

Thank you very much. I spent half the night setting it up and only then, after turning on debug mode, did I see that it was asking for the password to my account 😅

Hey Kionlife,
I've implemented the 2FA check in the script! After logging in—whether by scanning the QR code or entering your registered phone number and OTP—the script will now check for the presence of a 2FA password box for 5 seconds.
If a password box is detected, it will prompt you to enter your password, which it then securely passes to Telegram. I've tested it in various scenarios, including accounts with and without 2FA enabled, and everything seems to be working smoothly.
Could you please download claim.py from version 1.4.1 and let me know how it works for you? Your feedback would be greatly appreciated!

Thank you very much. I spent half the night setting it up and only then, after turning on debug mode, did I see that it was asking for the password to my account 😅

It's good that there is a debug mode, I also saw that I need to enter a 2FA code.

I'm very pleased to hear you got everything sorted, Sergey. I never thought to add that, as I don't use it myself. Maybe I should start!

@thebrumby Now we need to disable 2FA on telegram account and wait new version script?
screenshots take at debugIsOn = true

Hey Sergey,
It seems things didn't quite work out as I had hoped! I tested the code several times yesterday before rolling it out, and it worked perfectly each time.
I've just pushed today's changes up as version 1.4.2, which I hope will resolve your issues. Again, I tested it in various scenarios (QR code/OTP, with and without 2FA), and all are working as expected for me.
Please let me know how you get on.

Hello! I still see problems with enable 2FA on telegram account. I disable all active python task, cleared pm2 list and clear bakup folder from project. After I updated to 1.4.2 and run to work, but I still see problem...

I tried work without PM2, just create new session and enter Y for continue script. I see that first step 2FA has success result, but in future has probrem with it

Hi Serget,
Is the last image from PM2? If so, it appears you appear to be starting it incorrectly. The session name you used in the python3 claim.py command must also be included in the PM2 command when starting the script.

In this example, our session name/folder name is "2", you would start PM2 like this:
pm2 start claim.py --name myFirstPMWallet -- 2

While the name specified after --name can be any name of your choosing, it's crucial that -- 2 matches the folder name. In PM2, you should then see the following output:

Hi Sergey,
Is the last image from PM2? If so, it appears you may be starting it incorrectly. The session name you used in the python3 claim.py command must also be included in the PM2 command when starting the script.

In this example, our session name/folder name is "2", so you would start PM2 like this:
pm2 start claim.py --name myFirstPMWallet -- 2

While the name specified after --name can be any name of your choosing, it's crucial that -- 2 matches the folder name of the saved session. In PM2, you should then see the following output:

I think I have the right path name

Okay, I will try to use the simple naming of the session. Only digits.

In this screenshot, the errors at the end indicate that you are not successfully logged in. The message at the top reveals that it can't find the chat box to send commands. It attempted to restore from a backup, but the backup is also corrupted. Consequently, at the bottom, it fails to retrieve the wait timer.
I recommend stopping the PM2 process for now. Run python3 claim.py again, update the settings, and make sure to enable the last option to force a new login. Then, go through the setup process again for oSession5. At the end of the setup, choose 'n' to exit and resume later with PM2. Afterward, restart the process in PM2.

In this screenshot, the errors at the end indicate that you are not successfully logged in. The message at the top reveals that it can't find the chat box to send commands. It attempted to restore from a backup, but the backup is also corrupted. Consequently, at the bottom, it fails to retrieve the wait timer.
I recommend stopping the PM2 process for now. Run python3 claim.py again, update the settings, and make sure to enable the last option to force a new login. Then, go through the setup process again for oSession5. At the end of the setup, choose 'n' to exit and resume later with PM2. Afterward, restart the process in PM2.

What do you think. To begin with, maybe I should do it without PM2 and make sure that it works correctly? I have problems working without using PM2.

Absolutely my friend :)
"Before using PM2 to manage your wallet sessions, you should open the script with python3 claim.py and set up each wallet. After following the process to sign into Telegram and enter your seed phrase, you will be prompted if you want to exit before being handed over to the claim function. You can select 'n' to exit the script and resume the session with PM2 as outlined below."
Will we add an option this afternoon so it will offer to configure PM2 for you.

Absolutely my friend :)
"Before using PM2 to manage your wallet sessions, you should open the script with python3 claim.py and set up each wallet. After following the process to sign into Telegram and enter your seed phrase, you will be prompted if you want to exit before being handed over to the claim function. You can select 'n' to exit the script and resume the session with PM2 as outlined below."
Will we add an option this afternoon so it will offer to configure PM2 for you.

I still see problems ;( I see more screenshots of debug mode and I see that problem with 2FA.

I don't use unique naming. Just empty and take next free digits naming (1)
I don't use PM2. Only run command 'python3 claim.py'. Without naming
I have success login by QR code, success 2FA at first step!

Any ideas?

I added the option that it will set PM2 up for you now :) If you stop anything active in PM2 and update to the latest script :)

I added the option that it will set PM2 up for you now :) If you stop anything active in PM2 and update to the latest script :)

Sorry, I don't understand you). Maybe it's worth figuring out why the script doesn't work without using PM2? I've used two options and I always see the same thing. The problem with 2FA. The screenshots show the same window with a monkey. Recent screenshots without PM2. Maybe it's worth dealing with the script first?

@thebrumby I tried update project to lastest commit https://github.com/thebrumby/HotWalletClaimer/commit/c7b611ad4e69ca371f4147df051f49321ad13e75
and kill all project chrome, chromdriver, python, remove severals folders ... bakups, selenium and tried to run python3 claim.py.
I still see old problem ;(

@thebrumby I tried work by new option 'a' for pm2. I still see problem with 2FA on the next steps... attach all logs

Well, to clarify, the login with the seed phrase occurs at https://tgapp.herewallet.app/ (the game provider's site), so it's separate from the Telegram login process. Reviewing your screenshots more closely, it appears you never successfully log into Telegram, and your 2FA doesn't complete. This is puzzling as I've tested this on Ubuntu 22.04 and 20.04, on VPSs in Germany, the UK, and the USA, with accounts both with and without 2FA, and I haven't encountered this issue.
Are you using a VPS or a local machine? The quickest solution might be to temporarily disable 2FA to log into your accounts. Once logged in, you should remain logged in and can then re-enable 2FA. If that doesn't work, another option is to try using a cloud VPS running Ubuntu 22, which might offer a more consistent environment for running the script.

Local computer without proxy, VPN

Well, to clarify, the login with the seed phrase occurs at https://tgapp.herewallet.app/ (the game provider's site), so it's separate from the Telegram login process. Reviewing your screenshots more closely, it appears you never successfully log into Telegram, and your 2FA doesn't complete. This is puzzling as I've tested this on Ubuntu 22.04 and 20.04, on VPSs in Germany, the UK, and the USA, with accounts both with and without 2FA, and I haven't encountered this issue.
Are you using a VPS or a local machine? The quickest solution might be to temporarily disable 2FA to log into your accounts. Once logged in, you should remain logged in and can then re-enable 2FA. If that doesn't work, another option is to try using a cloud VPS running Ubuntu 22, which might offer a more consistent environment for running the script.

....it appears you never successfully log into Telegram, and your 2FA doesn't complete.....
yeah it's true! I saw a telegram chat and read about it.
message from chat
".......The code was entered correctly, but the correct password was not specified...."

Is it this message?

I got that when I forgot my 2FA (I couldn't remember what punctuation I used). I tried to remove it in Telegram, and it wouldn't let me, so I have to click recover by email and created a new password.

Is it this message?

I got that when I forgot my 2FA (I couldn't remember what punctuation I used). I tried to remove it in Telegram, and it wouldn't let me, so I have to click recover by email and created a new password.

Sorry, I remember 2FA code and correct write this. If you open my screenshot, you can see text about this!
...Step 01g - 2FA passwork entered successfully...
After I see the screenshots from debug that script has access to telegram and work with near wallet app.
I selected this text with green line for you!
Why do I have a problem after  Step 12? I see that latest correct step and after display error text
...We backed up the session data....

Thank you very much. I spent half the night setting it up and only then, after turning on debug mode, did I see that it was asking for the password to my account 😅

Hello. Have you tried the new version of the script? it's okay?

Hi,
I have now updated version 1.4.4 to help with debugging:
Version 1.4.4 includes enhanced validation for users with 2FA enabled. The update introduces two crucial checks: the first verifies that Telegram has not reported the password as incorrect, and the second detects any other errors, such as a blank password field, indicating that the password may not have been transmitted properly. If you encounter errors, please enable debugging in the settings. This will capture screenshots that can assist in diagnosing potential issues.

Thank you for your time! I have success result at enabled 2FA. I don't know what happend, but it's work!

Hey Sergey, I didn’t change any logic around 2FA, I only added the error checking, but so pleased it is working for you. Happy mining.