Script and output in different repos

We are looking to run this in Azure DevOps pipelines and need to have the script in one repo (which the build service account has read-only access to), and the output in another repo (which the build service account has modify access to).

(background to this is the "Limit Job authorisation scope" needs to be disabled on this DevOps project due to cross-project calls in other pipelines. This means that anyone who can run code on a build agent can write to the output repo. If the "script" repo and "output" repo are the same, then someone could overwrite the AzADServicePrincipalInsights script with new content which executes under the service connection (elevation of privilege, lateral movement etc).

The script seems to have two incompatible references. As we have 2 x repos referenced in the pipeline .yaml file
resources:
  repositories:
  - repository: scripts
    type: git
    name: ProjectName/Governance-Scripts
    ref: main
  - repository: output
    type: git
    name: ProjectName/Governance-Output
    ref: main

We then need to update the scriptPath variables as below
variables:
  WikiDir: Governance-Output/AzADServicePrincipalInsights # Note: Prefixed with repo reference https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#checkout-path
  ScriptDir: Governance-Scripts/AzADServicePrincipalInsights/pwsh # Note: Prefixed with repo reference https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#checkout-path

We then get this error
Get-Content: /home/vsts/work/1/s/Governance-Scripts/AzADServicePrincipalInsights/pwsh/AzADServicePrincipalInsights.ps1:3465
Line |
3465 |  … fications = Get-Content -raw .\pwsh\permissionClassification.json | C …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Cannot find path
     | '/home/vsts/work/1/s/pwsh/permissionClassification.json'
     | because it does not exist.

If we then set the workingDirectory on the Powershell task as below
  - task: AzurePowerShell@4
    inputs:
      azureSubscription: '$(ServiceConnection)'
      scriptType: filePath
      pwsh: true
      scriptPath: '$(System.DefaultWorkingDirectory)/$(ScriptDir)/$(Script)'
      scriptArguments: '-OutputPath $(WikiDir)'
      azurePowerShellVersion: latestVersion
      workingDirectory: '$(System.DefaultWorkingDirectory)/$(ScriptDir)'
    displayName: 'Run AzADServicePrincipalInsights v1'

We then get another error
path /home/vsts/work/1/s/Governance-Scripts/AzADServicePrincipalInsights/pwsh/Governance-Scripts/AzADServicePrincipalInsights/. does not exist - please create it!
Exception: /home/vsts/work/1/s/Governance-Scripts/AzADServicePrincipalInsights/pwsh/AzADServicePrincipalInsights.ps1:104
Line |
 104 |          Throw 'Error - check the last console output for details'
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Error - check the last console output for details

Any ideas?

can you please provide a working sample yml where you clone from the source repository, create outputs and then push to a second repository?

We use the config with AzGovViz and this is the working YAML pipeline for AzGovViz. We didn't have to make any changes to the AzGovViz powershell itself.
the main customsiations were

Addition of an OutputRepo variable
Addition of 2 x repo resources
Change the  working directory using cd AzGovViz-Scripts
Update $PipelineInfo parameter (Repository: $(OutputRepoName) )

# AzGovViz v6_major_20211018_1
# First things first:
# 1. edit line 59 and line 60
# 2. check line 74 and 85 if branch 'master' is applicable
# Documentation: https://github.com/JulianHayward/Azure-MG-Sub-Governance-Reporting
# Also check https://www.azadvertizer.net - AzAdvertizer helps you to keep up with the pace by providing overview and insights on new releases and changes/updates for Azure Governance capabilities such as Azure Policy's policy definitions, initiatives (set definitions), aliases and Azure RBAC's role definitions and resource provider operations.
#
# Parameters reference (use in line 108)
# LimitCriticalPercentage | default is '80' | example: -LimitCriticalPercentage 90  | WhatDoesItDo? marks capabilities that approch limits e.g. limit 100, usage 80 will mark with warning
# SubscriptionQuotaIdWhitelist | default is 'undefined' | example: -SubscriptionQuotaIdWhitelist MSDN_, EnterpriseAgreement_ | WhatDoesItDo? processes only Subscriptions that startWith the given QuotaIds
# HierarchyMapOnly | switch | example: -HierarchyMapOnly | WhatDoesItDo? only creates the Hierarchy Tree
# DoNotShowRoleAssignmentsUserData | switch | example: -DoNotShowRoleAssignmentsUserData | WhatDoesItDo? scrub user information
# NoASCSecureScore | switch | example: -NoASCSecureScore | WhatDoesItDo? disables ASC Secure Score request for Subscriptions and Management Groups
# CsvDelimiter | example: -CsvDelimiter "," | WhatDoesItDo? the world is split into two kind of delimiters - comma and semicolon - choose yours
# NoPolicyComplianceStates | example: -NoPolicyComplianceStates | WhatDoesItDo? will not query policy compliance states
# NoResourceDiagnosticsPolicyLifecycle | example: -NoResourceDiagnosticsPolicyLifecycle | WhatDoesItDo? will not create Resource Diagnostics Policy Lifecycle recommendations
# NoAADGroupsResolveMembers | example: -NoAADGroupsResolveMembers | WhatDoesItDo? will not resolve Azure Active Directory Group memberships for Role assignments where identity type is 'Group'
# AADServicePrincipalExpiryWarningDays | example: -AADServicePrincipalExpiryWarningDays 21 | WhatDoesItDo? define warning period for Service Principal secret and certificate expiry; default is 14 days
# NoAzureConsumption | example: -NoAzureConsumption | WhatDoesItDo? Azure Consumption data will not be collected/reported
# AzureConsumptionPeriod | example: -AzureConsumptionPeriod 7 | WhatDoesItDo? define for which time period Azure Consumption data should be gathered; default is 1 day
# NoAzureConsumptionReportExportToCSV | example: -NoAzureConsumptionReportExportToCSV | WhatDoesItDo? Azure Consumption data will not be exported (CSV)
# ThrottleLimit | example: -ThrottleLimit 10 | WhatDoesItDo? Define the number of script blocks running in parallel. Leveraging PowerShell Core´s parallel capability you can define the ThrottleLimit (default=5)
# DoTranscript | example: -DoTranscript | WhatDoesItDo? Log the console output
# AzureDevOpsWikiHierarchyDirection | example: -AzureDevOpsWikiHierarchyDirection "LR" | WhatDoesItDo? Define the direction the Hierarchy should be built in Azure DevOps WokiAsCode (Markdown) TD = TopDown (Horizontal), LR = LeftRight (Vertical)
# SubscriptionId4AzContext | example: -SubscriptionId4AzContext "<your-Subscription-Id>" | WhatDoesItDo? Define the Subscription Id to use for AzContext (default is to use a random Subscription Id)
# NoCsvExport | example: -NoCsvExport | WhatDoesItDo? Do not export enriched 'Role assignments' data, enriched 'Policy assignments' data and 'all resources' (subscriptionId, resourceType, id, name, location, tags, createdTime, changedTime)
# DoNotIncludeResourceGroupsOnPolicy | example: -DoNotIncludeResourceGroupsOnPolicy | WhatDoesItDo? Do not include Policy assignments on ResourceGroups
# DoNotIncludeResourceGroupsAndResourcesOnRBAC | example: -DoNotIncludeResourceGroupsAndResourcesOnRBAC | WhatDoesItDo? Do not include Role assignments on ResourceGroups and Resources
# ChangeTrackingDays | example: -ChangeTrackingDays 30 | WhatDoesItDo? Tracks newly created and updated custom Policy, PolicySet and RBAC Role definitions, Policy/RBAC Role assignments and Resources that occured within the last 14 days (default)
# FileTimeStampFormat | example: -FileTimeStampFormat "yyyyMM-dd_HHmm" | WhatDoesItDo? Time format for the output files (default is `yyyyMMdd_HHmmss`)
# NoJsonExport | example: -NoJsonExport | WhatDoesItDo? Do not export of ManagementGroup Hierarchy including all MG/Sub Policy/RBAC definitions, Policy/RBAC assignments and some more relevant information to JSON
# JsonExportExcludeResourceGroups | example: -JsonExportExcludeResourceGroups | WhatDoesItDo?| JSON Export will not include ResourceGroups (Policy & Role assignments)
# JsonExportExcludeResources | example: -JsonExportExcludeResources | WhatDoesItDo?| JSON Export will not include Resources (Role assignments)
# LargeTenant | example: -LargeTenant | WhatDoesItDo? A large tenant is a tenant with more than ~500 Subscriptions - the HTML output for large tenants simply becomes too big. If the parameter switch is true then the following parameters will be set: -PolicyAtScopeOnly $true, -RBACAtScopeOnly $true, -NoResourceProvidersDetailed $true, -NoScopeInsights $true
# PolicyAtScopeOnly | example: -PolicyAtScopeOnly | WhatDoesItDo? Removing 'inherited' lines in the HTML file for 'Policy Assignments'; use this parameter if you run against a larger tenants
# RBACAtScopeOnly | example: -RBACAtScopeOnly | WhatDoesItDo? Removing 'inherited' lines in the HTML file for 'Role Assignments'; use this parameter if you run against a larger tenants
# NoResourceProvidersDetailed | switch | example: -NoResourceProvidersDetailed | WhatDoesItDo? disables 'ResourceProvider Detailed' output for TenantSummary (compute intensive)
# NoScopeInsights | example: -NoScopeInsights | WhatDoesItDo? and why would you want to do this? In larger tenants the ScopeInsights section blows up the html file (up to unusable due to html file size). To further reduce the output use parameter -LargeTenant
# HtmlTableRowsLimit | example: -HtmlTableRowsLimit | WhatDoesItDo? Although the parameter -LargeTenant was introduced recently, still the html output may become too large to be processed properly. The new parameter defines the limit of rows - if for the html processing part the limit is reached then the html table will not be created (csv and json output will still be created). Default rows limit is 40.000
# AADGroupMembersLimit | example: -AADGroupMembersLimit 333 | WhatDoesItDo? Defines the limit (default=500) of AAD Group members; For AAD Groups that have more members than the defined limit Group members will not be resolved
# NoResources | example: -NoResources | WhatDoesItDo? Will speed up the processing time but information like Resource diagnostics capability and resource type stats (featured for large tenants)
# StatsOptOut | example: -StatsOptOut | WhatDoesItDo? Will opt-out sending stats

trigger: none

pr: none

parameters:
- name: exludedResourceTypesDiagnosticsCapableParameters
  type: object
  default:
    - microsoft.web/certificates
- name: subscriptionQuotaIdWhitelistParameters
  type: object
  default:
    - Enterprise,MSDNDevTest,PayAsYouGo

variables:
  ServiceConnection: AzGovViz #permissions set? https://github.com/JulianHayward/Azure-MG-Sub-Governance-Reporting#required-permissions-in-azure
  ManagementGroupId: xxxx-xxxx-xxxx-xxxx-xxxx
  SubscriptionQuotaIdWhitelist: ${{ join(',',parameters.subscriptionQuotaIdWhitelistParameters) }}
  ExludedResourceTypesDiagnosticsCapable: ${{ join(',',parameters.exludedResourceTypesDiagnosticsCapableParameters) }}
  WikiDir: AzGovViz-Output/wiki/tenant-name # Note: Prefixed with repo reference https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#checkout-path
  CsvDelimiter: ;
  ScriptDir: AzGovViz-Scripts/AzGovViz/pwsh # Note: Prefixed with repo reference https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#checkout-path
  Script: AzGovVizParallel.ps1
  OutputRepoName: AzGovViz-Output

schedules:
- cron: "0 0,12 * * *"
  displayName: every 12h
  always: true
  branches:
    include:
    - main #CHECK: branch 'master' is applicable? - delete me :)

resources:
  repositories:
  - repository: scripts
    type: git
    name: Project-Name/AzGovViz-Scripts
    ref: main
  - repository: output
    type: git
    name: Project-Name/AzGovViz-Output
    ref: main

jobs:
- job: AzGovViz
  timeoutInMinutes: 0

  pool:
    vmImage: 'ubuntu-18.04'

  steps:
  - checkout: self
  - checkout: output
  - task: PowerShell@2
    inputs:
      targetType: 'filePath'
      filePath: $(System.DefaultWorkingDirectory)/$(ScriptDir)/prerequisites.ps1
    displayName: 'Prerequisites check'
    env:
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
  - task: AzurePowerShell@4
    inputs:
      azureSubscription: '$(ServiceConnection)'
      scriptType: filePath
      pwsh: true
      scriptPath: '$(System.DefaultWorkingDirectory)/$(ScriptDir)/$(Script)'
      scriptArguments: '-ManagementGroupId $(ManagementGroupId) -OutputPath $(WikiDir) -CsvDelimiter "$(CsvDelimiter)" -AzureDevOpsWikiAsCode -SubscriptionQuotaIdWhitelist $(SubscriptionQuotaIdWhitelist) -ExludedResourceTypesDiagnosticsCapable $(ExludedResourceTypesDiagnosticsCapable) -StatsOptOut'
      azurePowerShellVersion: latestVersion
    displayName: 'Run AzGovViz v6'
  - powershell: |
      write-host "#################################"
      write-host "Push AzGovViz output to repository"
      write-host "#################################"
      pwd
      cd AzGovViz-Scripts
      $executionDateTimeInternationalReadable = get-date -format "dd-MMM-yyyy HH:mm:ss"
      $currentTimeZone = (Get-TimeZone).Id
      git config --global user.email<EMAIL_ADDRESS>      $PipelineInfo = "Pipeline: '$(Build.DefinitionName)' 'rev $(Build.BuildNumber)' (Project: $([uri]::EscapeDataString("$(System.TeamProject)")); Repository: $(OutputRepoName) ; Branch: $(Build.SourceBranchName) Commit: $(Build.SourceVersion))"
      git config --global user.name "$PipelineInfo"
      git config pull.rebase false
      write-host "Running git add"
      git add --all
      write-host "Running git commit"
      git commit -m "wiki $executionDateTimeInternationalReadable ($currentTimeZone)"
      write-host "Running git push"
      git -c http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)" push origin HEAD:main
    displayName: 'Push AzGovViz v6 output to repository'

for the moment please change the path inside the script. There are already some changes (dot sourcing) in the AzGovViz branch=bv which should handle such scenarios - can you please verify if this is working with your repo structure? If that works for you the AzADSPI will be updated accordingly. -thanks!

Thanks - I've updated line 3465 of AzADServicePrincipalInsights.ps1 as follows
    $getClassifications = Get-Content -raw ./<script repo name>/<AzSpInsights folder in repo>/pwsh/permissionClassification.json | ConvertFrom-Json -AsHashtable -ErrorAction Stop

In our environment that is
    $getClassifications = Get-Content -raw ./Governance-Scripts/AzADServicePrincipalInsights/pwsh/permissionClassification.json | ConvertFrom-Json -AsHashtable -ErrorAction Stop

and it got past that error for the file permissionClassification.json.
I'm now getting an error as below (I tried updating Line 185 and Line 202 to replace any references to .\pwsh with .\Governance-Scripts\AzADServicePrincipalInsights\pwsh
Import-Module: /home/vsts/work/1/s/Governance-Scripts/AzADServicePrincipalInsights/pwsh/AzADServicePrincipalInsights.ps1:191
Line |
 191 |  …             Import-Module "./Governance-Scripts/A …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The specified module
     | './Governance-Scripts/AzADServicePrincipalInsights/pwsh/AzAPICallModule/AzAPICall/1.1.1/AzAPICall.psd1' was not loaded because no valid module file was found in any module directory.

however the script output does state
Verify 'AzAPICall' (1.1.1)
  Try importing AzAPICall module (1.1.1)
  Importing AzAPICall module failed
  Saving AzAPICall module (1.1.1)
  Try importing AzAPICall module (1.1.1)
  Import PS module 'AzAPICall' (1.1.1) succeeded

Is the AzAPICall an issue?

please try the dev branch; the scriptDir will be passed through as parameter scriptPath

Sorry I just got some free bandwidth to try this - the dev branch seems to have gone. Was it removed?

merged :)

closed / inactive


Tooling: Create Yeoman Generator for Function.json

Considering the finite number of Triggers and Input/Output Bindings, it should be simple to create a Yeoman generated to help scaffold a Function.json file for a local development scenario. There was some feedback in the //Build content regarding this:

If using source control integration, is it required to create all the files from scratch by hand, or is it somehow possible to create a "starter project" based on an existing function? Are these files exportable somehow? -- Trond Hindenes

I've been thinking about tooling, so I'll factor this into my planning.

We have a Yeoman generator now.


[WIP] Custom Layout Factory

TODO: Exclude the module dependencies while repackaging.

okie. I added a commit to fix the maven wrapper :)
what should I do now? how would I test this?

Build the project once and please change the build plugin config like below.
  <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>2.0.0.RELEASE</version>
                <configuration>
                    <layoutFactory implementation="zipkin.layout.CustomLayoutFactory">
                        <name>custom</name>
                    </layoutFactory>
                    <!-- exclude dependencies already packaged in zipkin-server -->
                    <!-- https://github.com/spring-projects/spring-boot/issues/3426 transitive exclude doesn't work -->
                    <excludeGroupIds>org.springframework.boot,org.springframework,org.slf4j,commons-logging,com.google.code.gson</excludeGroupIds>
                    <!-- excludes direct dependency instead of the group id, as otherwise we'd exclude ourselves -->
                    <excludeArtifactIds>zipkin</excludeArtifactIds>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>io.zipkin.java</groupId>
                        <artifactId>zipkin-layout-factory</artifactId>
                        <version>0.0.1-SNAPSHOT</version>
                    </dependency>
                </dependencies>
            </plugin>

        </plugins>


[Question] Worker crashed with 'Cannot run program "streamparse_run" '

I'm having an issue running wordcount on the single storm. 'sparse run/submit' works well. After submitting to the cluster, the worker ran into the exception as below. The cluster is running on storm 1.0.3 python2.7.
So appreciated it if can get any clue. Thanks
Exception in the worker logs:
https://gist.github.com/caden714/f1f7ab37751a8784713de15eb1d58241
Caused by: java.io.IOException: Cannot run program "streamparse_run" (in directory "/usr/local/src/das/apache-storm-1.0.3/storm-local/supervisor/stormdi
st/wordcount-8-1496872394/resources"): error=2, No such file or directory
config.json
{
"serializer": "json",
"topology_specs": "topologies/",
"virtualenv_specs": "virtualenvs/",
"envs": {
"prod": {
"user": "xxx",
"nimbus": "<IP_ADDRESS>",
"workers": [
"<IP_ADDRESS>"
],
"virtualenv_root": "/home/xxxx/virtualenvs",
"log": {
"path": "/var/log/storm/xxx/",
"file": "pystorm_{topology_name}{component_name}{task_id}_{pid}.log",
"max_bytes": 100000,
"backup_count": 10,
"level": "info"
},
"use_ssh_for_nimbus": false,
"use_virtualenv": false
}
}
}

If you have use_virtualenv set to false, you have to make sure streamparse_run is in the storm user's PATH.
I also notice that you're setting virtualenv_root, but then setting and use_virtualenv to false, which is a little confusing. virtualenv_root won't be used at all if you have use_virtualenv disabled.


Enable Training on 6GB Cards... with DeepSpeed?

I am trying to squeeze training onto my 6GB laptop RTX 2060, and cant quite manage it with the "low memory" config:
accelerate launch --num_cpu_threads_per_process 8 train_db.py \
--pretrained_model_name_or_path="/home/alpha/Storage/AIModels/Stable-diffusion/panatomy05full_0.7-AIModels_Anything-V3.0-pruned-fp16_0.3-Weighted_sum-merged.ckpt" \
--train_data_dir="/home/alpha/Storage/TrainingData/test/training_data" \
--output_dir="/home/alpha/Storage/TrainingOutput/test/" \
--prior_loss_weight=1.0 \
--resolution=512 \
--train_batch_size=1 \
--learning_rate=1e-6 \
--max_train_steps=1600 \
--use_8bit_adam \
--xformers \
--mixed_precision="fp16" \
--cache_latents \
--gradient_checkpointing \
--save_precision="fp16" \
--full_fp16 \
--save_model_as="safetensors" \

So, I figured I would investigate Deepspeed cpu offloading with the accelerate config... but I keep running into errors on both the git version and the 0.7.7 release from pypi. Here is an error from the pypi release:
Traceback (most recent call last):
  File "/home/alpha/clone/sd-scripts/train_db.py", line 332, in <module>
    train(args)
  File "/home/alpha/clone/sd-scripts/train_db.py", line 154, in train
    unet, text_encoder, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
  File "/home/alpha/.local/lib/python3.10/site-packages/accelerate/accelerator.py", line 619, in prepare
    result = self._prepare_deepspeed(*args)
  File "/home/alpha/.local/lib/python3.10/site-packages/accelerate/accelerator.py", line 805, in _prepare_deepspeed
    engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
  File "/home/alpha/.local/lib/python3.10/site-packages/deepspeed/__init__.py", line 125, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/home/alpha/.local/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 330, in __init__
    self._configure_optimizer(optimizer, model_parameters)
  File "/home/alpha/.local/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1210, in _configure_optimizer
    self.optimizer = self._configure_zero_optimizer(basic_optimizer)
  File "/home/alpha/.local/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1455, in _configure_zero_optimizer
    optimizer = DeepSpeedZeroOptimizer(
  File "/home/alpha/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 532, in __init__
    self._param_slice_mappings = self._create_param_mapping()
  File "/home/alpha/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 544, in _create_param_mapping
    lp_name = self.param_names[lp]
KeyError: <exception str() failed>
[2023-01-12 13:13:52,241] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 5398
[2023-01-12 13:13:52,244] [ERROR] [launch.py:324:sigkill_handler] ['/usr/bin/python', '-u', 'train_db.py', '--pretrained_model_name_or_path=/home/alpha/Storage/AIModels/Stable-diffusion/panatomy05full_0.7-AIModels_Anything-V3.0-pruned-fp16_0.3-Weighted_sum-merged.ckpt', '--train_data_dir=/home/alpha/Storage/TrainingData/test/training_data', '--output_dir=/home/alpha/Storage/TrainingOutput/test/', '--prior_loss_weight=1.0', '--resolution=512', '--train_batch_size=1', '--learning_rate=1e-6', '--max_train_steps=1600', '--use_8bit_adam', '--xformers', '--mixed_precision=fp16', '--cache_latents', '--gradient_checkpointing', '--save_precision=fp16', '--full_fp16', '--save_model_as=safetensors'] exits with return code = 1
Traceback (most recent call last):
  File "/home/alpha/.local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/alpha/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 43, in main
    args.func(args)
  File "/home/alpha/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 827, in launch_command
    deepspeed_launcher(args)
  File "/home/alpha/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 540, in deepspeed_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['deepspeed', '--no_local_rank', '--num_gpus', '1', 'train_db.py', '--pretrained_model_name_or_path=/home/alpha/Storage/AIModels/Stable-diffusion/panatomy05full_0.7-AIModels_Anything-V3.0-pruned-fp16_0.3-Weighted_sum-merged.ckpt', '--train_data_dir=/home/alpha/Storage/TrainingData/test/training_data', '--output_dir=/home/alpha/Storage/TrainingOutput/test/', '--prior_loss_weight=1.0', '--resolution=512', '--train_batch_size=1', '--learning_rate=1e-6', '--max_train_steps=1600', '--use_8bit_adam', '--xformers', '--mixed_precision=fp16', '--cache_latents', '--gradient_checkpointing', '--save_precision=fp16', '--full_fp16', '--save_model_as=safetensors']' returned non-zero exit status 1.

Is there anything in particular that needs to be changed for this repo to support deepspeed? Or maybe there is some other tweak to squeeze LORA onto 6GB?

Tried upgrading accelerate and downgrading pytorch to 1.13.1, still no dice:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/alpha/clone/sd-scripts/train_db.py:332 in <module>                                         │
│                                                                                │
│   329 │   │   │   │   │     help="steps to stop text encoder training, -1 for no training / Te   │
│   330                                                                                │
│   331   args = parser.parse_args()                                                               │
│ ❱ 332   train(args)                                                                              │
│   333                                                                                │
│                                                                                │
│ /home/alpha/clone/sd-scripts/train_db.py:154 in train                                            │
│                                                                                │
│   151                                                                                │
│   152   # acceleratorがなんかよろしくやってくれるらしい                                          │
│   153   if train_text_encoder:                                                                   │
│ ❱ 154 │   unet, text_encoder, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(   │
│   155 │   │   unet, text_encoder, optimizer, train_dataloader, lr_scheduler)                     │
│   156   else:                                                                                │
│   157 │   unet, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(unet, optimize   │
│                                                                                │
│ /home/alpha/.local/lib/python3.10/site-packages/accelerate/accelerator.py:872 in prepare         │
│                                                                                │
│    869 │   │   │   old_named_params = self._get_named_parameters(*args)                          │
│    870 │   │                                                                                │
│    871 │   │   if self.distributed_type == DistributedType.DEEPSPEED:                            │
│ ❱  872 │   │   │   result = self._prepare_deepspeed(*args)                                       │
│    873 │   │   elif self.distributed_type == DistributedType.MEGATRON_LM:                        │
│    874 │   │   │   result = self._prepare_megatron_lm(*args)                                     │
│    875 │   │   else:                                                                             │
│                                                                                │
│ /home/alpha/.local/lib/python3.10/site-packages/accelerate/accelerator.py:1093 in                │
│ _prepare_deepspeed                                                                               │
│                                                                                │
│   1090 │   │   │   │   │   │   if type(scheduler).__name__ in deepspeed.runtime.lr_schedules.VA  │
│   1091 │   │   │   │   │   │   │   kwargs["lr_scheduler"] = scheduler                            │
│   1092 │   │   │                                                                                │
│ ❱ 1093 │   │   │   engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)           │
│   1094 │   │   │   if optimizer is not None:                                                     │
│   1095 │   │   │   │   optimizer = DeepSpeedOptimizerWrapper(optimizer)                          │
│   1096 │   │   │   if scheduler is not None:                                                     │
│                                                                                │
│ /home/alpha/.local/lib/python3.10/site-packages/deepspeed/__init__.py:125 in initialize          │
│                                                                                │
│   122 │   assert model is not None, "deepspeed.initialize requires a model"                      │
│   123 │                                                                                │
│   124 │   if not isinstance(model, PipelineModule):                                              │
│ ❱ 125 │   │   engine = DeepSpeedEngine(args=args,                                                │
│   126 │   │   │   │   │   │   │   │    model=model,                                              │
│   127 │   │   │   │   │   │   │   │    optimizer=optimizer,                                      │
│   128 │   │   │   │   │   │   │   │    model_parameters=model_parameters,                        │
│                                                                                │
│ /home/alpha/.local/lib/python3.10/site-packages/deepspeed/runtime/engine.py:330 in __init__      │
│                                                                                │
│    327 │   │   │   model_parameters = self.module.parameters()                                   │
│    328 │   │                                                                                │
│    329 │   │   if has_optimizer:                                                                 │
│ ❱  330 │   │   │   self._configure_optimizer(optimizer, model_parameters)                        │
│    331 │   │   │   self._configure_lr_scheduler(lr_scheduler)                                    │
│    332 │   │   │   self._report_progress(0)                                                      │
│    333 │   │   elif self.zero_optimization():                                                    │
│                                                                                │
│ /home/alpha/.local/lib/python3.10/site-packages/deepspeed/runtime/engine.py:1210 in              │
│ _configure_optimizer                                                                             │
│                                                                                │
│   1207 │   │   optimizer_wrapper = self._do_optimizer_sanity_check(basic_optimizer)              │
│   1208 │   │                                                                                │
│   1209 │   │   if optimizer_wrapper == ZERO_OPTIMIZATION:                                        │
│ ❱ 1210 │   │   │   self.optimizer = self._configure_zero_optimizer(basic_optimizer)              │
│   1211 │   │   elif optimizer_wrapper == AMP:                                                    │
│   1212 │   │   │   amp_params = self.amp_params()                                                │
│   1213 │   │   │   log_dist(f"Initializing AMP with these params: {amp_params}", ranks=[0])      │
│                                                                                │
│ /home/alpha/.local/lib/python3.10/site-packages/deepspeed/runtime/engine.py:1455 in              │
│ _configure_zero_optimizer                                                                        │
│                                                                                │
│   1452 │   │   │   │   │   │   "Pipeline parallelism does not support overlapped communication,  │
│   1453 │   │   │   │   │   )                                                                     │
│   1454 │   │   │   │   │   overlap_comm = False                                                  │
│ ❱ 1455 │   │   │   optimizer = DeepSpeedZeroOptimizer(                                           │
│   1456 │   │   │   │   optimizer,                                                                │
│   1457 │   │   │   │   self.param_names,                                                         │
│   1458 │   │   │   │   timers=timers,                                                            │
│                                                                                │
│ /home/alpha/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:532 in   │
│ __init__                                                                                │
│                                                                                │
│    529 │   │                                                                                │
│    530 │   │   self._link_all_hp_params()                                                        │
│    531 │   │   self._enable_universal_checkpoint()                                               │
│ ❱  532 │   │   self._param_slice_mappings = self._create_param_mapping()                         │
│    533 │                                                                                │
│    534 │   def _enable_universal_checkpoint(self):                                               │
│    535 │   │   for lp_param_group in self.bit16_groups:                                          │
│                                                                                │
│ /home/alpha/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:544 in   │
│ _create_param_mapping                                                                            │
│                                                                                │
│    541 │   │   │   param_mapping_per_group = OrderedDict()                                       │
│    542 │   │   │   for lp in self.bit16_groups[i]:                                               │
│    543 │   │   │   │   if lp._hp_mapping is not None:                                            │
│ ❱  544 │   │   │   │   │   lp_name = self.param_names[lp]                                        │
│    545 │   │   │   │   │   param_mapping_per_group[                                              │
│    546 │   │   │   │   │   │   lp_name] = lp._hp_mapping.get_hp_fragment_address()               │
│    547 │   │   │   param_mapping.append(param_mapping_per_group)                                 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

Hi, Unfortunately I have no experience about DeepSpeed.
However, you seem to run train_db.py. It is intended to train DreamBooth. If you want to train LoRA, please use train_network.py (some argments should be modified.) LoRA training might be work with 6GB VRAM without full_fp16 option.

🤦 I knew train_network.py was LORA, but I apparently copy pasted the wrong file.
I figured full-fp16 would reduce VRAM usage, but I will leave it out if thats not the case.

Yeah, looks like LORA training uses about 5.1GB without deepspeed, torch inductor or anything weird like that.

I am trying to squeeze training onto my 6GB laptop RTX 2060

Same here. Looking to run any training on 6GB vram. So far I had most success with LORA training on Automatic1111 after adjusting some settings, but even though I manage to complete training it didn't do any actual training for some reason. Even after 5000 epochs on 7 images. So I am still looking for the solution.
If you are desperate I can recommend this colab: https://github.com/TheLastBen/fast-stable-diffusion

LoRA training might be work with 6GB VRAM without full_fp16 option.

I figured full-fp16 would reduce VRAM usage, but I will leave it out if thats not the case.

As well, I am pretty sure fp16 supposed to reduce VRAM usage, as FP16 precision consumes only half the memory for floating points as FP32.

That is great news. Are you using latest Dreambooth version in Auto1111? Cuz I have to use older one, otherwise I am getting OOM errors.
The only settings configuration that let me even start training without OOM error is:
Gradient checkpointing
adam
fp16
xformers

I don't have to do full fp16 experimental.

What is that?

https://github.com/bmaltais/kohya_ss
I am using bmaltais GUI for training. I don't train in automatic's GUI.
the full fp16 experimental is less VRAM consuming than mixed precision training. It's an option in the GUI i'm using.
I use Dreamlike Photoreal v2.
Yes, I train the text encoder too.

I would still like to get Deepspeed working if only so I can train at faster settings higher batch size.
I'm trying to investigate whats going wrong, but I can't even get the git version of DeepSpeed to work (see that linked issue above) :/

Also, --full_fp16 doesn't seem to change VRAM consumption or training speed for me.

I am using bmaltais GUI for training.

@lolxdmainkaisemaanlu
Are you running this on Windows?

I am using bmaltais GUI for training.

@lolxdmainkaisemaanlu
Are you running this on Windows?
How much RAM do you have?

Yes, I am running it on Windows 11. But I use a custom Windows 11.
I have 16 GB RAM.
Recently I tried LoRA training with batch size = 2 and only some 48 mb could not be allocated! I think with some tweaks and maybe deepspeed implementation it would be possible to even do batch size = 2 on 6 GB VRAM.
I think if deepspeed stage 2 is implemented LoRA training would be possible even on 4 GB VRAM cards!

@lolxdmainkaisemaanlu
I can already manage a batch size of 2 on my 2060, even without gradient checkpointing enabled or any stage of deepspeed enabled.
I am running my display through a laptop with an IGP though, so its VRAM is totally empty.

Yes, I am running it on Windows 11. But I use a custom Windows 11.

@lolxdmainkaisemaanlu
Ok. So bmaltais GUI is compatible with Windows then. Cuz I am getting some errors during training. I was thinking its because
bitsandbytes is not supported on Windows.
Are you able to train in Dreambooth or only LORA?

Yes, I am running it on Windows 11. But I use a custom Windows 11.

@lolxdmainkaisemaanlu
Ok. So bmaltais GUI is compatible with Windows then. Cuz I am getting some errors during training. I was thinking its because bitsandbytes is not supported on Windows.
Are you able to train in Dreambooth or only LORA?

bmaltais GUI should be fully compatible with Windows.
I actually use a different library of bitsandbytes because 1000 series cards are not compatible with it. It gives me kernel not found error. https://github.com/james-things/bitsandbytes-prebuilt-all_arch - This is what I use.
I have not tried Dreambooth training because I think 8GB VRAM is the minimum to do it properly. I only train in LoRA.

@lolxdmainkaisemaanlu
I can already manage a batch size of 2 on my 2060, even without gradient checkpointing enabled or any stage of deepspeed enabled. Says it sits at about 5700mb last I checked.
I am running my display through a laptop with an IGP though, so its VRAM is totally empty.

Yeah I think your igpu is helping you fit in batch size = 2 in 6GB VRAM. I don't have an igpu ( i5 12400F) so my 1060 6GB is running the display as well.
Actually I'm just lacking 48mb to be able to use batch size = 2 myself! Only need a tiny bit of optimization now.

@lolxdmainkaisemaanlu
I am getting this issue. It looks like bitsandbytes is causing some problems. Did you encounter any of this?
File "C:\Programs\Python\Python310\lib\site-packages\bitsandbytes_init_.py", line 5, in
from .optim import adam
File "C:\Programs\Python\Python310\lib\site-packages\bitsandbytes\optim_init_.py", line 5, in
from .adam import Adam, Adam8bit, Adam32bit
File "C:\Programs\Python\Python310\lib\site-packages\bitsandbytes\optim\adam.py", line 11, in
from bitsandbytes.optim.optimizer import Optimizer2State
File "C:\Programs\Python\Python310\lib\site-packages\bitsandbytes\optim\optimizer.py", line 6, in
import bitsandbytes.functional as F
File "C:\Programs\Python\Python310\lib\site-packages\bitsandbytes\functional.py", line 13, in
lib = ct.cdll.LoadLibrary(os.path.dirname(file) + '/libbitsandbytes.so')
File "C:\Programs\Python\Python310\lib\ctypes_init_.py", line 452, in LoadLibrary
return self.dlltype(name)
File "C:\Programs\Python\Python310\lib\ctypes_init.py", line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: [WinError 193]

Deepspeed does not seem to support 8bitAdam. https://github.com/huggingface/diffusers/pull/735.
I succeeded to work train_db.py with deepspeed by removing 8bitAdam option.
Deepspeed offloads optimizer states to cpu so VRAM usage is reduced even if using regular Adam.
However, train_network.py did not work because it is not compatible with "LoRANetwork".

on windows? if so howd you install deepspeed?

linux.

Yeah, looks like LORA training uses about 5.1GB without deepspeed, torch inductor or anything weird like that.
But I am leaving this open because I can't get either to work yet.

thanks for testing this.
were you able to get deepspeed to work with kohya?


KNL-1383; add X-Sakai-Session header so we can log UsageSession ID

Closing this PR to fix the incorrect JIRA on the commit. Opening #1180


Evolve scene test to a full town sample

The town must include all things that exists now and also:

Farm
Fountain
Road

We must have a killer scene to attract people to the project.

In order to do then above scene the steps are:

Create the river
Create the bridges
Create lines at both side of the river
Create the houses along the river
Create lines going to the main areas and the things in each area

temple
jail
buildings
stadium (sphere)

We will do it first at low scale and then make it larger.

The coordinates in my plane world in which I am working are: (5000, 4, 5000)

Done:
https://github.com/juntosdesdecasa/minecraft/blob/develop/server/data/python/scene1_0.ipynb


Detection splitter fix

Summary
fix detection splitter issue that subsets are splitted based on the number of images, not the number of instances.

How to test

Checklist

[x] I submit my changes into the develop branch
[ ] I have added description of my changes into CHANGELOG
[ ] I have updated the documentation accordingly
[ ] I have added tests to cover my changes
[ ] I have linked related issues)

License

[x] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
[ ] I have updated the license header for each file (see an example below)

# Copyright (C) 2020 Intel Corporation
#
# SPDX-License-Identifier: MIT

Please update tests.

I'll upload PR again.


Stock firmware file?

Hi! Do you happen to have the stock firmware file somewhere? :)

have you tried looking at the monoprice unofficial wiki? https://www.mpselectmini.com/


Error in news summarizer app

The README can also be improved by adding more details like the OPENAI_API_KEY needs to be set.

is this a new deployment or the existing one?

New deployment

did you run the command that's defined in the readme? https://github.com/ploomber/doc/tree/main/examples/panel/rag-news

We need to run it in local first? Ok checking

yes

there are probably some undocumented steps, please update the readme
you need an openai key to run the script locally, so it might be that


pull request for e2e test at 12/04/2016 15:45:03

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Fix file SD test

The original plan was to be able to handle bad formatted files and basically we still are.
For actual file SD bridges (i.e. not using file SD as user-editable files outside of config) atomic renaming should be advertised as best practice, though.
The occasional testing deadlock happened due to a chain of reasons which are of no concern for the actual code.
Hopefully this fixes it - I could no longer reproduce it.
@juliusv

:+1:


Add Slack webhook to make a wish form

Not technically part of Apiary

Done, script source lives at https://github.gatech.edu/RoboJackets/scripts/blob/master/apiary_make_a_wish_to_slack.gs


Cookie loading broken under Parallels

Cookie loading doesn't appear to work when run under a Parallels environment:
ERROR! ERROR! ERROR! ERROR! ERROR! ERROR! ERROR! ERROR! ERROR! ERROR!
Traceback (most recent call last):
  File "browser_cookie3\__init__.pyc", line 450, in _decrypt
  File "browser_cookie3\__init__.pyc", line 441, in _decrypt_windows_chromium
  File "browser_cookie3\__init__.pyc", line 109, in _crypt_unprotect_data
RuntimeError: Failed to decrypt the cipher text with DPAPI

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "browser_cookie3\__init__.pyc", line 466, in _decrypt
  File "Cryptodome\Cipher\_mode_gcm.pyc", line 567, in decrypt_and_verify
  File "Cryptodome\Cipher\_mode_gcm.pyc", line 508, in verify
ValueError: MAC check failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "nyt.py", line 567, in main
  File "nyt.py", line 284, in get_puzzle
  File "nyt.py", line 246, in load_cookies
  File "nyt.py", line 135, in chrome_bugfix
  File "browser_cookie3\__init__.pyc", line 426, in load
  File "browser_cookie3\__init__.pyc", line 468, in _decrypt
browser_cookie3.BrowserCookieError: Unable to get key for cookie decryption
ERROR! ERROR! ERROR! ERROR! ERROR! ERROR! ERROR! ERROR! ERROR! ERROR!

Ok, this appears to be because the "v10" encryption technique on Windows either changed its method of encrypting the master password when DPAPI isn't available.  The logic that browser_cookie3 uses to decrypt from the key is throwing an exception.
Need to dig through the Chromium sources to understand what's happening.
In the meantime, FireFox on Parallels will work.


Automatically distribute new userscript versions

Adapts CI to push the built userscripts if the version number changed since a previous commit.
Each update is given its own commit (no commit is created when the version number wasn't changed).
Two caveats:

This breaks if there's a force-push to main. However, since main will be protected from direct pushes, that bug should never occur.
The major downside is that these are pushed to main, so you might have to merge main into feature branches more often. On the other hand, those merges should always be fast-forwards/rebases, and shouldn't conflict with anything.

Hold on, I've protected main in such a way that direct pushes are not allowed, not sure whether that applies to GH Action pushes as well...

I'm closing this for now. It's getting way too late for me to fix this, and the approach is suboptimal anyway.
I think I've reached a possible solution wherein the job on the main branch creates a release branch and pushes the automated release commit to that new branch; and then a separate workflow run on the release branch does the checks required for the main branch, merges itself into the main branch after the checks complete, thereby bypassing the requirement that pushed commits have the check completed. However, it doesn't work in an automated way, because workflows can't run on commits created by other workflows.
Three possible solutions:

Use a custom token in the first workflow, that will allow the second workflow to run on the release commit, but keeps the spaghetti.
Don't require checks on pushes to main, that eliminates the need for the second workflow but makes it possible to mess up main
Don't distribute the compiled code in the source tree and use releases instead. I'd prefer this, but I'm a bit hesitant since I've already changed the script location earlier this week. OTOH, I'm sure I'll switch to releases eventually, and the longer I wait, the more script locations I'll have to change in the future.

Actually, working with releases will be a mess as well. Each time a script is updated, we'd need to copy the compiled files of all other unchanged scripts as well, and versions would need to be unique across all scripts. There are some alternatives but they go against the core idea of releases.
Option 4, which I'm preferring now, is to use a separate dist branch used only for the bot to commit on and people to fetch updates from, which is completely disassociated from main.

Implemented the separate dist branch idea and clarified the force-push problem. I've done a small test and it seems to work properly, it successfully pushed to dist (I've already reverted that change though).
Post-merge, the file in the dist folder in main should be recompiled with a version increment so that auto-updates can be used to transparently switch to the new location.


Decal Rendering System

This is not a high priority compared to some other features but adding here for discussion and possible implementation down the road before I forget.
Is your feature request related to a problem? Please describe.
libGDX has DecalBatch and decal Rendering. Mundus currently doesn't offer a way to use them. You CAN use decals with Mundus projects by rendering them from the runtime but modifying them in editor would be nice.
Describe the solution you'd like
Editor side, decals are already implemented (Gizmo system) which displays decals for lights in editor. A DecalComponent and decal rendering could be added too, which would be part of the runtime as well. This would be a separate system from Gizmos.
Additional context
You can kind of fake a decal currently by using a plane mesh with a transparent texture, but this lacks other features of libGDX decals like billboarding.

You CAN use decals with Mundus projects by rendering them from the runtime.

How?   Where in the Mundus runtime can I access decalBatch.add or decalBatch.flush?

You CAN use decals with Mundus projects by rendering them from the runtime.

How? Where in the Mundus runtime can I access decalBatch.add or decalBatch.flush?

You don't, you would manage the DecalBatch and render the decals from the runtime.

By default you wont get decals to reflect on water but if we add it in to Mundus then it will.

This would be amazing.  Could you add shadows too?

By default you wont get decals to reflect on water but if we add it in to Mundus then it will.

This would be amazing. Could you add shadows too?

It is possible but would need some modifications from the regular libgdx decals probably. Not sure how much of a pain it would be offhand but could look into it as part of this feature to see if its doable.


Implement the new Design on index.html to the other pages

Starting page have a new look. Update the other page to start matching it.
(Can be small pull request to start looking like it doesn't have to be the whole page)

@BennyCarlsson I can do it! Can I just PR a few pages or just one?

@BennyCarlsson I am also trying to make some design changes

@m4tr1k @viveksdf both can go ahead and go crazy ;) just remember that bigger pull requests are harder to merge because we get a lot of pull requests :)

@BennyCarlsson Nothing to change in index.html as its already clean but added a link to my mean stack demo repo.  You can add or change almost anything even if there is no issue ;)

closing this but feel free to keep working on it :)


create shipping process doc

@Sheepy3 What's this for?

this issue was for creating a document describing how to pack a blot, which I believe you have been working on

I think I used this the other day so will mark this as done.
It's this correct?
https://github.com/hackclub/blot/blob/main/docs/packaging/PACKING.md


Switch to argparse for argument parsing

As suggested in #43, switch to the argparse module for argument parsing.

This is awesome--I just had one small nitpick, the comment I made inline. Thanks for the contribution, and sorry it took so long to review! :100: :sparkling_heart:

Thanks, @DanNixon! :+1:

No problem.


Some project from the DB are null, don't try to construct object for …

…them
Should fix #342 http://lib.haxe.org/t/cross/ fails with Invalid field access : name
but http://lib.haxe.org/t/cross.json/ works and contains some null

Hmm, @andyli is that because we removed some haxelibs from the projects table, but forgot to remove relevant links from tags? I'll merge this because it'll wont hurt anyway, but there's some invalid records in tags.


Error reporting is set too late

Affected Symphony version(s) : 2.7.x
PHP version(s) : 7.2
OS(es) : all
Officially Symphony 2.7 is not supported on PHP 7.2. But, given the fact that it's an LTS release, I nevertheless suggest to keep it working if possible.
PHP 7.2 has an interesting bug. Accoording to the docs, the variant parameter's default value for the idn_to_utf8 function is INTL_IDNA_VARIANT_2003 — but this value has been deprecated. (No, I am not kidding.) So calling the function without the parameter will raise a deprecation warning. The official bug report can be seen here: https://bugs.php.net/bug.php?id=75609.
If PHP's default error reporting is set to E_ALL, this PHP bug will cause the deprecation warning to be thrown on every page:
Deprecated: idn_to_utf8(): INTL_IDNA_VARIANT_2003 is deprecated in /var/www/foo/symphony/lib/boot/defines.php on line 218

Why? Symphony sets its own (more forgiving/suppressing) error reporting too late. If you look at index.php, you see:
// Include autoloader:
require_once DOCROOT . '/vendor/autoload.php';

// Include the boot script:
require_once DOCROOT . '/symphony/lib/boot/bundle.php';

Currently Symphony sets the error reporting in bundle.php. But the autoloader will load defines.php before! In this file (line 218) you will see:
define_safe('HTTP_HOST', function_exists('idn_to_utf8') ? idn_to_utf8($http_host) : $http_host);

There are two possible (simple) fixes for this:

Move the line that defines Symphony's default error reporting from bundle.php to index.php (before the autoloader). This would mean that Symphony's error handling is applied to everything that "happens" in the autoloaded files.
Fix only the line that causes the issue, by calling the function like so: idn_to_utf8($http_host, IDNA_DEFAULT, INTL_IDNA_VARIANT_UTS46).

@nitriques: I am open to both solutions, and I can send a PR if you tell me which route to take.

@michael-e Moving the default error reporting up seems like a breaking change, so for LTS, it leaves us with only solution 2. Are we sure that INTL_IDNA_VARIANT_UTS46 exists in php 5.3 (it looks like it) ? Also, do they produce the same result for the same input ?
As for 3.0.x, I am still hesitant. So please send a PR for 2.7.x with solution 2. (I have a 5.3 setup still available so I can test it)

Unortunately it is a breaking change no matter which route we take. PHP has screwed up here.
Using INTL_IDNA_VARIANT_UTS46, I do not expect the result to be the same. A further explanation is here: https://wiki.php.net/rfc/deprecate-and-remove-intl_idna_variant_2003. I wonder how this could ever happen. The author explicitly mentions that "code relying on the default variant might break". Nevertheless everyone agreed with this change.
Even worse for us: INTL_IDNA_VARIANT_UTS46 was not introduced before PHP 5.4, so we need to do s.th. like the following:
$http_host = server_safe('HTTP_HOST');
if (function_exists('idn_to_utf8')) {
    if (defined('INTL_IDNA_VARIANT_UTS46')) {
        $http_host = idn_to_utf8($http_host, 0, INTL_IDNA_VARIANT_UTS46);
    } else {
        $http_host = idn_to_utf8($http_host);
    }
}
define_safe('HTTP_HOST', $http_host);
unset($http_host);

What do you think?

@michael-e

Unortunately it is a breaking change no matter which route we take. PHP has screwed up here.

Why is a deprecation warning a breaking change? These warnings are good and important! And these are only hints that your code needs an update for the next versions of PHP. But it still works.
Btw. version 5.3, 5.4, 5.5 are dead. By the end of the year version 5.6 and 7.0 are also dead.

Symphony sets its own (more forgiving/suppressing) error reporting too late.

Symphony should not set anything here! Setting the error reporting to something lower hides only the problems in your code, but do not solve them.
If someone uses a dead or old PHP version, they can hide those deprecation warnings themselves.

I agree with most of your comment. However, we simply don't have the manpower to fix every warning in Symphony 2.7 LTS. (I have started to work on it, but still I have not come very far. I would be glad if some more experienced programmer would take care of it. We even invented a configuration setting called error_reporting_all which simplifies finding warnings that are suppressed otherwise.)

If someone uses a dead or old PHP version, they can hide those deprecation warnings themselves.

But: The deprecation warning happens in PHP 7.2.

Why is a deprecation warning a breaking change?

It is not. Our attempt to fix it is a breaking change, because we must use INTL_IDNA_VARIANT_UTS46 if possible.

@michael-e

However, we simply don't have the manpower to fix every warning in Symphony 2.7 LTS.

Right, a lot of work for the entire codebase.

I have started to work on it…

Great! 👍

But: The deprecation warning happens in PHP 7.2.

Sorry, I forgot to add a comment for solution / workaround: Your solution is a typical way to solve this type of problem.
(Notice: If a user-defined function is allowed to fill the gap, then you should use function_exists('idn_to_utf8') otherwise extension_loaded('intl').)

Thanks!
So the suggested fix is:
$http_host = server_safe('HTTP_HOST');
if (extension_loaded('intl')) {
    // In PHP 7.2, `idn_to_utf8` can not be called with default parameters,
    // because the default for `variant` has been deprecated. However, the
    // alternative variant `INTL_IDNA_VARIANT_UTS46` was not introduced until
    // PHP 5.4, so we must be careful.
    // https://wiki.php.net/rfc/deprecate-and-remove-intl_idna_variant_2003
    // https://bugs.php.net/bug.php?id=75609
    if (defined('INTL_IDNA_VARIANT_UTS46')) {
        $http_host = idn_to_utf8($http_host, 0, INTL_IDNA_VARIANT_UTS46);
    } else {
        $http_host = idn_to_utf8($http_host);
    }
}
define_safe('HTTP_HOST', $http_host);
unset($http_host);

Sorry, I missed one return value: false. This case should also be considered!
https://secure.php.net/manual/en/function.idn-to-utf8.php#refsect1-function.idn-to-utf8-returnvalues

Or, maybe clearer:
$http_host = server_safe('HTTP_HOST');
if (extension_loaded('intl')) {
    // In PHP 7.2, `idn_to_utf8` can not be called with default parameters,
    // because the default for `variant` has been deprecated. However, the
    // alternative variant `INTL_IDNA_VARIANT_UTS46` was not introduced until
    // PHP 5.4, so we must be careful.
    // https://wiki.php.net/rfc/deprecate-and-remove-intl_idna_variant_2003
    // https://bugs.php.net/bug.php?id=75609
    if (defined('INTL_IDNA_VARIANT_UTS46')) {
        $host_utf8 = idn_to_utf8($http_host, 0, INTL_IDNA_VARIANT_UTS46);
    } else {
        $host_utf8 = idn_to_utf8($http_host);
    }

    if ($host_utf8 !== false) {
        $http_host = $host_utf8;
    }
}
define_safe('HTTP_HOST', $http_host);
unset($http_host);

Boy does this change sucks. But I think it is the only viable solution. Here would be my amended version:
$http_host = server_safe('HTTP_HOST');
if (function_exists('idn_to_utf8')) {
    // In PHP 7.2, `idn_to_utf8` can not be called with default parameters,
    // because the default for `variant` has been deprecated. However, the
    // alternative variant `INTL_IDNA_VARIANT_UTS46` was not introduced until
    // PHP 5.4, so we must be careful.
    // https://wiki.php.net/rfc/deprecate-and-remove-intl_idna_variant_2003
    // https://bugs.php.net/bug.php?id=75609
    // @deprecated: This 'hack' can be removed when dropping php < 7.2
    if (defined('INTL_IDNA_VARIANT_UTS46')) {
        $host_utf8 = idn_to_utf8($http_host, 0, INTL_IDNA_VARIANT_UTS46);
    } else {
        $host_utf8 = idn_to_utf8($http_host);
    }

    if ($host_utf8 !== false) {
        $http_host = $host_utf8;
    }
    unset($host_utf8);
}
define_safe('HTTP_HOST', $http_host);
unset($http_host);

I would keep function_exists('idn_to_utf8') because I believe in user supplied versions!
I would also unset $host_utf8
I would add a deprecation notice in the comment
Remember that LTS does not cover php 7.2. 3.0.0 will.
Wait: there's secure.php.net ?

So @michael-e, wanna send a PR ? I might be easier to comment on it if a PR is open. Thanks guys.

@nitriques

I would keep function_exists('idn_to_utf8') because I believe in user supplied versions!

Mmmh, but your composer.json doesn't include anything! 😉
Or do you know a user-defined function for this case?

I didn't expect this to get so complicated… well, I will send a PR for further discussion.
@nitriques: Actually the hack could be removed when dropping PHP 5.3. For 5.4 and later we could use idn_to_utf8($http_host, 0, INTL_IDNA_VARIANT_UTS46). Calling the function without parameters is required in 7.2 and 7.3, since the deprecated default will not be changed before 7.4. PHP has really shot itself in the foot here.

I fixed a typo in the above comment. Sorry for that.

@froschdesign

Mmmh, but your composer.json doesn't include anything!

Can you elaborate more ? I do not see the relationship... sorry!
@michael-e Thanks!

@nitriques

Can you elaborate more ? I do not see the relationship... sorry!

Nothing special: you want allow a user-defined function to fill the gap, but the entire CMS does not use any other external packages or libraries. That is a contradiction.

@froschdesign Ah, in that sense, valid point. This is changing in 3.0.x, as we will depend on composer and let users install other things with it.
But a well-advised (or ill-advised, it depends!) user like myself will not hesitate to custom patch index.php to provide things I need to get things done.


Overlay keeps hiding beneath the game.

I've been using this one for quite a few weeks and it was working perfectly fine. Now recently I can't get the overlay to appear above my game anymore. No matter how much I switch, it's not popping up anymore. It's on the right screen, my game is fullscreen windowed. I don't understand why my overlay keeps dissapearing underneath my game.

Welp, I can tell you that you're not the only one at least -- I haven't been able to find out why this happens yet though :(
In the meantime, could you double-check that you're using the most recent version of both the console app (cs-hud-win.exe) and the overlay (cs-hud.exe from cs-hud-win32-x64.zip)?

I recently installed it again since I thought that would fix it. So it's the most recent version yeah.

Bit of a long shot, but could you try this build and see if it's an issue there too? https://drive.proton.me/urls/VW9GE52244#NiC2S16Xw2B4https://drive.proton.me/urls/VW9GE52244#NiC2S16Xw2B4
for reference (for me and others), this one has different Electron BrowserWindow params: autohideMenuBar: true, focusable: false, and it calls browserWindow.setAlwaysOnTop(true, 'screen-saver'), compared to https://github.com/drweissbrot/cs-hud/blob/bcecde36b9d08bfe1b32e995bcbcacd6c716ca4a/src/electron/hud.js

Sorry for the very late replay but I can't open that link!

Sorry for the very late replay but I can't open that link!

whoops, should work now :D

I came here because I encountered the same thing.
The build you sent doesn't even open for me BUT
I have found a temp fix for it.

Open cs-hud.exe
Drag any window over top of CS2 (for example discord) and make it fill the screen
Switch back to the HUD
Problem solved as long as you keep it open!

Very strange issue, but at least this workaround works (for whatever reason)

Вы можете попробовать такой вариант, в программе OBS выбрав браузер где у вас установлен HUD, можете наложить на него фильтр "хромакей". И тогда волшебным образом HUD останется на месте, а зеленое полотно за ним пропадет и будет видно ваш матч!

I had the same problem and solved it by opening it in a specific sequence:

Run cs-hud-win.exe
Open http://localhost:31982/hud in your favorite browser, and you should see the HUD.
Run cs-hud.exe on Windows
Start CS2 and Find a match to spectate in CS. The HUD should now overlay your screen.


Bad error handling in config

I just spent a long time debugging a debian docker image which segfaulted when running cargo metadata. During this I encountered a counter-intuitive error path here: https://github.com/rusterlium/rustler/blob/fbedf17f8249fe6a1b23d69ff5e838388a89b27f/rustler_mix/lib/rustler/compiler/config.ex#L80-L89
System.stop is documented as having the spec no_return(), but it actually calls an underlying Erlang function that schedules a shutdown and returns :ok. This means the code continues running for a short while, which in my case was long enough to hit a type error inside Jason.decode!.
I don't know what the "correct" fix is, but printing an error and calling System.halt instead would have saved me a couple of hours.

I think the correct way to handle this would be to either match on {metadata, 0} or by raising an exception I think. Shutting down the whole VM looks wrong, I didn't think that through in #386. I won't be near a working development environment for some time, so maybe somebody of @rusterlium/core could take a look into this.


[flink] adding Flink to right assist Function panel

What changes were proposed in this pull request?

#GH-1629 - [flink] adding Flink to right assist Function panel

How was this patch tested?

Tested manually in local, The Right assist panel has to show flink and associated group window functions

:+1:


es6-module-loader.js not found

Hi,
starting with gulp watch and opening http://localhost:9000 i am getting the  not found from
http://localhost:9000/es6-module-loader.js.
it is located in jspm_packages/es6-module-loader.js.

This happens only in chrome (Version 40.0.2214.93 m).
Firefox seems to hadle it correctly (http://localhost:9000/jspm_packages/es6-module-loader.js) version is 37.0a2.
IE 11 works too.

Really? Only in chrome. That is bizarre. It looks like the path is incorrect though. Can you tell me more about your setup?

for starters windows 7.
i can't think of something else, any other info?

funny thing. it works in incognito mode! maybe a plugin, extension?

it is a extensions that does this job. investigating.

i found the culprit. it is a extensions named poper blocker. crazy?

Strange. What does it do actually? It looks like the path to the script was wrong. Very odd.

only when i whitelist localhost or disable the extension does it work. never mind, i killed that extension.


Could not build phar

Hi. When I try to create phar, this appears:
Warning: yaml_parse_file(): scanning error encountered during parsing: could not find expected ':' (line 6, column 1), context while scanning a simple key (line 5, column 1) in /var/www/pmt.mcpe.fun/pages/zip2phar/convert.php on line 78
And then, this: Invalid plugin.yml provided
Help :(

Hello, please check your plugin.yml. The issue seems to come from there. There is a ':' which shouldn't be there.

I don't found anything unusual.

Can you show it to us? The error basicly shows us that your plugin.yml is malformed.

This will be fixed in v2..


change the url that scrape tweet json from it

the url changed
from :https://cdn.syndication.twimg.com/tweet?id=
to : https://cdn.syndication.twimg.com/tweet-result?id=

Thank you for me MR @abdouhl ! 😁


SLR72 is about Colombian Spanish not "Columbian"

"Columbia" and "Columbian" refer to locations in North America. The country is Colombia. The cited paper gets the spelling right, but this repo and the SLR website have misspelled it.

Sure
On Fri, 23 Apr 2021, 13:24 Pasindu De Silva, @.***>
wrote:

@oddurk https://github.com/oddurk can you take a look at this
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/language-resources/issues/43#issuecomment-825621810,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABPJMV5B2EBAT5WFYOCUQUTTKFRJVANCNFSM426VYATA
.

This should now be fixed.


multiuse-invites: Add edit feature for multiuse invites

This pull request adds the feature to allow the editing of multiuse invites.

Screenshots of the changes
Edit Modal

Changed Icons

Video Demonstration
Screencast from 2024-12-28 14-12-25.webm

Fixes: #22099

Self-review checklist

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[x] Highlights technical choices and bugs encountered.
[x] Calls out remaining decisions and concerns.
[x] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[x] Responsiveness and internationalization.
[x] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

Hey @alya , could you please take a look at this one?

@whilstsomebody, there is a typo in the commit - nvites: Add the ability to edit multiuse invites on backend..
Kindly fix that. Thank you.

Thanks @saubhagya-patel !


how I can display plot?

hello,
I have written small plot barchart R code.
now i want to execute that code using your library so how I can see Plot generated by R on php webpage?
please check below code :-
<?php
require_once 'vendor/autoload.php';
use Kachkaev\PHPR\RCore;
use Kachkaev\PHPR\Engine\CommandLineREngine;

$r = new RCore(new CommandLineREngine('/usr/bin/R'));
echo var_dump($r);
$result = $r->run(<<<EOF

responses = c(25, 44, 29, 60, 76, 4, 11, 33, 34, 22, 22, 11, 11, 89)
barplot(responses)
EOF
 );

echo $result;
?>

Thanks

Hi @dipenpatel235. I am afraid that this is impossible directly, because PHP and R only exchange text. However, what you can get an image if you save your boxplot to a file with R code and then read that PNG in PHP.
I'll close this issue as it is a question rather than a bug report, but feel free to add more comments.


Drag and drop handler

HI,
I would like to stop the drag and drop functionality in col alone, say I can add or remove rows but the cols has to be constant , may I know how to achieve this.

This is not a supported feature.


UI Test failure on .NET Core build verification

Edit:
Full log here: https://github.com/SAFE-Stack/SAFE-BookStore/issues/311#issuecomment-371190446
Environmental info
VS: D15.7stg/27505.3002 (15.7 Preview 1)
Node.js: V8.9.4 LTS
Yarn --version:1.5.1
Paket version: 5.148.0
VS .NET Core SDK - 2.1.100: 2.1.100 ( runtime - 2.0.5)
Repro steps
Install the needed deps for SAFE-Bookstore and run build.cmd. Error produced:
1) Fake.UnitTestCommon+FailedTestsException: Expecto test of assembly 'C:\Users\CORBVT\AppData\Local\Packages\Microsoft.MicrosoftEdge_8wekyb3d8bbwe\TempState\Downloads\SAFE-BookStore-master\SAFE-BookStore-master\test\UITests\bin\Debug\net461\UITests.exe' failed. Process finished with exit code 2.
   at Fake.Testing.Expecto.Expecto(FSharpFunc`2 setParams, IEnumerable`1 assemblies) in D:\code\fake\src\app\FakeLib\UnitTest\Expecto\Expecto.fs:line 126
   at FSI_0005.Build.clo@163-12.Invoke(Unit _arg10)
   at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in D:\code\fake\src\app\FakeLib\TargetHelper.fs:line 626

Note: the VS version is likely not needed; it's just there since that's what was installed.
I'm not sure how the UI tests here work with Canopy, but I suspect that this isn't related to the version of the .NET Core SDK that's under test.

can you please give full error? this not showing the actual expectot output

I confirm that, got this thing when running release script. Full error:
[17:00:16 ERR] client tests/create and remove book errored in 00:02:40.0230000 <Expecto>
OpenQA.Selenium.WebDriverTimeoutException: Timed out!
   at canopy.wait.waitResults[a](Double timeout, FSharpFunc`2 f)
   at canopy.core.op_ComposeLeft[a](a item, String text)
   at UITests.Tests.tests@70-3.Invoke(Unit unitVar0) in C:\Users\pese\Desktop\SAFE-BookStore\test\UITests\Tests.fs:line 85
   at Expecto.Impl.execTestAsync@874-1.Invoke(Unit unitVar)
   at Microsoft.FSharp.Control.AsyncBuilderImpl.callA@522.Invoke(AsyncParams`1 args)
[17:00:16 INF] EXPECTO! 4 tests run in 00:05:08.4797301 - 3 passed, 0 ignored, 0 failed, 1 errored.  <Expecto>
[ERROR - 2018-03-07T16:00:17.740Z] RouterReqHand - _handle.error - {"stack":"_translateKey@phantomjs://code/inputs.js:268:55\n_sendKeys@phantomjs://code/inputs.js:217:42\n_postValueCommand@phantomjs://code/webelement_request_handler.js:288:41\n_handle@phantomjs://code/we
belement_request_handler.js:72:30\n_reroute@phantomjs://code/request_handler.js:61:20\n_handle@phantomjs://code/session_request_handler.js:120:42\n_reroute@phantomjs://code/request_handler.js:61:20\n_handle@phantomjs://code/router_request_handler.js:78:46","line":268,"so
urceURL":"phantomjs://code/inputs.js"}

  phantomjs://platform/console++.js:263 in error
[INFO  - 2018-03-07T16:00:17.794Z] ShutdownReqHand - _handle - About to shutdown
Running build failed.
Error:
Expecto test of assembly 'C:\Users\pese\Desktop\SAFE-BookStore\test\UITests\bin\Debug\net461\UITests.exe' failed. Process finished with exit code 2.
Starting FinalTarget: KillProcess
Searching for processes with name = dotnet
Trying to kill process dotnet (Id = 13372)
Trying to kill process dotnet (Id = 5880)
Searching for processes with name = dotnet.exe

---------------------------------------------------------------------
Build Time Report
---------------------------------------------------------------------
Target              Duration
------              --------
Clean               00:00:00.0965289
InstallDotNetCore   00:00:00.2429785
InstallClient       00:00:03.2913431
SetReleaseNotes     00:00:00.0049714
BuildServer         00:00:04.3700232
BuildClient         00:00:28.4515895
BuildServerTests    00:00:06.4289201
RunServerTests      00:00:09.4134053
BuildClientTests    00:00:03.0007435
RenameDrivers       00:00:00.0009485
KillProcess         00:00:00.0317423
RunClientTests      Failure
Total:              00:06:35.0042812
---------------------------------------------------------------------
Status:             Failure
---------------------------------------------------------------------
---------------------------------------------------------------------
  1) Fake.UnitTestCommon+FailedTestsException: Expecto test of assembly 'C:\Users\pese\Desktop\SAFE-BookStore\test\UITests\bin\Debug\net461\UITests.exe' failed. Process finished with exit code 2.
   at Fake.Testing.Expecto.Expecto(FSharpFunc`2 setParams, IEnumerable`1 assemblies) in D:\code\fake\src\app\FakeLib\UnitTest\Expecto\Expecto.fs:line 126
   at FSI_0005.Build.clo@163-12.Invoke(Unit _arg10)
   at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in D:\code\fake\src\app\FakeLib\TargetHelper.fs:line 626
---------------------------------------------------------------------

Workaround for release: comment/remove   ==> "RunClientTests" and ==> "TestDockerImage" in the build script with all the consequences.

Thanks for the repro @psfinaki - I've been running around over the past day and haven't had the time to get the full error.

@psfinaki is it that canopy can't find the remove button?
@cartermp are you sure it's the same thing?

@forki, I do not know actually, it might take quite some time for me to understand.. I have not worked with canopy before and also still have a vague idea how things work and get tested here.

No problem. I was seeing that "remove" error today on one of my machines. So I was wondering if it's the same. But it does not happen on CI.

I suspect it may be the same; I can try it again sometime later this week but between MVP summit and dealing with our apartment getting sold I'm short for time this week 😢

don't worry. let's assume it's the same and fix. after a while you can check if you still see the error ;-)

Sounds like a good plan!

Repro on D16.0/28322.00 (16.0 preview 1.0)

Repro on D16.4/29512.137(16.4.0 Preview 5.0)

error message on SDK 6.0.102
Running build failed.
Error:
System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.HttpWebRequest.GetResponse () [0x00016] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.WebClient.DownloadBits (System.Net.WebRequest request, System.IO.Stream writeStream) [0x000e6] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.WebClient.DownloadFile (System.Uri address, System.String fileName) [0x00088] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.WebClient.DownloadFile (System.String address, System.String fileName) [0x00008] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadFile(string,string)
at Fake.DotNetCli.downloadSDK@521 (System.String tempDir, System.String downloadPath, System.String archiveFileName) [0x0005b] in <5c90a2e0ccf1c534a7450383e0a2905c>:0
at Fake.DotNetCli.InstallDotNetSDK (System.String sdkVersion) [0x00230] in <5c90a2e0ccf1c534a7450383e0a2905c>:0
at<EMAIL_ADDRESS>(Microsoft.FSharp.Core.Unit _arg2) [0x00006] in :0
at Fake.TargetHelper.runSingleTarget (Fake.TargetHelper+TargetTemplate`1[a] target) [0x00049] in <5c90a2e0ccf1c534a7450383e0a2905c>:0

Build Time Report
Target              Duration

Clean               00:00:00.0198982
InstallDotNetCore   Failure
Total:              00:21:52.3333578
Status:             Failure

System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.HttpWebRequest.GetResponse () [0x00016] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.WebClient.DownloadBits (System.Net.WebRequest request, System.IO.Stream writeStream) [0x000e6] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.WebClient.DownloadFile (System.Uri address, System.String fileName) [0x00088] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at System.Net.WebClient.DownloadFile (System.String address, System.String fileName) [0x00008] in <2cef2219e8e14eb5a8eef57e30f81ac0>:0
at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadFile(string,string)
at Fake.DotNetCli.downloadSDK@521 (System.String tempDir, System.String downloadPath, System.String archiveFileName) [0x0005b] in <5c90a2e0ccf1c534a7450383e0a2905c>:0
at Fake.DotNetCli.InstallDotNetSDK (System.String sdkVersion) [0x00230] in <5c90a2e0ccf1c534a7450383e0a2905c>:0
at<EMAIL_ADDRESS>(Microsoft.FSharp.Core.Unit _arg2) [0x00006] in :0
at Fake.TargetHelper.runSingleTarget (Fake.TargetHelper+TargetTemplate`1[a] target) [0x00049] in <5c90a2e0ccf1c534a7450383e0a2905c>:0

error message on SDK 6.102:
Logs.txt


Add TypeScript definitions + wsOptions to options object

There are now TypeScript definitions for the library, with support for arbitrary property names. Additionally, options.wsOptions is passed into the WebSocket server constructor.
Fixes #4

That's awesome, thanks! Added some documentation and merged it, it's up on npm now.


How can i get all RoadNotes between two different location in OSMDROID or Mapquest

I want to get graph of all roadnotes between two different location in osmdroid.How can i do that ?

Duplicate with https://stackoverflow.com/questions/52576059/how-can-i-get-all-roadnotes-between-two-different-location-in-osmdroid-or-mapque


VHIO-CRC exome word press page

Platform description has been recieved from VHIO-CRC. We simply need to create the word press page and test that the platform URLS link appropriately on DEV.
https://app.zenhub.com/files/78657127/d2d13073-731b-48a1-bd48-5ebac90644af/download
Acceptance criteria

[] word press page has been created
[] URLS have been added to URL sample platform
[] platforms link to word press page on DEV

https://app.zenhub.com/files/78657127/61d330e3-3beb-4385-91ee-9d679292ac3d/download

Word press page on Dev: http://dev.pdxfinder.org/platform/vhio-crc-wes/

word on DEV. done


.Net: include parameter in QueryEmbeddingsAsync() in ChromaClient should not be null

Describe the bug
The signature of this function is below,
public async Task QueryEmbeddingsAsync(string collectionId, ReadOnlyMemory[] queryEmbeddings, int nResults, string[]? include = null, CancellationToken cancellationToken = default)
The last second parameter - include - should not be null as default; otherwise Chroma will throw exception in its python code.
The screenshot of exception will happen when "include" is null.

The value of include should be "documents", "ids", and so on.
Why not implement it as Enum array instead of string array to avoid typo.

Hi @dmytrostruk, does this item still need to be open? Thanks.


v1.40 pc.createMesh implementation breaks custom collision meshes

The new implementation of pc.createMesh introduced in v1.40 breaks custom collision meshes consumed by Ammo.js. I can confirm when using the older v1.39.4 build everything works as expected.
Steps to Reproduce

Open the Terrain From Heightmap example
Launch and watch the collision mesh being out of place (seems inverted or something?)
Launch with ?use_local_engine=https://code.playcanvas.com/playcanvas-1.39.4.js, everything works as expected.

I see the issue .. CollisionMeshSystemImpl.createPhysicalShape assumes mesh format is interleaved .. I'll fix that code.
var stride = format.size / 4;

I've added this project to the engine test plan. I'll deploy 1.40.2 after this bug is fixed.


npm install is stuck on fetchmetadata

Can you please check if things are breaking on npm install?

Hi, I don't maintain this module anymore, so I recommend to not use it anymore.

Thanks for the update Luca. Would be great if you can mention deprecated or
not maintained in NPM registry as well so that developers won’t be confused.
But thank you for writing this code, I was able to implement the file
directly in my code base without using npm and it still works fine.
On Wed, 20 Nov 2019 at 4:57 PM, Luca Steeb<EMAIL_ADDRESS>wrote:

Hi, I don't maintain this module anymore, so I recommend to not use it
anymore.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/steebchen/text-to-picture/issues/4?email_source=notifications&email_token=AJD5WDKR7I6KDDBUCBV6J7DQUUNJ7A5CNFSM4JPEYITKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEERVDLA#issuecomment-555962796,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJD5WDORCD44NAYFIWKE4MTQUUNJ7ANCNFSM4JPEYITA
.

--
With Regards,
Praveen Thirumurugan.
https://stackoverflow.com/users/10314103/praveen-thirumurugan
https://github.com/thirupraveen18 https://linkedin.com/in/thirupraveen18
https://twitter.com/thirupraveen18 https://instagram.com/thirupraveen18
https://dribbble.com/thirupraveen18

Ok sounds good. I'll deprecate it.


Implement system call fuzz testing

We'd like to have fuzz testing implemented for system calls to give us more confidence in the correctness of our system call handler checks.
We could develop our own framework, or use something which exists and adapt it to Zephyr.
Reading material:
https://github.com/google/syzkaller
https://lwn.net/Articles/677764/
https://en.wikipedia.org/wiki/Fuzzing
https://codemonkey.org.uk/projects/trinity/

I've taken a look on this, it's worth to add afl-unicorn https://hackernoon.com/afl-unicorn-fuzzing-arbitrary-binary-code-563ca28936bf
afl-unicorn can be used to fuzz virtually everything, not only syscalls and it seems the easiest one to use on Zephyr.

@ceolin does this have as a prerequisite porting Zephyr to Unicorn? This does seem like it might be a good path forward for fuzzing on Zephyr, though. We'd talked about something with Qemu, but if these people have already put in effort with Unicorn, it might get us a lot closer.

Kinda related: FWIW, we have made an internal "CPU load fuzzer", a dummy load to determine system stabillity if/when we loose service (Bluetooth connection for us) or even crash.
{mean, variance}-randomized {period, duty cycle} over {coop, irq} loads.
Not meant for security but good as a canary in the coalmine for merges.
Let me know if you would be interested, I could throw up a RFC PR.

I would like to see this happen as well. Is the 2.4 target realistic for adding fuzz support?

@jettr I don't know if anyone is currently working on this.
I myself am occupied with delivering demand paging and some other MMU-related work for 2.5.
@ceolin is this still in your scope?


Browser Bundles for Packages

Overview
The purpose of this PR is to generate browser bundles for each of the standalone packages. This is useful because it would allow us to create some web tooling to produce custom PixiJS builds without needing a bundler (e.g., Webpack, Rollup). Also, it would allow users to test by importing only the package they needed.
Minimal PixiJS Example
This is the minimum packages need to create a renderer (~170KB minified, ~52KB gzipped)
https://jsfiddle.net/bigtimebuddy/Lgadeocs/
<script src="https://pixijs.download/dev-bundles/packages/runner.min.js"></script>
<script src="https://pixijs.download/dev-bundles/packages/constants.min.js"></script>
<script src="https://pixijs.download/dev-bundles/packages/settings.min.js"></script>
<script src="https://pixijs.download/dev-bundles/packages/ticker.min.js"></script>
<script src="https://pixijs.download/dev-bundles/packages/math.min.js"></script>
<script src="https://pixijs.download/dev-bundles/packages/display.min.js"></script>
<script src="https://pixijs.download/dev-bundles/packages/utils.min.js"></script>
<script src="https://pixijs.download/dev-bundles/packages/core.min.js"></script>
<script>
  const renderer = PIXI.autoDetectRenderer();
  document.body.appendChild(renderer.view);
</script>

Changed

Removes @pixi/fragments this is an unnecessary dependency, moved default vert shaders to core
Standardizes building minified browser bundles

What is namespaceInject and how does it help us?

Is a build option I created, not a convention. Basically it means append to existing namespace, don’t create it. For instance, all the classes in math, these should be added to PIXI global object and the math package should not set PIXI.

Would it be easier if each of them included the bit that created it if it didn't exist, and added themselves to it if it did? That way any bundle would create, and any bundle would add.
For example, why not have math create it if I only include math?

There are a handful of cases where it doesn’t do that: like accessibility, interaction, prepare, extract, utils where it’s easier to do what Rollup does out of the box. But I could probably just normalize around this behavior rather than having two build paths.

Hmm, I wonder if I should roll all the core dependencies into core bundle. Would really simplify things and those math, settings, etc packages need to be included in a specific order to make core work anyways.

Few more clean-up changes:

Reduce the dependencies between packages (ismobilejs, eventemitter3, url), this will prevent the dependency from being inlined twice if using bundle packages. Since most packages depend on utils, this doesn't affect anything or change internal package dependencies, but reduces the overall size.
Adds proper documentation for isMobile object exported by utils
Exports url dependency from utils to standardize external access to dependencies, just like EventEmitter, earcut and isMobile


Keep the window scale

any idea to limit the window size for the text? because when i use it in the full screen the text always fill the windows

That's the entire idea of that feature - that it fills the screen. Why do you want it to not do that?


It is possible to make this lib work with printers outside epson?

I know that is a SDK thing.. but im curious if someone made it work with other printers. I've tried several options in the config but I keep getting ERR_CONNECT. The printer using pure ESC/POS works well sending the bytes directly to the IP.

@marianolc  I suppose it's not possible ;(
Try to use https://www.npmjs.com/package/react-native-thermal-printer I suppose it should work with any type of printers since it's not based on any brand SDK


isConnected should be set on response closed and prevent pushing data

Response can also close without request emitting the close event.
both should be listened to set the isConnected boolean to false.
Also, pushing data should throw on a disconnected session.

Benchmarks look fine.
Before:
Benchmarking "Push events with channels".
better-sse x 45,644 ops/sec ±4.66% (85 runs sampled)

After:
Benchmarking "Push events with channels".
better-sse x 45,425.8 ops/sec ±5.66% (82 runs sampled)

Thx for the review !

These changes are now published in v0.11.0.


Dry tilestache

Moving get_tiles logic around so we dont repeat lines

This is no longer relevant, closing.


Send screen

Design and wire the send screen

Duplicate of https://github.com/get10101/10101/issues/58


Issue with auto indentation when hitting enter after a comment

Originally opened by @luabud as https://github.com/microsoft/pyrx/issues/4588

 Environment data

Language Server version: 2024.1.101
OS and version: win32 x64
Python version (and distribution if applicable, e.g. Anaconda):
python.analysis.indexing: true
python.analysis.typeCheckingMode: off

 Code Snippet

x = 10

def foo():
    """ this is foo """
    # comment

# this is another comment <--- hit enter here
def bla():
    """ this is bar """

 Repro Steps

Get the code sample below and hit enter on line 9

 Expected behavior
hitting enter after the commend would bring the cursor to the same level as the comment

 Actual behavior
hitting enter brings the cursor one level more than it should

 Logs

pylanceindentbug30124.txt

Duplicate of https://github.com/microsoft/pylance-release/issues/5488


Change search_in to array

This commit changes the search_in param to accept an array rather than a comma-separated string, in order to be consistent with other similar params like fields[].
Depends on https://github.com/alphagov/gds-api-adapters/pull/664.

looks good - any chance you could swap the single quotes to double quotes? 🙏
I'm going to see if we can get Rails strong params in there so we can ensure array at controller point.

Done


attempt at output standardization for approximate counting

This is an attempt to fix #907 and was also mentioned in #898

Is this ready to be merged @stormofice

Great, thanks!


add non insult bot option and fix typo in pillage command

After some consideration I've decided that since this isn't a really coveted feature and because it just creates more confusion in the structure of the db, etc.. I don't want this on Villager Bot, sorry.


Compiler Error: Module validation failed: Call parameter type does not match function signature!

The error message is;
Module validation failed: Call parameter type does not match function signature!
  %83 = load i8, i8* %82, !dbg !25
 %"Hash(String, Int16 | Int32 | Int64 | String | Time | UInt32)"*  %84 = call %"Hash(String, Int16 | Int32 | Int64 | String | Time | UInt32)"* @"*Hash(String, Hash(String, Int16 | Int32 | Int64 | String | Time | UInt32) | String)@Hash(K, V)#[]=<String, Hash(String, Int16 | Int32 | Int64 | String | Time | UInt32)>:Hash(String, Int16 | Int32 | Int64 | String | Time | UInt32)"(%"Hash(String, Hash(String, Int16 | Int32 | Int64 | String | Time | UInt32) | String)"* %47, %String* bitcast ({ i32, i32, i32, [5 x i8] }* @"'data'" to %String*), i8 %83), !dbg !24
 (Exception)
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues

The error comes during casting types when fetching postgres db data.
my table contains a JSONB field with a sample data tags = {"nature":5, "news":7 }
Doing this throws the error;
post = DB.query_one? "select unqid, tags, created_at from posts where unqid = $1", post_id,
                    as: {unqid: String, tags: Hash(String, Int32), created_at: Time}

Changing Hash(String, Int32) to JSON::Any fixes the issue but the compiler should handle this error gracefully and give a much more user friendly message.

Duplicate of https://github.com/crystal-lang/crystal/issues/6997


[FEATURE]: custom meta data for models and fields

Describe what you want
Hi,
Having the ability to attach custom metadata to models and fields could help make drizzle significantly useful for concerns that go beyond database interaction, e.g. auto forms generation, data bootstrapping scripts.
There are a few important areas where automatic code generation can help significantly speed up the app development process. These areas are also closely related to the model. For example:

form generation. Metadata can be used to indicate the preferred form type: e.g., select dropdown vs autocomplete combobox, whether select values are conditioned on another field, whether a field should be displayed or required in CRUD forms, etc.
admin site generation
potential to leverage these metadata to customize the drizzle studio with application specific constraints. This could turn drizzle studio immediately into an admin site.
auto generation of field values, e.g. slug field that is the slugified value of another field, or sum field that is the total of a field and b field.

Having this sort of meta data live with the model themselves makes sense because this type of metadata actually help describe the model constraints.
P.S. This isn't a new idea on my part. It's mostly influenced by the Django ORM. I'd love to see this available as it would immensely speed up the data model iteration part of app development.

+1

+1


Usability Timeformats

The Observation constructor takes a "start_date_and_time" datetime.
A string can not be given, instead the datetime.fromisoformat("...") function needs to be called. Both should be possible.

Should be a simple fix, no?


Generating Correct Number of Frames in 3:2 Pull Down

Hi
Referring to issue https://github.com/Intel-Media-SDK/MediaSDK/issues/1801, I can confirm that by setting ExtendedPicStruct to 1, proper picture structure can be retrieved, via MFXVideoDECODE_GetVideoParam call. However, I can't figure out how to force the decoder to generate the correct number of frames at the end of the cadence, e.g., for TBT,BT,BTB input to generate TB,TB,TB,TB.
Any ideas?
Cheers,

Hi @Nastooh-Inca ,
Sorry for delayed reply.
Is my understanding correct that you're asking about Inverse Telecine:  29,97 telecined to 24p ?
If so, our decoders don't do this. It's out of their scope. This functionality was available in PTIR (Premium Telecine Interlace Reverser). PTIR was distributed as a part of MediaServerStudio Professional. Let me take a pause to clarify our strategy regarding PTIR.

@dmitryermilov better to ask VPP onwers

@FurongZhang , can you please check and reply

Sure.

Hi Nastooh,
Sorry for delayed reply. Please try it via MSDK VPP. To check the idea, you can use sample_vpp this way:
sample_vpp -i in.nv12 -o out.nv12 -lib hw -sh 144 -sw 176 -dh 144 -dw 176 -sf 30 -df 24 -spic 0 -dpic 1 -iopattern d3d_to_d3d -pts_check -di_mode 1
It enables framerate conversion with deinterlacing. FRC is 30i -> 24p.
If it works for you, you can integrate VPP with corresponding setting in your pipeline.
Rgeards,
Dmitry

Hi Dmitry,
Thank you for your reply.Our use case is somewhat opposite, i.e., we need to go from 4 frames to 5. I suppose then the flow would be:
1-  Use MFXVideoDECODE_Query to detect MFX_PICSTRUCT_FIELD_REPEATED
2- Signal VPP to create  the new frame
Regarding Step 2, does MSDK have any capacities to weave top field of a frame with bottom field of the next one?
Cheers,

Hi Nastooh,
Ahh. Sorry for misleading then.

Regarding Step 2, does MSDK have any capacities to weave top field of a frame with bottom field of the next one?
Yes. But in a "manual" mode:  please see https://github.com/Intel-Media-SDK/MediaSDK/blob/master/doc/mediasdk-man.md#vppfieldprocessingmode

It means you can trigger two copy operations (per each field) but it would require two RunVPPFrameAsync calls.
Also, feel free to refer to implementations of this field copy:
https://github.com/Intel-Media-SDK/MediaSDK/blob/ba78c36cb0a0e52ececc8ab6bea2000f8fa09767/_studio/mfx_lib/vpp/src/mfx_vpp_hw.cpp#L3987 (this is an actual call to GPU copy kernel) and code related to imfxFPMode (it defines copy mode.)
Regards,
Dmitry

Seems it was addressed. Feel free to reopen if not.