SQLite3 over HTTPS is failing, Rails server crashing on MacBook M2 chip. Error: "Missing Symbol Called"

Issue Summary:
When attempting to use SQLite3 over HTTPS on a MacBook M2 chip, I'm encountering an error resulting in crashing of the Rails server.
Error Message:
Getting this error message without any trace:
"dyld[XXXXX]: missing symbol called"
Reproduction Steps:
Set up a Rails project on a MacBook M2 chip.
Configure the project to use SQLite3 over HTTPS using thin.
Start the Rails server.
Open any valid URL in the browser.
Observe the error and subsequent crash.
Expected Behavior:
The SQLite3 over HTTPS configuration should work seamlessly on MacBook M2 chips without encountering missing symbol errors or causing the Rails server to crash.
Environment:
Operating System: macOS (14.1.1 (23B81))
Ruby Version: (3.0.2)
Rails Version: (7.0.8)
SQLite3 Version: (1.6.8-arm64-darwin)
MacBook M2 Chip
Possible Solutions:
I can run the application with the same setup and configuration, but using Postgres instead of Sqlite3.

Hi! Thanks for reporting this. I'll try to help.
Can you give me more information about "Configure the project to use SQLite3 over HTTPS using thin"? What are you doing, specifically? I'll need to reproduce what you're seeing in order to diagnose the problem.

In the meantime, a likely workaround would be to avoid using a precompiled native platform gem -- see https://github.com/sparklemotion/sqlite3-ruby/blob/master/INSTALLATION.md#avoiding-the-precompiled-native-gem for help on how to do that.

Hi! Thanks for reporting this. I'll try to help.
Can you give me more information about "Configure the project to use SQLite3 over HTTPS using thin"? What are you doing, specifically? I'll need to reproduce what you're seeing in order to diagnose the problem.

Thank you for your response.
I'm generating an SSL certificate for development using the below command:
openssl req -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -out tmp/dev-ssl.crt -keyout tmp/dev-ssl.key
Using this certificate to run the rails server over HTTPS using the below command:
thin --ssl --ssl-disable-verify --ssl-key-file tmp/dev-ssl.key --ssl-cert-file tmp/dev-ssl.crt --debug start
[Please also see the attached screenshot for error]

This setup is working perfectly fine when I use Postgres instead of Sqlite3.
Thank you for suggesting the workaround, I will try using that.

In the meantime, a possible workaround is: avoid using a precompiled native platform gem -- see https://github.com/sparklemotion/sqlite3-ruby/blob/master/INSTALLATION.md#avoiding-the-precompiled-native-gem for help on how to do that.

This workaround did not work. Getting the same error dyld[XXXXX]: missing symbol called and server crashes.

@umairmehar Can you please help me reproduce this? What is in your config.ru, etc.? I'll need to reproduce what you're seeing in order to diagnose the problem.

OK, I've tried and cannot reproduce what you're seeing. Let me tell you what I'm doing, and you need to tell me what's different about your setup.

output from `bundle env`
Environment
Bundler       2.4.22
  Platforms   ruby, arm64-darwin-23
Ruby          3.0.2p107 (2021-07-07 revision 0db68f023372b634603c74fca94588b457be084c) [arm64-darwin-23]
  Full Path   /Users/flavorjones/.rubies/ruby-3.0.2/bin/ruby
  Config Dir  /Users/flavorjones/.rubies/ruby-3.0.2/etc
RubyGems      3.2.22
  Gem Home    /Users/flavorjones/.gem/ruby/3.0.2
  Gem Path    /Users/flavorjones/.gem/ruby/3.0.2:/Users/flavorjones/.rubies/ruby-3.0.2/lib/ruby/gems/3.0.0
  User Home   /Users/flavorjones
  User Path   /Users/flavorjones/.gem/ruby/3.0.0
  Bin Dir     /Users/flavorjones/.gem/ruby/3.0.2/bin
Tools
  Git         2.39.2
  RVM         not installed
  rbenv       not installed
  chruby      not installed

Bundler Build Metadata
Built At          2023-11-09
Git SHA           ec2089640
Released Version  true

Bundler settings
pkgs.shopify.io
  Set for the current user (/Users/flavorjones/.bundle/config): "token:[REDACTED]"
set
  Set for the current user (/Users/flavorjones/.bundle/config): "jobs 2"

Gemfile
Gemfile
source "https://rubygems.org"

# Bundle edge Rails instead: gem "rails", github: "rails/rails", branch: "main"
gem "rails", "~> 7.1.2"

# The original asset pipeline for Rails [https://github.com/rails/sprockets-rails]
gem "sprockets-rails"

# Use sqlite3 as the database for Active Record
gem "sqlite3", "~> 1.4"

# Use the Puma web server [https://github.com/puma/puma]
gem "puma", ">= 5.0"

# Use JavaScript with ESM import maps [https://github.com/rails/importmap-rails]
gem "importmap-rails"

# Hotwire's SPA-like page accelerator [https://turbo.hotwired.dev]
gem "turbo-rails"

# Hotwire's modest JavaScript framework [https://stimulus.hotwired.dev]
gem "stimulus-rails"

# Build JSON APIs with ease [https://github.com/rails/jbuilder]
gem "jbuilder"

# Use Redis adapter to run Action Cable in production
gem "redis", ">= 4.0.1"

# Use Kredis to get higher-level data types in Redis [https://github.com/rails/kredis]
# gem "kredis"

# Use Active Model has_secure_password [https://guides.rubyonrails.org/active_model_basics.html#securepassword]
# gem "bcrypt", "~> 3.1.7"

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem "tzinfo-data", platforms: %i[ windows jruby ]

# Reduces boot times through caching; required in config/boot.rb
gem "bootsnap", require: false

# Use Active Storage variants [https://guides.rubyonrails.org/active_storage_overview.html#transforming-images]
# gem "image_processing", "~> 1.2"

group :development, :test do
  # See https://guides.rubyonrails.org/debugging_rails_applications.html#debugging-with-the-debug-gem
  gem "debug", platforms: %i[ mri windows ]
end

group :development do
  # Use console on exceptions pages [https://github.com/rails/web-console]
  gem "web-console"

  # Add speed badges [https://github.com/MiniProfiler/rack-mini-profiler]
  # gem "rack-mini-profiler"

  # Speed up commands on slow machines / big apps [https://github.com/rails/spring]
  # gem "spring"
end

group :test do
  # Use system testing [https://guides.rubyonrails.org/testing.html#system-testing]
  gem "capybara"
  gem "selenium-webdriver"
end

gem "thin", "1.8.2"

Gemfile.lock
GEM
  remote: https://rubygems.org/
  specs:
    actioncable (7.1.2)
      actionpack (= 7.1.2)
      activesupport (= 7.1.2)
      nio4r (~> 2.0)
      websocket-driver (>= 0.6.1)
      zeitwerk (~> 2.6)
    actionmailbox (7.1.2)
      actionpack (= 7.1.2)
      activejob (= 7.1.2)
      activerecord (= 7.1.2)
      activestorage (= 7.1.2)
      activesupport (= 7.1.2)
      mail (>= 2.7.1)
      net-imap
      net-pop
      net-smtp
    actionmailer (7.1.2)
      actionpack (= 7.1.2)
      actionview (= 7.1.2)
      activejob (= 7.1.2)
      activesupport (= 7.1.2)
      mail (~> 2.5, >= 2.5.4)
      net-imap
      net-pop
      net-smtp
      rails-dom-testing (~> 2.2)
    actionpack (7.1.2)
      actionview (= 7.1.2)
      activesupport (= 7.1.2)
      nokogiri (>= 1.8.5)
      racc
      rack (>= 2.2.4)
      rack-session (>= 1.0.1)
      rack-test (>= 0.6.3)
      rails-dom-testing (~> 2.2)
      rails-html-sanitizer (~> 1.6)
    actiontext (7.1.2)
      actionpack (= 7.1.2)
      activerecord (= 7.1.2)
      activestorage (= 7.1.2)
      activesupport (= 7.1.2)
      globalid (>= 0.6.0)
      nokogiri (>= 1.8.5)
    actionview (7.1.2)
      activesupport (= 7.1.2)
      builder (~> 3.1)
      erubi (~> 1.11)
      rails-dom-testing (~> 2.2)
      rails-html-sanitizer (~> 1.6)
    activejob (7.1.2)
      activesupport (= 7.1.2)
      globalid (>= 0.3.6)
    activemodel (7.1.2)
      activesupport (= 7.1.2)
    activerecord (7.1.2)
      activemodel (= 7.1.2)
      activesupport (= 7.1.2)
      timeout (>= 0.4.0)
    activestorage (7.1.2)
      actionpack (= 7.1.2)
      activejob (= 7.1.2)
      activerecord (= 7.1.2)
      activesupport (= 7.1.2)
      marcel (~> 1.0)
    activesupport (7.1.2)
      base64
      bigdecimal
      concurrent-ruby (~> 1.0, >= 1.0.2)
      connection_pool (>= 2.2.5)
      drb
      i18n (>= 1.6, < 2)
      minitest (>= 5.1)
      mutex_m
      tzinfo (~> 2.0)
    addressable (2.8.5)
      public_suffix (>= 2.0.2, < 6.0)
    base64 (0.2.0)
    bigdecimal (3.1.4)
    bindex (0.8.1)
    bootsnap (1.17.0)
      msgpack (~> 1.2)
    builder (3.2.4)
    capybara (3.39.2)
      addressable
      matrix
      mini_mime (>= 0.1.3)
      nokogiri (~> 1.8)
      rack (>= 1.6.0)
      rack-test (>= 0.6.3)
      regexp_parser (>= 1.5, < 3.0)
      xpath (~> 3.2)
    concurrent-ruby (1.2.2)
    connection_pool (2.4.1)
    crass (1.0.6)
    daemons (1.4.1)
    date (3.3.4)
    debug (1.8.0)
      irb (>= 1.5.0)
      reline (>= 0.3.1)
    drb (2.2.0)
      ruby2_keywords
    erubi (1.12.0)
    eventmachine (1.2.7)
    globalid (1.2.1)
      activesupport (>= 6.1)
    i18n (1.14.1)
      concurrent-ruby (~> 1.0)
    importmap-rails (1.2.3)
      actionpack (>= 6.0.0)
      activesupport (>= 6.0.0)
      railties (>= 6.0.0)
    io-console (0.6.0)
    irb (1.9.1)
      rdoc
      reline (>= 0.3.8)
    jbuilder (2.11.5)
      actionview (>= 5.0.0)
      activesupport (>= 5.0.0)
    loofah (2.22.0)
      crass (~> 1.0.2)
      nokogiri (>= 1.12.0)
    mail (2.8.1)
      mini_mime (>= 0.1.1)
      net-imap
      net-pop
      net-smtp
    marcel (1.0.2)
    matrix (0.4.2)
    mini_mime (1.1.5)
    minitest (5.20.0)
    msgpack (1.7.2)
    mutex_m (0.2.0)
    net-imap (0.4.7)
      date
      net-protocol
    net-pop (0.1.2)
      net-protocol
    net-protocol (0.2.2)
      timeout
    net-smtp (0.4.0)
      net-protocol
    nio4r (2.6.1)
    nokogiri (1.15.5-arm64-darwin)
      racc (~> 1.4)
    psych (<IP_ADDRESS>)
      stringio
    public_suffix (5.0.4)
    puma (6.4.0)
      nio4r (~> 2.0)
    racc (1.7.3)
    rack (2.2.8)
    rack-session (1.0.1)
      rack (< 3)
    rack-test (2.1.0)
      rack (>= 1.3)
    rackup (1.0.0)
      rack (< 3)
      webrick
    rails (7.1.2)
      actioncable (= 7.1.2)
      actionmailbox (= 7.1.2)
      actionmailer (= 7.1.2)
      actionpack (= 7.1.2)
      actiontext (= 7.1.2)
      actionview (= 7.1.2)
      activejob (= 7.1.2)
      activemodel (= 7.1.2)
      activerecord (= 7.1.2)
      activestorage (= 7.1.2)
      activesupport (= 7.1.2)
      bundler (>= 1.15.0)
      railties (= 7.1.2)
    rails-dom-testing (2.2.0)
      activesupport (>= 5.0.0)
      minitest
      nokogiri (>= 1.6)
    rails-html-sanitizer (1.6.0)
      loofah (~> 2.21)
      nokogiri (~> 1.14)
    railties (7.1.2)
      actionpack (= 7.1.2)
      activesupport (= 7.1.2)
      irb
      rackup (>= 1.0.0)
      rake (>= 12.2)
      thor (~> 1.0, >= 1.2.2)
      zeitwerk (~> 2.6)
    rake (13.1.0)
    rdoc (6.6.0)
      psych (>= 4.0.0)
    redis (5.0.8)
      redis-client (>= 0.17.0)
    redis-client (0.18.0)
      connection_pool
    regexp_parser (2.8.2)
    reline (0.4.1)
      io-console (~> 0.5)
    rexml (3.2.6)
    ruby2_keywords (0.0.5)
    rubyzip (2.3.2)
    selenium-webdriver (4.15.0)
      rexml (~> 3.2, >= 3.2.5)
      rubyzip (>= 1.2.2, < 3.0)
      websocket (~> 1.0)
    sprockets (4.2.1)
      concurrent-ruby (~> 1.0)
      rack (>= 2.2.4, < 4)
    sprockets-rails (3.4.2)
      actionpack (>= 5.2)
      activesupport (>= 5.2)
      sprockets (>= 3.0.0)
    sqlite3 (1.6.9-arm64-darwin)
    stimulus-rails (1.3.0)
      railties (>= 6.0.0)
    stringio (3.1.0)
    thin (1.8.2)
      daemons (~> 1.0, >= 1.0.9)
      eventmachine (~> 1.0, >= 1.0.4)
      rack (>= 1, < 3)
    thor (1.3.0)
    timeout (0.4.1)
    turbo-rails (1.5.0)
      actionpack (>= 6.0.0)
      activejob (>= 6.0.0)
      railties (>= 6.0.0)
    tzinfo (2.0.6)
      concurrent-ruby (~> 1.0)
    web-console (4.2.1)
      actionview (>= 6.0.0)
      activemodel (>= 6.0.0)
      bindex (>= 0.4.0)
      railties (>= 6.0.0)
    webrick (1.8.1)
    websocket (1.2.10)
    websocket-driver (0.7.6)
      websocket-extensions (>= 0.1.0)
    websocket-extensions (0.1.5)
    xpath (3.2.0)
      nokogiri (~> 1.8)
    zeitwerk (2.6.12)

PLATFORMS
  arm64-darwin-23

DEPENDENCIES
  bootsnap
  capybara
  debug
  importmap-rails
  jbuilder
  puma (>= 5.0)
  rails (~> 7.1.2)
  redis (>= 4.0.1)
  selenium-webdriver
  sprockets-rails
  sqlite3 (~> 1.4)
  stimulus-rails
  thin (= 1.8.2)
  turbo-rails
  tzinfo-data
  web-console

BUNDLED WITH
   2.4.22

how I setup my rails app
$ rails new foo
...
$ bin/rails generate scaffold post title:string body:text published:boolean
      invoke  active_record
      remove    db/migrate/20231130175226_create_posts.rb
      create    db/migrate/20231130175415_create_posts.rb
   identical    app/models/post.rb
      invoke    test_unit
   identical      test/models/post_test.rb
       force      test/fixtures/posts.yml
      invoke  resource_route
       route    resources :posts
      invoke  scaffold_controller
      create    app/controllers/posts_controller.rb
      invoke    erb
      create      app/views/posts
      create      app/views/posts/index.html.erb
      create      app/views/posts/edit.html.erb
      create      app/views/posts/show.html.erb
      create      app/views/posts/new.html.erb
      create      app/views/posts/_form.html.erb
      create      app/views/posts/_post.html.erb
      invoke    resource_route
      invoke    test_unit
      create      test/controllers/posts_controller_test.rb
      create      test/system/posts_test.rb
      invoke    helper
      create      app/helpers/posts_helper.rb
      invoke      test_unit
      invoke    jbuilder
      create      app/views/posts/index.json.jbuilder
      create      app/views/posts/show.json.jbuilder
      create      app/views/posts/_post.json.jbuilder

$ bin/rails db:migrate
==<PHONE_NUMBER>5415 CreatePosts: migrating ======================================
-- create_table(:posts)
   -> 0.0031s
==<PHONE_NUMBER>5415 CreatePosts: migrated (0.0032s) =============================

output from running the server
$ bundle exec thin --ssl --ssl-disable-verify --ssl-key-file dev-ssl.key --ssl-cert-file dev-ssl.crt --debug start
2023-11-30 13:01:05 -0500 Using rack adapter
2023-11-30 13:01:06 -0500 Thin web server (v1.8.2 codename Ruby Razor)
2023-11-30 13:01:06 -0500 Debugging ON
2023-11-30 13:01:06 -0500 Maximum connections set to 1024
2023-11-30 13:01:06 -0500 Listening on <IP_ADDRESS>:3000, CTRL+C to stop
8146902720:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1556:SSL alert number 46
8146902720:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1556:SSL alert number 46
8146902720:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1556:SSL alert number 46
8146902720:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1556:SSL alert number 46
8146902720:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1556:SSL alert number 46
8146902720:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1556:SSL alert number 46
8146902720:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1556:SSL alert number 46
8146902720:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1556:SSL alert number 46

And the app responds fine. I'm able to read/write sqlite3 database records fine.

Closing. If you can provide steps for me to reproduce this, please do comment and I'm happy to investigate.


Shadow Not Display in Android

Description
Hello Team,
I am currently working on react-native and I'm trying to display shadow to my view as shown below in Android. Same thing works well in IOS.
style={{
backgroundColor: "#FFF",
padding: 10,
elevation: 2,
shadowColor: "#000",
shadowOffset: {
width: 0,
height: 2,
},
shadowOpacity: 0.25,
shadowRadius: 3.84,
}}
But unfortunately, I am not seeing any change in my UI. The following are the dependencies used:
"dependencies": {
"@react-native-community/async-storage": "^1.12.1",
"@react-native-community/datetimepicker": "^6.2.0",
"@react-native-community/push-notification-ios": "^1.10.1",
"@react-native-firebase/app": "^15.2.0",
"@react-native-firebase/messaging": "^15.2.0",
"@react-navigation/native": "^6.0.11",
"@react-navigation/stack": "^6.2.2",
"bugsnag-react-native": "^2.23.10",
"i18n-js": "^3.9.2",
"lodash": "^4.17.21",
"moment": "^2.29.4",
"prop-types": "^15.8.1",
"react": "17.0.2",
"react-native": "0.68.2",
"react-native-animatable": "^1.3.3",
"react-native-background-timer": "^2.4.1",
"react-native-config": "^1.4.6",
"react-native-country-picker-modal": "^2.0.0",
"react-native-datepicker": "^1.7.2",
"react-native-document-picker": "^8.1.1",
"react-native-gesture-handler": "^2.5.0",
"react-native-image-crop-picker": "^0.38.0",
"react-native-iphone-x-helper": "^1.3.1",
"react-native-orientation-locker": "^1.5.0",
"react-native-push-notification": "^8.1.1",
"react-native-responsive-fontsize": "^0.5.1",
"react-native-safe-area-context": "^3.4.1",
"react-native-screens": "^3.15.0",
"react-native-simple-toast": "^1.1.4",
"react-native-splash-screen": "^3.3.0",
"react-native-swipe-gestures": "^1.0.5",
"react-native-vector-icons": "^9.2.0",
"react-navigation-shared-element": "^3.1.3",
"react-redux": "^8.0.2",
"redux": "^4.2.0",
"redux-persist": "^6.0.0",
"redux-thunk": "^2.4.1",
"sendbird": "^3.1.23"
},
It's been a week now trying to resolve this issue, but I'm unable to get any solution. Could you please help me out here?
Thanks in advance
Version
0.68.2
Output of npx react-native info
System:
OS: macOS 11.6.8
CPU: (8) x64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
Memory: 30.79 MB / 16.00 GB
Shell: 5.8 - /bin/zsh
Binaries:
Node: 16.15.0 - /usr/local/bin/node
Yarn: 1.22.18 - /usr/local/bin/yarn
npm: 8.5.5 - /usr/local/bin/npm
Watchman: 2<IP_ADDRESS> - /usr/local/bin/watchman
Managers:
CocoaPods: 1.11.3 - /usr/local/bin/pod
SDKs:
iOS SDK:
Platforms: DriverKit 21.2, iOS 15.2, macOS 12.1, tvOS 15.2, watchOS 8.3
Android SDK: Not Found
IDEs:
Android Studio: 2021.2 AI-212.5712.43.2112.8609683
Xcode: 13.2.1/13C100 - /usr/bin/xcodebuild
Languages:
Java: javac 15 - /usr/bin/javac
npmPackages:
@react-native-community/cli: Not Found
react: 17.0.2 => 17.0.2
react-native: 0.68.2 => 0.68.2
react-native-macos: Not Found
npmGlobalPackages:
react-native: Not Found
Steps to reproduce
<View style={{
backgroundColor: "#FFF",
padding: 10,
elevation: 2,
shadowColor: "#000",
shadowOffset: {
width: 0,
height: 2,
},
shadowOpacity: 0.25,
shadowRadius: 3.84,}}>
Hi

Snack, code example, screenshot, or link to a repository

Hi @dippathak26, Can you please share your exact code snippet? I tried it with same configuration and it is working fine. Code snippet -
      <View
        style={{
          backgroundColor: '#FFF',
          padding: 10,
          elevation: 2,
          shadowColor: '#000',
          shadowOffset: {
            width: 0,
            height: 2,
          },
          shadowOpacity: 0.25,
          shadowRadius: 3.84,
        }}>
        <Text>Hi</Text>
      </View>
      <View
        style={{
          backgroundColor: '#FFF',
          padding: 10,
          elevation: 2,
          shadowColor: '#000',
          shadowOffset: {
            width: 0,
            height: 2,
          },
          shadowOpacity: 0.25,
          shadowRadius: 3.84,
        }}>
        <Text>Hi</Text>
      </View>
      <View
        style={{
          backgroundColor: '#FFF',
          padding: 10,
          elevation: 2,
          shadowColor: '#000',
          shadowOffset: {
            width: 0,
            height: 2,
          },
          shadowOpacity: 0.25,
          shadowRadius: 3.84,
        }}>
        <Text>Hi</Text>
      </View>

Add some margin in order to make it clear, just like this -
       <View
        style={{
          backgroundColor: '#FFF',
          padding: 10,
          elevation: 2,
          shadowColor: '#000',
          shadowOffset: {
            width: 0,
            height: 2,
          },
          shadowOpacity: 0.25,
          shadowRadius: 3.84,
          margin:10
        }}>
        <Text>Hi</Text>
      </View>


update lib core and remove package lock

Description

cli plugins should not have a package-lock see https://github.com/sindresorhus/ama/issues/479#issuecomment-310661514
update to latest config lib which does not include the dotenv patch-package anymore: see https://github.com/adobe/aio-lib-core-config/issues/27#issuecomment-635749371

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have signed the Adobe Open Source CLA.
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

@moritzraho I'm not sure I get the first bullet point. Is there a reference to this decision in an @adobe repository? Seems weird to have a pointer to someone's personal repo if this is a policy for aio-cli plugins. Please also create issues for these two items (which really should be separate PRs IMO).

Hi @justinedelson

no package-lock: the idea is that modules that are dependencies (libraries, plugins,..) should not have a package-lock.json. We took the decision (internally) to follow this principle for all aio plugins and libraries. The package-lock.json file is never included in published package, and while some locked dependencies might work for the local setup, there might have been a patch or minor update in the dependency tree that breaks your plugin for dependent modules (aio-cli). And because there is a package-lock in the dev local setup, the breaking update in the dependency tree will only be noticed at CI level. More details in this article: https://www.twilio.com/blog/lockfiles-nodejs. Hope this helps 🙂

Ok for doing two PRs, as those are easy changes we can discuss directly on the PR 🙂


chore: remove LevelDecoder

Which issue does this PR close?

Closes #2379
Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Hi,
In my use of parquet-rs, I have a flow that reads and computes over the raw dictionary IDs in data pages, to get to those I'm using LevelDecoder to read repetition and definition levels, this change broke my code.
I'd like to put LevelDecoder back in WDYT?

LevelDecoder to read repetition and definition levels

What do you think of:

Use RleDecoder's directly
Maintain LevelDecoder within your own project

I would rather avoid maintaining an experimental API that isn't used by any of the first-class APIs

I am using RleDecoder of course to decode the RLE into dictionary IDs.
I am using LevelDecoder for 2 things:

Get the definition levels (so I know which positions are null or defined)
Find the position within the ByteBufferPtr where the RLE encoded values start to feed into the RLE decoder in V1 pages since those pages don't have the size of the definition/repetition levels in the page header

The page header contains the encoding, either Encoding::RLE or Encoding::BIT_PACKED, based on this you can select either to use RleDecoder or BitReader, which should do what you require. This is why LevelDecoder is no longer used, it is redundant.


docker images with missing libs do not print errors

Currently if the system is missing dependencies the CLI Tool just swallows errors instead of printing them out.

I'm seeing this. cypress run just prints nothing. What are the dependencies needed?

This is more thoroughly documented here. https://github.com/cypress-io/cypress/issues/317
Also our docs have all of the dependencies listed: https://docs.cypress.io/docs/continuous-integration


How do you use telemetry_server correctly?

I have the following code that reads a CSV over the network with telemetry I want to feed to PX4:
import asyncio
import io
import csv
import numpy as np
from functools import partial
from typing import Dict, Any
from decimal import Decimal
from mavsdk import System
from mavsdk.telemetry_server import (
    Position, VelocityNed, Heading, Imu,
    AccelerationFrd, AngularVelocityFrd, MagneticFieldFrd
)

G_CONSTANT = 9.80665
ACCEL_DATA_G = []
TIME_INTERVAL = 1 / 30 # We're sampling at 30Hz

async def send_telemetry(drone: System, data: Dict[Any,Any]):
    global ACCEL_DATA_G

    ACCEL_DATA_G.append(
        [
            float(data['accelerometerAccelerationX(G)']),
            float(data['accelerometerAccelerationY(G)']),
            float(data['accelerometerAccelerationZ(G)'])
        ]
    )

    accel_data_g = np.array(ACCEL_DATA_G)
    accel_data_m_s2 = accel_data_g * G_CONSTANT
    velocity = np.zeros_like(accel_data_m_s2)

    for i in range(1, len(accel_data_m_s2)):
        velocity[i] = velocity[i-1] + (accel_data_m_s2[i-1] + accel_data_m_s2[i]) / 2 * TIME_INTERVAL

    average_velocity = np.mean(velocity, axis=0)
    #print(f"Velocity in m/s (NED): {average_velocity}")

    await drone.telemetry_server.publish_position(
        position=Position(
            latitude_deg=float(data['locationLatitude(WGS84)']),
            longitude_deg=float(data['locationLongitude(WGS84)']),
            absolute_altitude_m=float(data['locationAltitude(m)']),
            relative_altitude_m=float(data['altimeterRelativeAltitude(m)'])
        ),
        velocity_ned=VelocityNed(
            north_m_s=average_velocity[0],
            east_m_s=average_velocity[1],
            down_m_s=average_velocity[2]
        ),
        heading=Heading(
            heading_deg=float(data['locationTrueHeading(°)'])
        )
    )

    """
    #drone.telemetry_server.publish_imu()
    await drone.telemetry_server.publish_raw_imu(
        Imu(
            acceleration_frd=AccelerationFrd(
                forward_m_s2=float(data['accelerometerAccelerationX(G)']),
                right_m_s2=float(data['accelerometerAccelerationY(G)']),
                down_m_s2=float(data['accelerometerAccelerationZ(G)'])
            ),
            angular_velocity_frd=AngularVelocityFrd(
                forward_rad_s=float(data['gyroRotationX(rad/s)']),
                right_rad_s=float(data['gyroRotationY(rad/s)']),
                down_rad_s=float(data['gyroRotationZ(rad/s)'])
            ),
            magnetic_field_frd=MagneticFieldFrd(
                forward_gauss=float(Decimal(data['motionMagneticFieldX(µT)']) / Decimal(100)), # 1 Gauss is equal to 100 µT
                right_gauss=float(Decimal(data['motionMagneticFieldY(µT)']) / Decimal(100)),
                down_gauss=float(Decimal(data['motionMagneticFieldZ(µT)']) / Decimal(100))
            ),
            temperature_degc=0,
            timestamp_us=int(float(data['locationTimestamp_since1970(s)'])) * 1_000_000
        )
    )
    """

async def handle_echo(reader, writer, drone: System):
    addr = writer.get_extra_info('peername')
    print(f"Received connection from {addr}")

    fields = None

    with open('telem.csv', 'w') as f:
        while True:
            data = await reader.readline()
            message = data.decode()
            #addr = writer.get_extra_info('peername')
            #print(f"Received connection from {addr}")
            #print(message)
            f.write(message)

            if 'loggingTime' in message:
                fields = message.strip().split(',')
                print(message)
                continue

            buf = io.StringIO(message)
            csv_reader = csv.DictReader(buf, fieldnames=fields)

            for row in csv_reader:
                #asyncio.create_task(send_telemetry(drone, row))
                await send_telemetry(drone, row)

async def read_telem(drone: System):
    async for position in drone.telemetry.position():
        print(position)
        await asyncio.sleep(5)

async def main():
    drone = System()
    await drone.connect(system_address="udp://:14540") # PX4
    #await drone.connect(system_address="udp://:14550") # Ardupilot

    #asyncio.create_task(read_telem(drone))

    server = await asyncio.start_server(
        partial(handle_echo, drone=drone), '<IP_ADDRESS>', 8888)

    addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets)
    print(f'Serving on {addrs}')

    async with server:
        await server.serve_forever()

asyncio.run(main())

I'm starting PX4 in SITL mode with the following command:
HEADLESS=1 make px4_sitl gz_x500

At first glance this seems to work as telemetry_server.publish_position() doesn't return an error (telemetry_server.publish_imu() seems to be broken but that's another discussion).
However, it seems like PX4 is not actually reading the data I send it which can be confirmed by opening up a GCS or querying it's position via MAVLink.
Am i fundementally misundestanding what telemetry_server is supposed to do? or is this a bug and/or user error?
Thanks

Wrong repo. My bad.


Add save option to authorize

For programmatic deployments of The Things Network, we need to be able to generate tokens and to get them. Currently, when tokens are generated, they are only shown on the command output - which means getting the token requires parsing the output. It's not sustainable, since the CLI output might change over time.
This PR introduces a new option to the authorize commands, --save, which allows the user to point to a file where to save the token. This way, when programmatically deploying The Things Network, it is possible to retrieve the token safely. If this is not satisfying, we could imagine an option to hide the logs from the output.

Coverage remained the same at 74.731% when pulling ea271e3d1d39817aee26ffc9379a719a34ea1f46 on feature/save-token into 775f58d0bc86328750f614086b8b1153bfa3f51e on develop.

Coverage remained the same at 74.731% when pulling 9547bff660c7b49e33f343a04ee791c7f3fb2c11 on feature/save-token into f3610c5d31b255e18dffa3654363262478ba9a14 on develop.

Isn't it easier to just remove the unneeded log lines and just print the token directly? These cmds are for expert users anyway.

PR closed because the needs are already fulfilled by --no-cli-logs.

Coverage increased (+0.09%) to 74.824% when pulling 1257d6d6872647a8689e1a9d8b37f64d45061ea5 on feature/save-token into f3610c5d31b255e18dffa3654363262478ba9a14 on develop.


OOM will occur when large amounts of data are collected

🐞 Bug Report
Affected Services [REQUIRED]

 The issue is located in:
When collecting a large amount of data, even if the Retention policy is configured in core-data, the database service that will edgex after a period of time will still experience OOM.
vms:
cpu: 4
/deviceprofile
Description and Minimal Reproduction [REQUIRED]

Start a opcua server simulator. You can write an opcua server-side program using python.
Build the docker compose file
Using environment variables to override the core-data configuration

RETENTION_ENABLED: true
RETENTION_INTERVAL: 10s
RETENTION_MAXCAP: 10
RETENTION_MINCAP: 5

start docker compose services.
Create deviceprofile and device via api
Enable logging of core-data service as DEBUG in consul and observation service resource utilization

🔥 Exception or Error

Increasing number of keys in redis, OOM in VMs.
debug logs:   "Prepare to delete 0 readings"

🌍  Your Environment
Deployment Environment:
ubuntu22.04-wsl2
EdgeX Version [REQUIRED]:
3.2.0
Anything else relevant?

Please try to set WRITABLE_PERSISTDATA: 'false' in core-data to confirm whether it is the device-opc-ua issue


Add filtering for consumer groups and make results case insensitive

Amazing change! @suyash-naithani thank you so much! I will rebase your PR right now.
Have you verified this change with kafka-ui?


Global Registry Notary cpu and memory reservation doesn't work

rancher/rancher:master
Steps:

Change the cpu reservation of Notary to 450m
Enable Global Registry
Check the reservation on Notary workload

Results:
It's not 450m which I configured

Version: Master (v2.3) (6/7/19)
This was actually already validated in the UI Test Plan recently. I carefully confirmed that for each and every field in the Global Registry config (all cpu/memory reservation/limit options) are working correctly.
This test was done via:

Set default settings, Enable Global Registry, ensure these all take effect in all areas (all workloads/containers).
Disable, and now set unique settings, Enable Global Registry, ensure these all take effect in all areas (all workloads/containers).
EDIT, and set new unique settings, Save, ensure these all take effect in all areas (all workloads/containers).

Notary settings for cpu limit/reservation now work.


IDS Alert - Search on Destination IP

Please add the destination field as a searchable item in the IDS Alert Search.

https://github.com/opnsense/core/pull/2351


feat(shipyard-controller): Introduce API Endpoint for retrieving Sequence Executions

Closes #6823
This PR adds a new API endpoint for retrieving items from the SequenceExecution collection that the shipyard controller uses to orchestrate sequences. The endpoint is reachable via:
/api/controlPlane/v1/sequence-execution/{projectName}

reverting to draft - as agreed, the project name will be a mandatory query parameter instead of a path parameter in the API


Automate the Image Mirroring Process for KDM Releases

With the implementation for https://github.com/rancher/rancher/issues/41416,
the  regsync.yaml file is generated and updated in the rancher/charts repo automatically, and the file can be used to perform the image mirroring by using the regsync tool manually.
We want to automate the process further:
Integrate the invocation of resgsync tool into the Drone CI so that newly-added images and tags are mirrored automatically.

The automation has been validated in the latest OOB KDM (dev-v2.7-2023-06-patches) release process.
Here is the link to the build , where images are mirrored automatically.


Error handling

Hi!
I was asked by Niklas Leide to ask this question here on GitHub.
I'm wondering in what way the reference implementation is able to detect errors. More specifically:

Are the endpoints /log and /statistics implemented?
If so, what happens with the information sent there? For instance:

Are log entries and statistics stored in the database?
Is there a way to be notified when an error or warning arrives?
Do certain warnings or errors lead to any actions by the reference implementation (such as attempting to retry something, sending a new notification to the webhook or similar)

Are the numbers in the StatisticsEntry object verified against what would be expected numbers?
If a notification can't be sent (e.g. the target URL for the webhook can't be reached) , what happens?
If a notification can be sent, but the consuming system doesn't act on it (fails to ask for the changes), or acts incorrectly (asks a question that doesn't retrieve the intended changes), does anything happen?

This is definitely the place for technical questions!

Yes. It's handled by the logging and statistics services.
See the source code.
2.1. It's stored in the database.
2.2 Depending on what you mean with "to be notified". The SS12000 API is being notified and the current behavior is to store it in the database. The project is open source and serves as a reference implementation. A solution could be to allow for subscriptions listening to log and statistics data types, it would be a minor change.
2.3 No, all logs and statistic entries are handled the same. The most common cause of a errors are things that won't be fixed with a retry.
No.
There's a built in retry logic. As specified in the SS12000 standard, 3 retries will be done in 5 minutes and after that there's no further attempts and the webhook will not be sent.
If the client receives a webhook and responds with a 2XX response the  webhook has been persisted in a job queue. Multiple retries will be made to honor the webhook, if it can't be processed a error log is reported back (also with multiple retries). In case of a catastrophic failure there's no logic in the reference api to handle it as it's not defined in the standard.


Strip surrounding whitespace from commit message

Hey Kevin,
thanks a lot for your work on this really useful extension.
We set it up with a template that allows for optional details at the end of the message.
"vscodeGitCommit.template": [
	"{type}: {message}, {action} {reference}",
	"",
	"{details}"
]

Now if details are kept empty, we will end up with two empty lines in the commit message. It would be ideal if it could be stripped from whitespace at the end of the template processing. What do you think?
Thanks, Jens

Hi, sorry for delay, that's a good idea, I look for your PR this week 😉

I'm sorry I don't have time to test the extension with the PR, I do it this week for sure, I can unlock time tomorrow or thursday for it 😉

Would it also be possible to trim each line (could be whitespace) and also replacing multiple 3+ new lines with just 2? For example, if have the below template:
"vscodeGitCommit.template": [
	"{type}: {message}, {action} {reference}",
	"",
	"{details}",
	"",
	"{issue_reference}"
]

If details were optional, there would be a large space between the first and last lines

Would it also be possible to trim each line (could be whitespace) and also replacing multiple 3+ new lines with just 2? For example, if have the below template:
"vscodeGitCommit.template": [
	"{type}: {message}, {action} {reference}",
	"",
	"{details}",
	"",
	"{issue_reference}"
]

If details were optional, there would be a large space between the first and last lines

@jamessampford the idea is good, but it can't be automatic, it probably create unwanted things, so I add a settings property for it


udpate 'hyprland-nvidia-git' to 'hyprland-nvidia'

hyprland-nvidia-git is still at 0.27, but hyprland-nvidia is updated to 0.29 now

its a git package so any changes on the main hyprland github repo will be reflected to it. All the packages in AUR with -git are updated automatically.
The hyprland-nvidia is a bin packages packed by maintainer


[WIP] 🌱 Bump CAPI to v1.9.3

/kind cleanup
What this PR does / why we need it:
Updates CAPI to v1.9.3.
Which issue(s) this PR fixes:
N/A
Special notes for your reviewer:
See controller-runtime v0.19 release notes: https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.19.0
See CAPI 1.8 - 1.9 migration guide: https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/book/src/developer/providers/migrations/v1.8-to-v1.9.md
CAPZ's PR, for comparison: kubernetes-sigs/cluster-api-provider-azure#5316
CAPA's PR, for comparison: kubernetes-sigs/cluster-api-provider-aws#5209

/close
See #338.


How to get the .csv file from .mat for training properly?

I have the NYUv2 Dataset in .mat format(~3GB), which include the training data naming images(shape=(1449, 3, 640, 480), depths(shape=(1449, 640, 480), labels etc. ( 1449 samples for each item). Now I want to convert it into .csv format then using CSVIter to avoid reading whole training data.
What I have did is, iteretly read in one image and one depth  sample and then use ravel() and numpy.concatenate on them to get a row vector. Finally, write this 1D vector into a .csv file using csv.writerrow().
Howerer, the origin .mat file is only ~3GB,  and the .csv file turned out to be more than 8GB. What happened to this process? More strange, if I wirte the image and depth datas into two seperate .csv files and do not flatten them, the .csv files are only ~5MB.
All datas are in float32 type.

@Roshrini  Got it. thanks


Can not read property 'string' of undefined

Can not read property 'string' of undefined.. seems like prop-type is creating issues.

Hi @veiculu ,
Yup, I need to update the lib 😅
Been super busy these past few months. I'll try to take care of it today :)

Hi @veiculu,
It should be all good in<EMAIL_ADDRESS>Don't hesitate to ask if you have other issues 😄


Spec: Add file and position delete files

One option for encoding row-level deletes is to track rows by file and row position in the file. This format should be documented in the spec.

What format do we want for the delete files? Avro?
We need to read through the filtered parquet to construct the delete files, right?

Just for information and answer questions myself. I did some POC to generate delete files in Parquet format with additional meta columns. The approach is to generate dataframe according to filters and then extend the internal row with metadata. It works fine but looks a bit strange that we spawn spark tasks in a table format, which means it has some write amplification when deleting by filters. But it should be ok if the deletion records are provided by Spark like Hudi. This reminds me that we can add an API like deleteByRowKeys(Iteratable<InternalRows>)

Just FYI, I did an early PoC based on file and position deletion files here. Since the file name should be most same so I store it as parquet file for dictionary encoding.  I will submit a PR for this soon.

I think the delete file format (Avro, Parquet, ORC) should be up to the implementation. I would use the table's default format. That is normally Parquet, but ORC users would probably want both data and delete files to be ORC. Similarly, Avro doesn't require additional modules, so if a table sets its format to Avro then it makes sense to also use Avro for deletes.
Looking forward to a read/writer PR for this. Thank you, @chenjunjiedada!

Make sense. Thanks!

Hi @rdblue , I 'd like to confirm a couple of things to continue:

For the write path, before we have spark interfaces and SortOrder, my idea is to build a filtered and sorted data frame and save it to staging table, append the added files and then drop staging table metadata.  What do you think?

    dataFrame.select( s"_file", s"_pos").sort(s"_file")
      .write
      .format("iceberg")
      .option(DEFAULT_FILE_FORMAT, format)
      .mode("append")
      .save(tablePath)

For the read path, my understanding is we don't want the deletion file filtering logic right now because that depends on the sequence number. So the read path here contains only the logic that extracts the deletion files from scan task and read them as Iterators for mergeInterator. Does this make sense to you?

@chenjunjiedada, I'm not sure I understand.
This issue tracks updates to the spec that detail how to write file/position delete files. That boils down to things like this:

The files must be a supported Iceberg format and are valid Iceberg data files, just like all Iceberg metadata files
The schema of the file/position files; probably string file path and long position in a file. Maybe that should be int instead of long?
Rows in the delete file must be sorted by position. Should we also sort by path?

We also need a reader and writer implementation for these.

I put an early PR for the spec.
It uses integer for the position, that should be enough for most cases unless we have files that larger than 8G (roughly 2^31 rows of integer). But there may exist very special cases like files with most same rows and that could be dictionary and RLE encoded.
For the sorting requirement, I prefer to sort with both path and position(I assume the records already cluster by path, so sort with path should be easy) so that the merge sort could take advantage of it.
For the reader and writer, I will create a separate PR later.

I don't understand what #885 has to do with the spec. That updates DataFile. The spec is this document that details the Iceberg at-rest format. Maybe you used the wrong branch?
For position, I think we should use a long after all. That's what DataFile uses to encode the number of rows, and I've seen Parquet files with more than 2 billion rows before.
I agree that we should sort by both path and position, but this will require support from the processing engine because the file contents could be large.

Sorry for the confusion,  I just open https://github.com/apache/incubator-iceberg/pull/887.
No matter what column it sort with, we all need a processing engine to handle it, don't we? I 'm thinking of the case of the Spark SupportsDelete interface, with the given filter we have to call the spark to build the dataframe for deletion file. But Iceberg may have some generic APIs such as deleting rows in file scope or partition scope in the future. That may not need the processing engine.

SupportsDelete won't be used to create delete files because Iceberg should not call into Spark's API as you describe. Instead, Spark should attempt the metadata-level delete and fall back to a merge-based delete.

Hi @rdblue , I'm working on the reader and writer for delete files. As for the reader, the input and output are clear as the delete files would be extracted from FileScanTask and the output would be iterable of InternalRow or GenericRecord. While for the writer, I 'm not very sure since how metadata-level delete will be implemented. For the spark engine, I could imagine two possible ways:

spark build the table scan with the filter expression and projection of the partition columns and metadata columns then save the data frame in append mode with an option like data=diff. In this case, Iceberg can generate partition level delete files from the input partition. That means we may not need a separate writer, maybe some update in the current table writer.

spark has a DeleteScan plan like the table scan, each task read the input file with filter(residual) and then write to delete file.  In this case, we need a separate writer that takes the input file and filter expression as input and writes the filtered records. This would need residual evaluation for every supported format, but we can use IcebergGenerics to handle this.

Both of them depend on spark side implementation, any thought?

The object model used by these files can be swapped out just like any Avro file. If we want to pass this information to Spark, then we would use the Spark reader classes. For most internals, we build classes that implement IndexedRecord.

@rdblue , I don't quite follow the last comment. Do you mean the object model for delete files could be Avro in-memory data model so that it could be used by other engines as well?

Whatever is consuming the file can use its own object model. When Iceberg loads metadata files, we use classes that extend IndexedRecord so that Avro can construct them directly like a specific record. An example of this is GenericDataFile.
However, there are cases where the metadata files are read by engines, like the metadata tables. When you read the entries metadata table, Iceberg creates tasks out of the manifest Avro files and Spark reads the data as InternalRow directly.
Delete files would be similar to these cases. If the engine needs to read a delete file, it would be able to use its object model with Avro. But I think that we will need an IndexedRecord implementation for file/position deletes so that we can build a filter that merges that stream with a data file record stream.

So for the reader side, we want to load delete files into memory as Avro in-memory model so that engines are able to convert them to their own data model. Basically we need

The data access mode for delete files.

class DeletedRecord implements IndexedRecord, SpecificData.SchemaConstructable

Format related value readers for IndexedRecord

// ParuqetAvroValueReader for IndexedRecord
class IndexedRecordReader<R extends IndexedRecord> extends StructReader<Record, Record>

// OrcArvoValueReader for IndexedRecord
// ...
// AvroValueReader for IndexedRecord
// should already exists

Engine specific file readers.

Please help to correct me if my understanding isn't correct.

But I think that we will need an IndexedRecord implementation for file/position deletes so that we can build a filter that merges that stream with a data file record stream.

@rdblue , I 'm still confusing about this, could you please help me to understand why we need this? Every engine should have its data model that means it could build the filter based on its data model. For example, the spark engine could build the filter on InternalRow that from base file and delete file. Thanks in advance.
I also create a draft PR: https://github.com/apache/incubator-iceberg/pull/971.

I'm going to close this because the spec part is done. We can track the reader in #971, or open another issue for it in the milestone.


bump nbody

@SebastianM-C could you take a look at this? Do you expect it to take more than 10 hours?

Hmm, it could be due to the stepsize calibration for the symplectic solvers. I'll take a look and double check that.

I've re-ran the timestep calibration code
https://github.com/SciML/SciMLBenchmarks.jl/blob/a7df7e7c0fe27c2915769312a8cbd27608c90043/benchmarks/NBodySimulator/liquid_argon.jmd#L123-L129
and I get
9-element Vector{Float64}:
  1.0
  2.9735980871351813
  3.0005619459841046
  3.0164920767484356
  4.985990322613598
  6.9410438162718835
  8.011579310978117
 18.053161246947383
 35.93506077349945

as opposed to the hardcoded
c_symplectic = [
    1.00,   # VelocityVerlet
    1.05,   # VerletLeapfrog
    0.98,   # PseudoVerletLeapfrog
    1.02,   # McAte2
    2.38,   # CalvoSanz4
    2.92,   # McAte5
    3.74,   # Yoshida6
    8.44,   # KahanLi8
    15.76   # SofSpa10
]

I'm not sure why the difference is so big, but it could be either that we have longer simulation times or lower number of steps. Either way, we are now passing lower c_symplectic values than it would be needed, and since dt=τ*cₐ
https://github.com/SciML/SciMLBenchmarks.jl/blob/a7df7e7c0fe27c2915769312a8cbd27608c90043/benchmarks/NBodySimulator/liquid_argon.jmd#L113
we get smaller timesteps for the symplectic integrators, which are by far the most time consuming part of the benchmark. We give roughly half the value that I obtained, so I think that could double the simulation time (assuming linear scaling). Since this took more than a couple of hours already, I would assume that it could be an explanation for the 10h+ runtime.
I'll try to look to the absolute runtimes from the past and what I get now over the next days.

any update?

In the past ~6 hours I've been running run_benchmark!(results, t, symplectic_integrators, τs[1], c=c_symplectic), i.e. just compute the results for the first timesteps and it looks like the runtimes are larger. For example the progress bar for CalvoSanz4 indicates around 1h 7min remaining (and it started a while ago, it says 32%) and the time in https://benchmarks.sciml.ai/html/NBodySimulator/liquid_argon.html for that one is 42.344 mins.
Even if we ignore the slowdown, the original benchmark took more than 10 hours, just by looking at the runtime column on the "fixed simulation time and variable timestep" benchmark. I recall that I ran it partially on a server for several hours (with less than 9 integrators) and just for the PR I enabled all of them. We have 9 integrators that go through 10 sets of timesteps and with the first 2 we can say that we have an average runtime of 40mins. This already gives us a minimum runtime of 12h.
The solution would be to reduce the total simulation time (the t variable) and maybe also to recalibrate c_symplectic and use larger time steps in τs.

I just set the max really really high. Let's see how that goes.


Can wdio-aws-device-farm-service be used to run appium tests?

Can wdio-aws-device-farm-service be used to run appium tests?

Hi @vbhardwaj-matchless, we answered this in https://github.com/awslabs/wdio-aws-device-farm-service/issues/6.


[libigl] fix imgui feature

What does your PR fix?
Fixes imgui feature for libigl port. https://github.com/microsoft/vcpkg/pull/19677 changed the feature name opengl3-glew-binding to opengl3-binding which is used on https://github.com/microsoft/vcpkg/blob/master/ports/libigl/CONTROL for imgui feature.

Which triplets are supported/not supported? Have you updated the CI baseline?
Same as before

Does your PR follow the maintainer guide?

Yes

Thanks for the fix!


8925 Write a migration to add a column named archTypeCode in systemMachineImages

https://github.com/Shippable/pm/issues/8925

Write a migration to add a column named archTypeCode in systemMachineImages table with default value system code of 'x86_64' i.e. 8000.
Add a unique index on isDefault and archTypeCode for atmost one value to be true

.

Completed.


 However, I can filter hidden files with Clap files! ++finder=fd --type f . or Clap files! ++finder=fd --type f -H .  (Please note that -H is same as --hidden)
Clap debug

            has ctags: Universal Ctags 6.0.0 (+json)
            has cargo: 1
            has maple: /Users/phong.nguyen/.vim/plugged/vim-clap/target/release/maple
           maple info: version 0.1.44, compiled at: 2023-06-19 23:37:11.084961 +07:00, built for aarch64-apple-darwin by rustc 1.70.0 (90c541806 2023-05-31).

        rustc version: rustc 1.70.0 (90c541806 2023-05-31)

     Current FileType:
Third Party Providers: []
       Global Options:
    let g:clap#autoload_dir = '/Users/phong.nguyen/.vim/plugged/vim-clap/autoload'
    let g:clap#popup#display = {'shrink': function('61'), 'open': function('<SNR>36_create_display'), 'shrink_if_undersize': function('60')}
    let g:clap#popup#preview = {'line_count': function('<SNR>35__line_count'), 'show': function('62'), 'hide': function('63'), 'clear': function('64'), 'add_highlight': function('66'), 'get_lines': function('<SNR>35__get_lines'), 'getbufvar': function('<SNR>35__getbufvar'), 'setbufvar_batch': function('<SNR>35__setbufvar_batch'), 'setbufvar': function('<SNR>35__setbufvar'), 'win_is_valid': function('<SNR>35__win_is_valid'), 'goto_win': function('<SNR>35__goto_win'), 'set_syntax': function('65')}
    let g:clap#provider_alias = {'gfiles': 'git_files', 'hist:': 'command_history', 'hist/': 'search_history'}
    let g:clap_actions = ['open-config', 'generate-toc', 'update-toc', 'delete-toc']
    let g:clap_background_shadow_blend = 50
    let g:clap_disable_bottom_top = 0
    let g:clap_disable_matches_indicator = v:false
    let g:clap_disable_run_rooter = v:false
    let g:clap_enable_background_shadow = v:false
    let g:clap_enable_debug = v:false
    let g:clap_enable_icon = 0
    let g:clap_forerunner_status_sign = {'done': '•', 'running': '!', 'using_cache': '*'}
    let g:clap_insert_mode_only = v:false
    let g:clap_layout = {'relative': 'editor'}
    let g:clap_multi_selection_warning_silent = 0
    let g:clap_no_matches_msg = 'NO MATCHES FOUND'
    let g:clap_open_action = {'ctrl-v': 'vsplit', 'ctrl-x': 'split', 'ctrl-t': 'tab split'}
    let g:clap_open_preview = 'always'
    let g:clap_popup_border = 'rounded'
    let g:clap_preview_direction = 'UD'
    let g:clap_preview_size = 5
    let g:clap_providers_relaunch_code = '@@'
    let g:clap_search_box_border_style = 'nil'
    let g:clap_search_box_border_symbols = {'nil': ['', ''], 'curve': ['', ''], 'arrow': ['', '']}
    let g:clap_theme = 'atom_dark'
  Provider Variables:
                     []

To Reproduce
Steps to reproduce the behavior:

Create the minimal vimrc min.vim:

set encoding=utf-8
scriptencoding utf-8

set nocompatible
set cmdheight=2
set termguicolors
set runtimepath^=~/.vim/plugged/vim-clap

syntax on
filetype plugin indent on

let g:clap_theme = 'atom_dark'
let g:clap_layout = { 'relative': 'editor' }
let g:clap_preview_direction = 'UD'

Start (neo)vim with command: vim -u min.vim

Please view screencast

See error

The performance of Rust backend has been significantly improved by using an interprocess instead of collecting the output from an external process (see #875), so there are some changes on the user end, but it generally simplifies the flow:

! in :Clap files! is no longer necessary, the cache thing has been deprecated, now the results are always the latest and it's even faster.
The files provider has been re-implemented on the Rust side, ++finder has been retired, now you only need to specify --hidden to search the hiddle files.

https://github.com/liuchengxu/vim-clap/blob/9560801750812580cf71c0e03c53bb9aa65cf64b/crates/maple_core/src/stdio_server/provider/files.rs#L19
Therefore,  :Clap files --hidden is enough, we can also add this flag in the config.toml so that you don't have to always specify it every time.

Thanks for your updates.

Therefore, :Clap files --hidden is enough, we can also add this flag in the config.toml so that you don't have to specify it every time.

What I need to update? As followings?
[files]
hidden = true

Also, If files provider was implemented on server side, fd or rg is not used on server as well?

we can also add this flag in the config.toml so that you don't have to specify it every time.

Not yet implemented, but trivial to support it.

fd or rg is not used on server as well?

True, these binaries are needless and unused, all you need is the Rust binary maple.

Thanks. Closing the issue now.

The argument parsing will be completely rewritten, feel free to subscribe https://github.com/liuchengxu/vim-clap/issues/981 and leave your feedback.


API to get data about a leaderboard from CodaLab

Is there an API that I can use to get the data from the leaderboard of a Codalab competition?

I don't think so. Maybe a work around can be found.


retrieving field value before form is validated

Hello,
Thanks for this lovely plugin.  However, we are facing a major issue which is holding us back. I have a form where I need to observe a field within a form (quantity) and update a field total price as the field is changed.
How to access the value before the form is validated?
Thanks

Hi @introvert, thanks for your nice feedback! :smile:
Did you already try using the on-update option on your input field? This basically makes it one-way-bound.

Hey @czosel,
Thanks for your quick reply! I did - it changes to the one-way bound as you said, but it doesn't update the field as its supposed to. Do you maybe have any working examples?
Much appreciated.

Since the field is one-way-bound, you'll have to update it manually.  See this twiddle for an example. Let me know if that helped, and feel free to tweak the twiddle a bit to illustrate your issue better if it didn't! :smiley:

Awesome @czosel.
Thanks a lot for this!

You're welcome @introvert! I'll close this - feel free to open a new issue anytime you have another question or a bug to report!


type-at-pos fails with literal string enum types

e.g.
type MyType = "one" | "two"
let variable: MyType;
//    ^ type-at-pos here, to find out the type of `variable`, reports "unknown"

version: v0.22.0
(just tried literal number enums, and they also fail)

seems like it'd be nice to have a --verbose mode or something for debugging this stuff

ah, looks like the type_printer just doesn't think that singletons are printable -- which is funny, b/c they totally are. I'll throw a pr together


feat(upgrade): add controller phases

Add controller and phases for upgrade

bors try

bors try

bors try

bors try

bors try

bors try

bors try

bors try

bors merge

bors try

bors merge


docs: document custom step usage

DO NOT MERGE
This PR aims to provide documentation for custom step usage in the bolt framework
Testing

Pull this branch
cd ./docs
npm install
npm start
http://localhost:3000/bolt-js/concepts/custom-steps

Requirements (place an x in each [ ])

[x] I've read and understood the Contributing Guidelines and have done my best effort to follow them.
[x] I've read and agree to the Code of Conduct.

Removed the existing docs/content/custom-functions documentation
Reason for this is that it included information that was not bolt specific and duplicated in api.slack.com. It also seems like overkill to have a dedicated sidebar section for custom steps


webpack@2 compatibility?

peerDependencies list webpack@1, so this module cannot be used with webpack@2 yet

#128 will support the latest webpack 2.1.0 beta releases.


PR for Issue #1

Did some refactoring to allow for testing the environment load; had to replace the constants with class-level accessible variables.  Modified existing rake tasks accordingly.
Fixed Issue #1 which caused the DB_CONFIG_ADMIN constant to override the database setting imported from database.yml.

@SeanMeh Yo, thanks for opening the issue and making a PR.
Can I ask you for a tiny favour please? Can you squash the commits into one please and I will merge it straight away?
Also, would you like commit rights to the repository as well?
Thanks again.
👍

Hey there.  I just saw your message after fixing another issue I found.  Yeah, I can squash the commits.  Thanks for the quick response.
If you'd like to give commit permissions, I'll take it.  I don't know that I'll come across any other issues, but I'm new to working with ActiveRecord without Rails's magic behind it so other things might not be readily apparent.

@SeanMeh What database engine are you using? Because, I tried it with just now (excluding your changes) and it worked just fine. postgres is a default database for posrgresql's engine.

@andhapp We're using MySQL, specifically the mysql2 adapter.  The 'database' attribute in the config settings (for MySQL anyway) tells ActiveRecord which MySQL database to use (as defined in the database.yml), so when specified prior to creating the database MySQL doesn't like that too much.

Out of curiosity, is there any particular reason anyone should want to use a dbase when creating the dbase the app is going to use?  Not sure what the value of that is.

Thanks. I see what the problem is now. I have a better solution that will work across all the supported database engines. I will create a fix and open a PR for it and you can comment on it as well. Sounds good?

@SeanMeh Hey, I found this gem, standalone-migrations which seems to be well-supported. I am going to stop this project in favour of this gem instead as there is no point keeping two gems that tackle similar problem. Give that a try and see if that works for your use case.
Thanks.

Closing this issue now.


useQueryParams / useQueryParam

https://www.npmjs.com/package/use-query-state
https://github.com/pbeshai/use-query-params
https://www.npmjs.com/package/use-url-search-params

[x] useQueryParam
[ ] useQueryParams

No need for multiple params. Once required - implement


Invoke-WebRequest test failure: "Denial of service.Image Parsing"

Invoke-WebRequest test "Denial of service.Image Parsing" is failing intermittently:

PowerShell-CI-linux failed in build 45608
PowerShell-CI-linux failed in build 44792

Error message: Expected the actual value to be greater than 5, but got 0.368549962301977.
Stack trace:
at <ScriptBlock>, /home/vsts/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1: line 1932
1932:             $pathologicalRatio | Should -BeGreaterThan 5

Test has not failed in last 30 days, closing.

This seems like it may be a fragile test, as it is randomly failing again on release builds.


Add linter Feature/86bxjkyxt add linter

Adding C/C++ linter to the project. This will be only for additional informative checks without being restrictive to being able to merge.

I think I need to review the comments that the linter does. Will continue a little my investigation to narrow down the current implementation of the linter so it can be useful and informative for us.

Done, now the checks, warnings and errors are c++ related


Failed to authenticate

Describe the bug
I try to authenticate with my Spotify username and password (as uncomfortable as that makes me), but authentication fails. I have a Premium Student account.
To Reproduce
Execute spotify-player, type username, then type password
Expected behavior
Authentication succeeds and I can finally listen to my music
Log and backtrace
Doesn't get far enough to generate a log
Environment

OS: Void Linux x86_64 glibc
Application version: latest commit
Application features: built with cargo build -r --target x86_64-unknown-linux-gnu -Z build-std=std,panic_abort -Z build-std-features=panic_immediate_abort and the following added to all Cargo.toml files:

[profile.release]
strip = true
lto = true
opt-level = 3
panic = "abort"

Additional context
I suspect my password having unusual characters, specifically a double-quote, may be causing this issue.

Premium Student

what is this?

I'm also on my premium student account right now. However, I've never failed to authenticate.

Username needs to be email; that fixes it.


Pre-render setState errors

Using watchIO' from Excelsior in a Component's construct method and having a conditional render in the Component's renderer can cause a _.parentNode.replaceChild(_,_) error.  This is because the view is unmounted, but a setState is trying to swap it out.
A quick solution is to change the replaceNodes js implementation from:
current.parentNode.replaceChild(current, new);

to:
if (current.parentNode) { current.parentNode.replaceChild(current, new);

At the moment, it seems like this should work.  I think the mount that happens after the setState will read the current view from a mutable reference.  If not, this could be an issue.  I'll have to keep an eye on it.

I'm going to say this is a non-issue/user error since Component.mounted exists.  I will eventually revert the conditional parentNode check.


Fix overflow contents view table

Added horizontal scrollbar if table contents exceed content width.
https://user-images.githubusercontent.com/51911425/128003389-1be228ba-d8da-4ec5-9ce2-8ed55c4622f2.mov

@giuliaghisini same PR here: https://github.com/plone/volto/pull/1697

@giuliaghisini same PR here: #1697

oh, see now.
But.. i think this PR is more complete, because scrollbar will appear only if needed.
If you add only the 'overflow-x: auto' without handle the table styles, scrollbar will appear even if not needed because of .ui.attached.table styles.
This PR takes care of this and of table styles.

:) how do we get that last column pinned?

:) how do we get that last column pinned?

I guess we don't have any better option if not something really tricky and hacky, and I wouldn't go for that right now

I do not aggree with this PR. The folder contents view is for information and is being used to select an item for editing. The action 'edit' is already hidden in an '...' menu. So this menu should at least be visible, my opinion.
See this suggestion from last year: https://github.com/plone/volto/pull/1568#issuecomment-639467899 and imagine a beautiful designed version of it.

(not related to the PR, just a place to vent) That Contents.jsx is one of those big component files that need to be refactored, to allow us to "grow" with new functionality

@tiberiuichim one such example https://github.com/plone/volto/pull/2596

I kind of agree with @ksuess, although is the kind of thing we were all searching for, it has the drawback that the ... menu is potentially not visible at first sight.
For reference, this is the Quanta mockup:

Albert designed it even without it being always visible, I guess to remove visual clutter. I kind of like Google Drive's approach (it's behind a right click on the element), but agreed that it can be a bit too hidden for the average user.
Also, there's the ongoing agreement that the contents folder should have an app-ish look and feel, not to be constrained by the website "frame" structure. So I lean more towards Katja solution + Quanta additions + app-ish look and feel.

I wonder if using react-table is sensible here:
https://codesandbox.io/s/react-table-fixed-columns-official-kowzlm5jp7
Using fixed columns seems easy. Then we could have all the filtering options.

+1 on adding react-table as a dependency to Volto

+1 for fixing column 'title' and column 'actions' ('react-table' seems to be an elegant implementation for fixing columns)
+1 for showing an actions icon like it is done now (not a right click menu like in Google Drive. IMHO it is not obvious enough to be understood by rarely working users)

@plone/volto-team Let's merge this, if possible.

Agreed to merge unless we have a better approach based on the comments above in:
https://github.com/plone/volto/issues/2991


Accessing DO_WHILE 'iteration' value inside the the looped tasks.

Hi, anyone know how to access the iterator value inside the tasks, which are running as part of the DO_WHILE loop. For eg: I want to read an array value inside the running tasks. The array would be an input to the looped task from another task output, which was executed before the loop started. I am not sure how we can use the DO_WHILE 'iteration' variable for reading the array values.
I want to use the 'iteration' value something like below inside a task defined inside the DO_WHILE loop. Here 'preOrder_service' is task which runs before the the loop starts and 'products' is an array.
preOrder_service_get.output.response.body..products['iteration'].pluCode

Hi @krishnaputhran ,
Please check the documentation here https://netflix.github.io/conductor/configuration/systask/#do-while-task.
You can use the LAMBDA task to parse the input array that is coming from the above task.
Here is the sample json that might help.
{
    "name": "test_loopTask",
    "taskReferenceName": "test_loopTask",
    "inputParameters": {
        "orders_count": "${workflow.input.orders.size}"
    },
    "type": "DO_WHILE",
    "loopCondition": "if ($.test_loopTask['iteration'] <= $.orders_count) { true; } else { false; }",
    "loopOver": [
        {
            "name": "get_order_at_index",
            "taskReferenceName": "get_order_at_index",
            "inputParameters": {
                "orders": "${workflow.input.orders}",
                "iterator": "$.test_loopTask['iteration']",
                "scriptExpression": "return $.orders.get($.iterator);"
            },
            "type": "LAMBDA"
        }
    ]
}

Hi @manan164  thank you very much for the response and I appreciate your timely response. I have tried reading the do-while-task documentation, but couldn't get answer for my problem, but somehow I didn't consider the use fo LAMDA tasks.
I tried to follow your suggestion. But somehow, I feel that LAMDA task inside the loop is getting executed only once and hence same output from the first execution is getting passed in subsequent executions without getting the next value of the array. Please see my test workflow def and wrorkflow start scripts.
WorkfFlow Definition
{
"name": "test_do_while_workflow",
"description": "test_do_while_workflow",
"tasks": [
{
"name": "LoopTask",
"taskReferenceName": "LoopTask",
"type": "DO_WHILE",
"inputParameters": {
"preOrderListSize": "${workflow.input.value.response.body.products.length()}"
},
"loopCondition": "if ( $.LoopTask['iteration'] < $.preOrderListSize )  { true; } else { false; }",
"loopOver": [
{
"name": "get_order_at_index",
"taskReferenceName": "get_order_at_index",
"inputParameters": {
"orders": "${workflow.input.value.response.body.products}",
"iterator": "$.LoopTask['iteration']",
"scriptExpression": "return $.orders.get($.iterator);"
},
"type": "LAMBDA"
}
],
}
]
}
Input to the workflow
{
"name": "test_do_while_workflow",
"correlationId": "test_do_while_workflow",
"input": {
"value":{
"response": {
"body": {
"products": [
{
"pluCode":<PHONE_NUMBER>498,
"quantity": 2
},
{
"pluCode":<PHONE_NUMBER>873,
"quantity": 3
}
]
},
}
}
}
}
Even the Conductor UI shows the execution of LAMDA function only once, although my input array products size is 2.

But in the DO-WHILE task output shows two executions.
DO-WHILE task output from the conductor UI
{
"1": {
"get_order_at_index": {
"result": {
"pluCode":<PHONE_NUMBER>498,
"quantity": 2
}
}
},
"2": {
"get_order_at_index": {
"result": {
"pluCode":<PHONE_NUMBER>498,
"quantity": 2
}
}
},
"iteration": 2
}
If I add a normal task, like HTTP, after the LAMDA task it gets executed twice. I removed it here for reducing the text size.

@manan164  I could see this is a defect. By adding a dummy task before this one makes the next task to be executed. So issue is that, in a DO_WHILE loop, the first task in the loopOver is gets executed only once. So a dummy task need to be added to address this issue.
But one more issue I saw. Hope you reply to my concern.  As per your sample code, I tried to access the array values inside the loop, but it always gives the 0th array value.
"iterator": "$.test_loopTask['iteration']",
"scriptExpression": "return $.orders.get($.iterator);"
from the above script expression, the value returned is always the zeroth value from the orders.

Hi @krishnaputhran There may be an issue with the sequence number. Please check do_while task output there you will see all loopOvertaks output.
Till the time I will come up with the finding, Please check DYNAMIC_FORK This will solve your purpose. You have to generate input to this task from the previous http task. The only thing is it will execute this task in parallel instead of sequential.

@manan164  thanks for the response. The do_while task output i have checked, and its has the expected output. Only issue is getting the "iteration" value in subsequent iterations.
I had little doubt regarding the usage of $.test_loopTask['iteration'], because as per jasonpath documentation $  refers to the current object, where in, when inside the task, $ refers to the current task under execution, than the actual root task. So I gave another try as shown below.
	"iterator": "${test_loopTask.output.iteration}",
	"scriptExpression": "return $.orders.get($.iterator)"

to my surprise, this worked, but had some strange issue. During the  execution the value of iterator fetched using "${test_loopTask.output.iteration}" goes like this
1,1,2,3....n
If there are only two iterations, the iterator will always have the value 1. meaning for the second execution the value of iteration will still be 1 and after second execution the iteration value starts increasing.
But if you check the do..while task output, they are properly stored in the key value pair with 1, 2, 3 as keys.
But I am afraid, as per my reading on DYNAMIC_FORK will solve my problem. My issue is I want to fetch the array value one by one, which forms input to my set of tasks to be executed in sequence. This is a typical looping use-case.  My task execution list is not dynamic. Anyway I'll give a try.

@krishnaputhran
Did looping issue got resolved? I used same code that you mentioned above but same object is getting passed in all the iteration.
Please let us know in case if you get answer.
Thanks

@manan164 Can you please chime in on this one? Thanks

Sure. @apanicker-nflx

Experiencing similar issues in binding the iteration to a task within "loopOver".
If there is only a single task defined then I can't bind the iteration. Input always results as null.
If there are one or more tasks defined then I can't bind the iteration to the first task in the loopOver array. Input always results as null. Using the same binding "${taskref.output.iteration}" on the other tasks works fine.

Hi @krishnaputhran ,
Please check the documentation here https://netflix.github.io/conductor/configuration/systask/#do-while-task.
You can use the LAMBDA task to parse the input array that is coming from the above task.
Here is the sample json that might help.
{
    "name": "test_loopTask",
    "taskReferenceName": "test_loopTask",
    "inputParameters": {
        "orders_count": "${workflow.input.orders.size}"
    },
    "type": "DO_WHILE",
    "loopCondition": "if ($.test_loopTask['iteration'] <= $.orders_count) { true; } else { false; }",
    "loopOver": [
        {
            "name": "get_order_at_index",
            "taskReferenceName": "get_order_at_index",
            "inputParameters": {
                "orders": "${workflow.input.orders}",
                "iterator": "$.test_loopTask['iteration']",
                "scriptExpression": "return $.orders.get($.iterator);"
            },
            "type": "LAMBDA"
        }
    ]
}

Any update on this issue? This proposed solution does not work, and yields the first element of the array for every iteration.
There seems to be no way to iterate over members of an array.

There seems to be an error in the workflow task (loopOver[0].inputParameters.iterator), I think it should be ${test_loopTask.iteration} and not $.test_loopTask['iteration']. See:
        {
            "name": "get_order_at_index",
            "taskReferenceName": "get_order_at_index",
            "inputParameters": {
                "orders": "${workflow.input.orders}",
                "iterator": "${test_loopTask.iteration}",
                "scriptExpression": "return $.orders.get($.iterator);"
            },
            "type": "LAMBDA"
        }

I tried that. It doesn't work either. iterator becomes null, and script still returns the 0 element in the array.
I have a workaround using lambas to track a loop_index that I hack onto workflow.input. This should work.
{
    "name": "init_loop_index",
    "taskReferenceName": "init_loop_index",
    "inputParameters": {
        "wfinput": "${workflow.input}",
        "scriptExpression": "$.wfinput.loop_index=0; return 0;"
    },
    "type": "LAMBDA"
},
{
    "name": "test_loopTask",
    "taskReferenceName": "test_loopTask",
    "inputParameters": {
        "orders_count": "${workflow.input.orders.size}"
    },
    "type": "DO_WHILE",
    "loopCondition": "if ($.test_loopTask['iteration'] <= $.orders_count) { true; } else { false; }",
    "loopOver": [
        {
            "name": "get_order_at_index",
            "taskReferenceName": "get_order_at_index",
            "inputParameters": {
                "orders": "${workflow.input.orders}",
                "wfinput": "${workflow.input}",
                "iterator": "${workflow.input.loop_index}",
                "scriptExpression": "$.wfinput.loop_index++;return $.orders.get($.iterator);"
            },
            "type": "LAMBDA"
        }
    ]
}

Better not use workflow.input as it is immutable (or at least suppose to be)
I found out that iteration is stored in the DO_WHILE output. see the following:
{
  "schemaVersion": 2,
  "name": "test_do_while_workflow",
  "description": "test_do_while_workflow",
  "tasks": [
    {
      "name": "mock_data",
      "taskReferenceName": "mock_data",
      "inputParameters": {
        "scriptExpression": "var products = new java.util.ArrayList();\nproducts.add({\"pluCode\":<PHONE_NUMBER>498,\"quantity\": 2});\nproducts.add({\"pluCode\":<PHONE_NUMBER>873,\"quantity\": 3});\nreturn {\"products\": products};"
      },
      "type": "LAMBDA"
    },
    {
      "name": "test_loopTask",
      "taskReferenceName": "test_loopTask",
      "inputParameters": {
        "orders_count": 2
      },
      "type": "DO_WHILE",
      "loopCondition": "$.test_loopTask['iteration'] < $.orders_count",
      "loopOver": [
        {
          "name": "get_order_at_index",
          "taskReferenceName": "get_order_at_index",
          "inputParameters": {
            "orders": "${mock_data.output.result.products}",
            "iterator": "${test_loopTask.output.iteration}",
            "scriptExpression": "return $.orders.get($.iterator);"
          },
          "type": "LAMBDA"
        }
      ]
    }
  ],
  "ownerEmail"<EMAIL_ADDRESS>  "version": 1,
  "outputParameters": {
    "result": "${test_loopTask.output}"
  }
}

Returns the following output:
{
  "result": {
    "1": {
      "get_order_at_index": {
        "result": {
          "pluCode":<PHONE_NUMBER>498,
          "quantity": 2
        }
      }
    },
    "2": {
      "get_order_at_index": {
        "result": {
          "pluCode":<PHONE_NUMBER>873,
          "quantity": 3
        }
      }
    },
    "iteration": 2
  }
}

Interesting.. I did try output.iteration, but I found, as reported here by others, that the output.iteration sequences as "1,1,2,3,4" instead of "0,1,2,3,4", which is not usable. perhaps this has been fixed lately? This would def. be preferred to my hack. We are running our own instance of conductor.. perhaps it is old..

Hi @FrankOwen The seq number issue has been fixed via 1815.
We are putting iteration value in DO_WHILE task output Please check documentation https://netflix.github.io/conductor/configuration/systask/#do-while-task.
Do while task output number of iterations with iteration as key and value as number of iterations
Putting it in each loop over task is not needed as we can figure out from DO_WHILE task.

Thanks, @manan164 . I guess our deployment must be more than 17 days old! I will see about getting updated. Would be good to remove my hack. Putting it in output is a little weird, since I think of 'output' as FINAL value of a task. But I'll take whatever works!

d

Hi, I am trying to execute a task 'A' inside do while task and I want to change the input to task 'A' on every iteration. Can we do this?

Hi @Priyank0711 , Yes it is possible. Please check @elisherer 's comment, https://github.com/Netflix/conductor/issues/1677#issuecomment-675687392


Add a detailed README.md

This should describe:

the purpose of the project
the current status, especially which devices are currently supported
how to use the project (in particular install webots, checkout recursively, build, copy jar to appropriate folder, etc)
how to name webots nodes so that they are connected

PR #39 addresses everything except how to name webots nodes. Since that will be irrelevant once #24 is addressed, I think we can skip it for now.

PR #39 addresses everything except how to name webots nodes. Since that will be irrelevant once #24 is addressed, I think we can skip it for now.


Using linkdex to validate migrated data

I'd like to use linkdex (or something similar) to validate data migrated from IPFS Cluster to S3.
The tweaks I would like are:

Allow indexing CARs with complete/ prefix.
Ensure each block hashes to reported CID from CAR i.e. Block.create instead of Block.createUnsafe here (an optional query string param?).
Get completeness of a particular DAG within the CAR(s) (allows us to ensure the CAR contains what we expect).

Largely addressed by https://github.com/web3-storage/linkdex-api/pull/6, https://github.com/web3-storage/linkdex-api/pull/11 and https://github.com/web3-storage/linkdex-api/pull/13


record_log takes too much time for medium - large scale simulation

I also did some benchmarks using example_04en_automatic_network_generation.py (440 links, 12100 platoons, 1440 timesteps) and surprised that Vehicle.record_log takes long time (~50% of simulation) considering its simple task. I suspect too many use of append is the reason.
Since I won't be able to work on it for a while, I'll keep a record of it for later update.
with record_log

without record_log

Replacing the list to np.array did not improve the performance as expected.
I will consider to add options to lower the logging frequency or turn it off completely.

Added logging settings.
By setting World.vehicle_logging_timestep_interval=-1, the record_log is turned off, and the simulation time can be significantly reduced  (~40% speed up) as shown in the above post.
The logging setting does not affect the internal simulation accuracy. Only the outputted trajectories are affected.
With World.vehicle_logging_timestep_interval=2, the simulation time can be reduced (~20% speed up), and we can obtain vehicle trajectory data with slightly less accuracy.

With World.vehicle_logging_timestep_interval=2:

With World.vehicle_logging_timestep_interval=1 (default, complete logging):


When there is only 1 attribute in the Realm Attributes UI, you can't delete it

When there is only 1 attribute in the Realm Attributes UI, you can't delete it (i.e. clicking the minus button does nothing).

Also, when you save on this screen, there is only a saveSuccess message in the toast, rather than the actual message.

Fixed. It was actually a "feature" of the Attributes form. The form was always requiring at least one attribute. So if you wanted to clear them all you'd had to of manually deleted the last one, which seems like a bug honestly. I made it so you can customize that functionality.


Version 0.14.0

v0.14.0 (2020-04-14)
Bug Fixes

Rework how hanging lambda/parens are handled (732f09f)
Accept filenames that start with ./.. as modules (01e450b)
Don't deadlock when collecting and cloning a thread concurrently (5886f59)
check:  Reject programs which misspecifies the number of patterns (248387d, closes #807)
doc:  Correct the style.css path (4e629ab)
format:  Improve tuple multiline formatting (5122fe3)

Features

Add AstClone to clone arena allocated ASTs (3ee7bd2)
Compile modules in parallel using salsa-async (e0ab181)
Compile modules in parallel (57fca16)
Add Function::call_any (2c06104)
Export Array from the prelude (16eb345)
Allow serde_json::Value to be marshalled to std.json.Value (aabdec8)
check:

Avoid propagating errors on lift_io! misuse (3dbabe5)
Avoid generating more errors from a type that could not be imported (752e2bc)

Performance

Avoid some unnecessary allocations (089bae4)
Shrink the size of Pattern (75fb884)
Avoid hashing symbols twice (0cfb52c)
Allocate all temporaries into the same Vec (af945c4)
compiler:  Hoist a remove_alias call out from match alternatives (c13172e)

bors r+

bors r+


the phantom module using<EMAIL_ADDRESS>does not work

the latest version of phantom is using 0.4.1, Why the package.json specifies the SHA1 at d4e40cae23c069c35500b8282cf3e41d3afb6d1e ?

It contained some (previously unpublished) patches. I'm happy to accept a PR to upgrade the version if it's tested!


Project Member list is empty for cluster owners.

Rancher versions: v2.0.0-alpha19
Steps to Reproduce:
Rancher server version - v2.0.0-alpha19
Steps to reproduce the problem:
As admin , Create DO cluster c1 and assign "user1" as "owner" of this cluster.
Add few projects in c1.
Log in as user1.
He is able to see the projects under c1.
In Edit project view , member list for project is shown.

But member list is empty in project's Member tab.

Version - 2.0 master 4/2


Max Download Reached Exits Python Program.

Checklist

[x] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2020.12.29
[x ] I've checked that all provided URLs are alive and playable in a browser
[x ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x ] I've searched the bugtracker for similar bug reports including closed ones
[x ] I've read bugs section in FAQ

Verbose log
[youtube:tab] PLjTI-fmjC4hWzJWz4_GTx1OzU8IZPUpfO: Downloading webpage
[download] Downloading playlist: Movie Clips
[youtube:tab] Downloading page 1
[youtube:tab] playlist Movie Clips: Downloading 192 videos
[download] Downloading video 1 of 192
[download] Sing (2016) - Shake It Off Scene (6/10) | Movieclips has already been recorded in archive
[download] Downloading video 2 of 192
[download] Despicable Me 2 (9/10) Movie CLIP - The Purple Minion Attacks (2013) HD has already been recorded in archive
[download] Downloading video 3 of 192
[download] Despicable Me 2 (10/10) Movie CLIP - Battling the Minions (2013) HD has already been recorded in archive
[download] Downloading video 4 of 192
[download] Despicable Me 2 (8/10) Movie CLIP - Worst Date Ever (2013) HD has already been recorded in archive
[download] Downloading video 5 of 192
[download] Despicable Me 2 (7/10) Movie CLIP - Margo In Love (2013) HD has already been recorded in archive
[download] Downloading video 6 of 192
[download] Despicable Me 2 (1/10) Movie CLIP - The Most Magical Fairy Princess (2013) HD has already been recorded in archive
[download] Downloading video 7 of 192
[download] Despicable Me 2 (4/10) Movie CLIP - A Minion in Love (2013) HD has already been recorded in archive
[download] Downloading video 8 of 192
[download] Fast & Furious 6 (8/10) Movie CLIP - Dom Saves Letty (2013) HD has already been recorded in archive
[download] Downloading video 9 of 192
[download] Full Frontal (7/8) Movie CLIP - Happy Ending (2002) HD has already been recorded in archive
[download] Downloading video 10 of 192
[download] Fast Five (4/10) Movie CLIP - A Woman's Job (2011) HD has already been recorded in archive
[download] Downloading video 11 of 192
[youtube] TIbZNmvyLBI: Downloading webpage
[18532:23840:1229/002903.219:ERROR:device_event_log_impl.cc(211)] [00:29:03.219] Bluetooth: bluetooth_adapter_winrt.cc:1073 Getting Default Adapter failed.
[download] Destination: raw_videos\Fast Five (5_10) Movie CLIP - You're Under Arrest (2011) HD.mp4
[download] 100% of 7.58MiB in 00:00
[download] Downloading video 12 of 192
[youtube] k2PsfXZ3wyY: Downloading webpage
[download] Destination: raw_videos\Fast Five (7_10) Movie CLIP - Hobbs vs. Toretto (2011) HD.mp4
[download] 100% of 12.22MiB in 00:00
[download] Downloading video 13 of 192
[youtube] uzHwlt3dmmo: Downloading webpage
[download] Destination: raw_videos\Fast & Furious 6 (7_10) Movie CLIP - They Got a Tank (2013) HD.mp4
[download] 100% of 14.21MiB in 00:01
[download] Downloading video 14 of 192
[youtube] xEAsC8A1Ins: Downloading webpage
[download] Destination: raw_videos\Fast Five (10_10) Movie CLIP - The Bridge Showdown (2011) HD.mp4
[download] 100% of 16.29MiB in 00:00
[download] Downloading video 15 of 192
[youtube] gN1BjYlhKvc: Downloading webpage
[download] Destination: raw_videos\Fast Five (6_10) Movie CLIP - Million Dollar Race (2011) HD.mp4
[download] 100% of 12.59MiB in 00:00
[download] Downloading video 16 of 192
[youtube] 9aldDI2WF7g: Downloading webpage
[download] Destination: raw_videos\Fast Five (8_10) Movie CLIP - Street Ambush (2011) HD.mp4
[download] 100% of 16.71MiB in 00:00
[download] Downloading video 17 of 192
[youtube] frE9rXnaHpE: Downloading webpage
[download] Destination: raw_videos\Ben-Hur (3_10) Movie CLIP - The Chariot Race (1959) HD.mp4
[download] 100% of 47.96MiB in 00:01
[download] Downloading video 18 of 192
[youtube] km3VtR6mqmg: Downloading webpage
[download] Destination: raw_videos\Fast & Furious 6 (9_10) Movie CLIP - Boarding the Plane (2013) HD.mp4
[download] 100% of 23.36MiB in 00:01
[download] Downloading video 19 of 192
[youtube] Atb5LNPuxyU: Downloading webpage
[download] Destination: raw_videos\Fast & Furious 6 (10_10) Movie CLIP - The End of Owen Shaw (2013) HD.mp4
[download] 100% of 15.55MiB in 00:00
[download] Downloading video 20 of 192
[youtube] 4lNoSUurdKc: Downloading webpage
[download] Destination: raw_videos\Despicable Me 2 (5_10) Movie CLIP - That Pollo is Loco (2013) HD.mp4
[download] 100% of 15.97MiB in 00:00
[download] Downloading video 21 of 192
[youtube] vmm8P0V1W4g: Downloading webpage
[info] Maximum number of downloaded files reached.
Traceback (most recent call last):
  File "C:\Users\rbret\Documents\GitHub\Downloader\upload.py", line 149, in <module>
    download_new_videos()
  File "C:\Users\rbret\Documents\GitHub\Downloader\upload.py", line 68, in download_new_videos
    ydl.download(["https://www.youtube.com/playlist?list=PLjTI-fmjC4hWzJWz4_GTx1OzU8IZPUpfO"])
  File "C:\Users\rbret\AppData\Local\Programs\Python\Python39-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 2028, in download
    res = self.extract_info(
  File "C:\Users\rbret\AppData\Local\Programs\Python\Python39-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 796, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "C:\Users\rbret\AppData\Local\Programs\Python\Python39-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 803, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Users\rbret\AppData\Local\Programs\Python\Python39-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 835, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\Users\rbret\AppData\Local\Programs\Python\Python39-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1012, in process_ie_result
    entry_result = self.__process_iterable_entry(entry, download, extra)
  File "C:\Users\rbret\AppData\Local\Programs\Python\Python39-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 803, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Users\rbret\AppData\Local\Programs\Python\Python39-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1044, in __process_iterable_entry
    return self.process_ie_result(
  File "C:\Users\rbret\AppData\Local\Programs\Python\Python39-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 906, in process_ie_result
    return self.process_ie_result(
  File "C:\Users\rbret\AppData\Local\Programs\Python\Python39-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 869, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "C:\Users\rbret\AppData\Local\Programs\Python\Python39-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1654, in process_video_result
    self.process_info(new_info)
  File "C:\Users\rbret\AppData\Local\Programs\Python\Python39-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1746, in process_info
    raise MaxDownloadsReached()
youtube_dl.utils.MaxDownloadsReached

Description

I'm using the python library for youtube-dl and I've specified max_downloads in the ydl_opts. The problem I'm having is that after the max download has been reached the program exits with a traceback error. I would like to continue the program after the max download has been reached and I'm not sure how to do that. Thanks

Catch the exception obviously.


Upgrade weaveworks-ui-components to 0.3.8

I think those ones could be done without review, the original PR in ui-components has been reviewed after all.

Note: this will need to be bumped a bit more once the npm package isn't missing exports.