Add documentation for using with Kolla

Since we're more familiar with OSA, we can probably do #152 first and build this off of the same.

I started with the integration in Kolla-Ansible: https://review.opendev.org/c/openstack/kolla-ansible/+/896292.
With this patch and installed magnum-cluster-api plugin inside the magnum container images I have this error when trying to create the COE:
Cluster type (vm, Unset, kubernetes) not supported (HTTP 400) (Request-ID: req-71b9046a-61d2-498d-b0f4-243ce729f9cf)

Wrong image ID. It works with a correct image ID:
Request to create cluster template k8s-v1.27.5 accepted
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| insecure_registry     | -                                    |
| labels                | {'kube_tag': 'v1.27.5'}              |
| updated_at            | -                                    |
| floating_ip_enabled   | True                                 |
| fixed_subnet          | -                                    |
| master_flavor_id      | SCS-2V-4-10                          |
| uuid                  | df0e74af-6938-40a6-9062-2f9de3c316da |
| no_proxy              | -                                    |
| https_proxy           | -                                    |
| tls_disabled          | False                                |
| keypair_id            | -                                    |
| public                | False                                |
| http_proxy            | -                                    |
| docker_volume_size    | -                                    |
| server_type           | vm                                   |
| external_network_id   | public                               |
| cluster_distro        | ubuntu                               |
| image_id              | 929b90b6-202a-4651-b3d2-2b70e39d592f |
| volume_driver         | -                                    |
| registry_enabled      | False                                |
| docker_storage_driver | overlay2                             |
| apiserver_port        | -                                    |
| name                  | k8s-v1.27.5                          |
| created_at            | 2023-09-22T20:16:16+00:00            |
| network_driver        | calico                               |
| fixed_network         | -                                    |
| coe                   | kubernetes                           |
| flavor_id             | SCS-2V-4-10                          |
| master_lb_enabled     | True                                 |
| dns_nameserver        | <IP_ADDRESS>                              |
| hidden                | False                                |
| tags                  | -                                    |
+-----------------------+--------------------------------------+

After installing helm in the magnum-conductor container image + provisioning of Manila & Barbican I can start the creation of new clusters.  I also have a magnum-system namespace in the Cluster API management cluster:
$ kubectl get namespaces
NAME                                STATUS   AGE
default                             Active   2d14h
kube-system                         Active   2d14h
kube-public                         Active   2d14h
kube-node-lease                     Active   2d14h
metallb-system                      Active   2d14h
cert-manager                        Active   2d14h
capi-system                         Active   2d14h
capi-kubeadm-bootstrap-system       Active   2d14h
capi-kubeadm-control-plane-system   Active   2d14h
capo-system                         Active   2d14h
magnum-system                       Active   41m

Creation of the cluster itself is hanging:
2023-09-22 21:07:21.165 7 DEBUG magnum.service.periodic [-] Updating status for cluster 9 update_status /var/lib/kolla/venv/lib/python3.10/site-packages/magnum/service/periodic.py:67
2023-09-22 21:07:21.257 7 DEBUG magnum.service.periodic [-] Status for cluster 9 updated to CREATE_IN_PROGRESS (CAPI Cluster status: Provisioning: Cluster kube-bpm9m is Provisioning. CAPI OpenstackCluster status reason: ) update_status /var/lib/kolla/venv/lib/python3.10/site-packages/magnum/service/periodic.py:84

Maybe the used Cluster API & CAPO versions are too new?
CAPI_VERSION="v1.5.1"
CAPO_VERSION="v0.8.0"
clusterctl init \
  --core cluster-api:${CAPI_VERSION} \
  --bootstrap kubeadm:${CAPI_VERSION} \
  --control-plane kubeadm:${CAPI_VERSION} \
  --infrastructure openstack:${CAPO_VERSION}

@berendt I noticed that 1.5 is failing to provision new cluster, I haven't gotten around the root cause, I think it might be fixed in the recent commits.  1.4.x works fine, but you'll find out about 1.5 here
https://github.com/vexxhost/magnum-cluster-api/pull/216

I haven't had time to continue yet. I will probably have time for the topic on Thursday.

@mnaser Is it possible to do the release today? We want to do some tests tomorrow and would be helpful to be able to work with the bug fix.

Wrong image ID. It works with a correct image ID:
Request to create cluster template k8s-v1.27.5 accepted
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| insecure_registry     | -                                    |
| labels                | {'kube_tag': 'v1.27.5'}              |
| updated_at            | -                                    |
| floating_ip_enabled   | True                                 |
| fixed_subnet          | -                                    |
| master_flavor_id      | SCS-2V-4-10                          |
| uuid                  | df0e74af-6938-40a6-9062-2f9de3c316da |
| no_proxy              | -                                    |
| https_proxy           | -                                    |
| tls_disabled          | False                                |
| keypair_id            | -                                    |
| public                | False                                |
| http_proxy            | -                                    |
| docker_volume_size    | -                                    |
| server_type           | vm                                   |
| external_network_id   | public                               |
| cluster_distro        | ubuntu                               |
| image_id              | 929b90b6-202a-4651-b3d2-2b70e39d592f |
| volume_driver         | -                                    |
| registry_enabled      | False                                |
| docker_storage_driver | overlay2                             |
| apiserver_port        | -                                    |
| name                  | k8s-v1.27.5                          |
| created_at            | 2023-09-22T20:16:16+00:00            |
| network_driver        | calico                               |
| fixed_network         | -                                    |
| coe                   | kubernetes                           |
| flavor_id             | SCS-2V-4-10                          |
| master_lb_enabled     | True                                 |
| dns_nameserver        | <IP_ADDRESS>                              |
| hidden                | False                                |
| tags                  | -                                    |
+-----------------------+--------------------------------------+

I started with the integration in Kolla-Ansible: https://review.opendev.org/c/openstack/kolla-ansible/+/896292.
With this patch and installed magnum-cluster-api plugin inside the magnum container images I have this error when trying to create the COE:
Cluster type (vm, Unset, kubernetes) not supported (HTTP 400) (Request-ID: req-71b9046a-61d2-498d-b0f4-243ce729f9cf)

I encountered this problem too. Can you share me solution for it.
Thank you,

Hello. my problem happened because magnum capi using image_id instead image name when we create cluster template.

I added more details how make Magnum CAPI to work with Openstack
Requirements:

Openstack Cluster was deployed by Kolla-Ansbile# I tested on Openstack Yoga
Octavia must work.
Cinder(optional)

Install K8S cluster management
We setup a k8s cluster to manage workload clusters which were created by Magnum.
We need get /root/.kube/config from cluster management.
Magnum
We need log in(as root) to magnum api and magnum conductor and install.
docker exec -u root 'magnum container' /bin/bash
apt update
pip install magnum-cluster-api
wget https://get.helm.sh/helm-v3.13.2-linux-amd64.tar.gz # We can choose other vervion
tar -zxvf helm-v3.13.2-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/

We need helm to make this work. Because There is no helm in magnum containers.
Copy  /root/.kube/config from cluster management to magnum container under path  /var/lib/magnum/.kube/config

I tested on runtime container. If you want to build images. You must do above steps when use kolla build. You need read followed link too.
https://review.opendev.org/c/openstack/kolla-ansible/+/896292

Feel free to ask me

The progress to improve the Kolla support is in place:

https://review.opendev.org/c/openstack/kolla/+/902101 to add it to the images
https://review.opendev.org/c/openstack/kolla-ansible/+/896292 to allow injecting kubeconfig

The progress to improve the Kolla support is in place:

https://review.opendev.org/c/openstack/kolla/+/902101 to add it to the images
https://review.opendev.org/c/openstack/kolla-ansible/+/896292 to allow injecting kubeconfig

Hello. Would we add helm in container as default?

I think you're right, we are missing helm in there.

Regarding proxy service (nice work there btw.), because I assume this also should be considered here in case of kolla-ansible type installation. Do you have some documentation about how this should be done?

If there's anything needed from Kolla team (review patches, anything else) I'm happy to help.
the CAPI driver is in the Kolla built images for some time now - also Kolla-Ansible has handling for disabling CAPI driver when kubeconfig is not found.

Kolla-Ansible has handling for disabling CAPI driver when kubeconfig is not found.

My experience has been that this does not work as intended, and even though it marks the driver as disabled, Magnum calls into that part of the driver regardless. And it's not just me, either. See https://bugs.launchpad.net/kolla/+bug/2047360

Ah, ok - https://review.opendev.org/c/openstack/kolla-ansible/+/930414 - that should help - let me merge that and facilitate the backports.

Ah, ok - https://review.opendev.org/c/openstack/kolla-ansible/+/930414 - that should help - let me merge that and facilitate the backports.

Thanks - any chance you'd be able to get the helm binary added to the magnum container images, as it is a dependency of this Cluster API driver? Currently we have to build the images locally using kolla-build with a template-overrides.j2 containing the following:
{% extends parent_template %}

{% block magnum_base_footer %}
RUN curl -L https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz -o helm-v3.16.2-linux-amd64.tar.gz
RUN tar -xzvf helm-v3.16.2-linux-amd64.tar.gz
RUN mv linux-amd64/helm /usr/local/bin/helm
{% endblock %}

Ah, ok - https://review.opendev.org/c/openstack/kolla-ansible/+/930414 - that should help - let me merge that and facilitate the backports.

Thanks - any chance you'd be able to get the helm binary added to the magnum container images, as it is a dependency of this Cluster API driver? Currently we have to build the images locally using kolla-build with a template-overrides.j2 containing the following:
{% extends parent_template %}

{% block magnum_base_footer %}
RUN curl -L https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz -o helm-v3.16.2-linux-amd64.tar.gz
RUN tar -xzvf helm-v3.16.2-linux-amd64.tar.gz
RUN mv linux-amd64/helm /usr/local/bin/helm
{% endblock %}

Working on that in https://review.opendev.org/c/openstack/kolla/+/937619


[Enhancement] set grace period for mv checker

Why I'm doing:

Active every 1 minute would print too much log

What I'm doing:

Add grace period for mv checker, avoid active the mv too frequently

Fixes #issue
What type of PR is this:

[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[ ] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.2
[x] 3.1
[x] 3.0
[ ] 2.5

@Mergifyio rebase


[community] remap antrea.io/live & antrea.io/notes

Describe the bug
antrea.io/live points to a google doc, but @yashbhutwala had a great idea to use a hackmd.io file.

please remap antrea.io/notes to https://hackmd.io/@jayunit100/antrea-LIVE/edit
please remap antrea.io/live to
https://github.com/jayunit100/k8sprototypes/blob/master/antrea-LIVE/README.md

(or if theres a way i can PR / edit this just let me know ill do it)
thanks !
Expected
Antrea live links are easy to read and access/interact with.

@jayunit100 please try to manage with a single link (antrea.io/live) or if you really need 2 links use antrea.io/livenotes for the second one.
I'm transferring the issue to the website repo. You can update a PR directly there to configure the redirections. Look at https://github.com/antrea-io/website/pull/12 for reference.


feat: empty-block

Example:
class VxLan(PartialGenerator):
    def acl_huawei(self, device):
          return fr"""
              evpn
                  mac-duplication
                      {self.block_holder}
          """

    def run_huawei(self, device):
        with self.block("evpn"):
            with self.block("mac-duplication"):
                yield self.block_holder

Patch:
evpn
  mac-duplication
    quit
  quit

Without this feature, patch is invalid:
evpn
  mac-duplication
  quit

Is there any particular reason not to make this behaviour for the empty blocks such as
with self.block("mac-duplication"):
    pass

I would argue that it's a more natural way. If it's done on purpose, could you please add an explicit test for these cases in test_generators.


Test failures

Hi,
There are some test failures:
FFF..............

Failures:

  1) TTY::Which#which with extension handles path with executable file C:\Program Files\Git\bin\git
     Failure/Error: expect(Which.which(path_with_exe_file)).to eq(expected_path)

     RuntimeError:
       CRITICAL: RUBYGEMS_ACTIVATION_MONITOR.owned?: before false -> after true
     # ./spec/unit/which_spec.rb:86:in `block (3 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # NameError:
     #   uninitialized constant RSpec::Support::Differ
     #   ./spec/unit/which_spec.rb:86:in `block (3 levels) in <top (required)>'

  2) TTY::Which#which with extension searches path for executable git
     Failure/Error: expect(Which.which(cmd)).to eq(expected_path)

     NameError:
       uninitialized constant RSpec::Support::Differ
     # ./spec/unit/which_spec.rb:122:in `block (3 levels) in <top (required)>'

  3) TTY::Which#which with extension searches path for executable git.exe
     Failure/Error: expect(Which.which(cmd)).to eq(expected_path)

     NameError:
       uninitialized constant RSpec::Support::Differ
     # ./spec/unit/which_spec.rb:103:in `block (3 levels) in <top (required)>'

Top 2 slowest examples (0.01244 seconds, 36.8% of total time):
  TTY::Which#which with extension handles path with executable file C:\Program Files\Git\bin\git
    0.00952 seconds ./spec/unit/which_spec.rb:72
  TTY::Which#exist? fails to find executable in the path
    0.00292 seconds ./spec/unit/exist_spec.rb:11

Top 2 slowest example groups:
  TTY::Which#which
    0.00319 seconds average (0.02234 seconds / 7 examples) ./spec/unit/which_spec.rb:3
  TTY::Which#exist?
    0.00185 seconds average (0.0037 seconds / 2 examples) ./spec/unit/exist_spec.rb:3

Finished in 0.03384 seconds (files took 0.1605 seconds to load)
17 examples, 3 failures

Failed examples:

rspec ./spec/unit/which_spec.rb:72 # TTY::Which#which with extension handles path with executable file C:\Program Files\Git\bin\git
rspec ./spec/unit/which_spec.rb:107 # TTY::Which#which with extension searches path for executable git
rspec ./spec/unit/which_spec.rb:89 # TTY::Which#which with extension searches path for executable git.exe

Randomized with seed 58345

Any clue about the fix?

Hi Utkarsh,
Thanks for using the gem. Looking at the output, I can see that the failure is not caused by the tty-which gem. The  uninitialized constant RSpec::Support::Differ means that you have an unsupported version of RSpec. Try upgrading to at least version 3.

Hi @piotrmurach,
I'm running into this as well, and don't understand your recommendation: The version of rspec in use is 3.9.0, as shipped in Debian unstable.
Any further idea?
Thanks!

Hi @ge-fa,
To run specs you need to preload all dependencies, the easiest and most common way in Ruby world is to use bundler.
So using bundler and rake run the following:
bundle exec rake spec

Alternatively, you can skip using rake and run rspec directly on the test directory:
bundle exec rspec spec/

Thanks, I'm aware of the workings of bundler. This issue is specifically about Debian, and bundler isn't used to package stuff. I think it's related to the recent switch in Debian making Ruby 2.7 the default, however, I'm unsure about the specific reason.
Note: This issue is not only related to this lib, it poped up elsewhere as well, see for example this issue.

I run the test suite on Ruby 2.6.5 using only rspec executable and all the tests pass fine. On Ruby 2.7.0 running the same tests fails as demonstrated in the bug report. When I dug deeper into one of the tests I could see that mocking a Ruby core class in RSpec:
allow(::File).to receive(:join).with(...).and_return(...)

fails due to arguments to with being actual the current test file location and rspec matcher object which is completely wrong and totally weird. The expected arguments are never received and hence the spec fails. I could replace the test with
allow(::File).to receive(:join).and_return(...)

This makes the test pass. But what is the point of a test that doesn't test the implementation? I feel as though here we're working to fix how rspec works on Ruby 2.7.0 and 2.7.1. Again, running via bundler loads files correctly and hence makes all the tests pass.
When you look closely at the error message:
RuntimeError:
       CRITICAL: RUBYGEMS_ACTIVATION_MONITOR.owned?: before false -> after true

You can see that's something to do with Rubygems threading issues and thus failing to load files correctly on Ruby 2.7.x. Hence the RSpec::Support::Differ missing - which indicates that rspec-support is not loaded correctly.
Bundler is a default gem packaged with Ruby. This issue doesn't affect how the test suite works in my current workflow that uses bundler. The tests are here to support releases. I'm in two minds about changing tests and making them less useful to work around a bug in Ruby 2.7 and Rubygems integration.

Thanks for your follow-up and your findings, I appreciate this a lot. I'll try to find the cause and see what we're able to do in Debian about this.

Thanks for making me dig a little bit deeper into this problem. I'm not necessarily sure what the core problem is here but everything seems to point to some issue with how rubygems loads dependencies in Ruby 2.7.x. I checked some of my other tty projects by running rspec without the bundler against an entire test suite and they don't 'suffer' from this issue. I'd be curious to know more if you manage to find the cause. Maybe there is something in tty-which setup that triggers this issue. It may be a good port of call to report this on the rubygems project to see if they have some ideas.

ACK, thanks, I'll let you know, in case I'm able to find anything valuable.

(forwarded this issue to the rubygems folk!) :smile:
Also, similar issues:

https://github.com/chefspec/chefspec/issues/766
https://github.com/Sutto/rocket_pants/issues/114

@utkarsh2102 Could you provide the link to the rubygems issue?

@utkarsh2102 The chefspec issue is also dealing with mocking a core Ruby File class. The allow(File).to receive(...).with(...) fails in this instance as well. The mocking somehow 'leaks out' from the test and mocks File class inside RSpec leading to failure.

@ge-fa,
I've forwarded this issue link to David, he'll take a look. And I'll write back as soon as I hear from him :)

@utkarsh2102 Thanks!

Hello!! :wave: :wave:
This is quite convoluted :laughing:.
So, this is the backtrace I get printed together with the "CRITICAL: RUBYGEMS_ACTIVATION_MONITOR.owned?: before false -> after true" error:
#<Thread:0x0000561f76d66de8 run>
uninitialized constant RSpec::Support::Differ
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/error_generator.rb:305:in `differ'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/error_generator.rb:289:in `diff_message'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/error_generator.rb:275:in `error_message'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/error_generator.rb:62:in `raise_missing_default_stub_error'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/proxy.rb:225:in `raise_missing_default_stub_error'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/proxy.rb:210:in `message_received'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/proxy.rb:345:in `message_received'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/method_double.rb:77:in `proxy_method_invoked'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/verifying_proxy.rb:161:in `proxy_method_invoked'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/method_double.rb:64:in `block (2 levels) in define_proxy_method'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:59:in `block (2 levels) in require'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:57:in `each'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:57:in `block in require'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:46:in `each'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:46:in `require'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/error_generator.rb:305:in `differ'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/error_generator.rb:289:in `diff_message'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/error_generator.rb:275:in `error_message'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/error_generator.rb:62:in `raise_missing_default_stub_error'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/proxy.rb:225:in `raise_missing_default_stub_error'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/proxy.rb:210:in `message_received'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/proxy.rb:345:in `message_received'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/method_double.rb:77:in `proxy_method_invoked'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/verifying_proxy.rb:161:in `proxy_method_invoked'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/method_double.rb:64:in `block (2 levels) in define_proxy_method'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:59:in `block (2 levels) in require'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:57:in `each'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:57:in `block in require'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:46:in `each'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:46:in `require'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-expectations-3.9.2/lib/rspec/matchers.rb:573:in `eql'
/home/deivid/Code/tty-which/spec/unit/executable_file_spec.rb:17:in `block (2 levels) in <top (required)>'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/example.rb:257:in `instance_exec'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/example.rb:257:in `block in run'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/example.rb:503:in `block in with_around_and_singleton_context_hooks'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/example.rb:460:in `block in with_around_example_hooks'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/hooks.rb:481:in `block in run'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/hooks.rb:619:in `run_around_example_hooks_for'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/hooks.rb:481:in `run'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/example.rb:460:in `with_around_example_hooks'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/example.rb:503:in `with_around_and_singleton_context_hooks'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/example.rb:254:in `run'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/example_group.rb:644:in `block in run_examples'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/example_group.rb:640:in `map'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/example_group.rb:640:in `run_examples'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/example_group.rb:606:in `run'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/runner.rb:121:in `block (3 levels) in run_specs'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/runner.rb:121:in `map'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/runner.rb:121:in `block (2 levels) in run_specs'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/configuration.rb:2058:in `with_suite_hooks'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/runner.rb:116:in `block in run_specs'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/reporter.rb:74:in `report'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/runner.rb:115:in `run_specs'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/runner.rb:89:in `run'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/runner.rb:71:in `run'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/runner.rb:45:in `invoke'
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/exe/rspec:4:in `<top (required)>'
/home/deivid/.rbenv/versions/2.7.1/bin/rspec:23:in `load'
/home/deivid/.rbenv/versions/2.7.1/bin/rspec:23:in `<main>'

If we observe the backtrace bottom-up, we see the following stuff:

First important line is

/home/deivid/Code/tty-which/spec/unit/executable_file_spec.rb:17:in `block (2 levels) in <top (required)>

This is the line inside specs were the error happens.

Next two lines also give us useful information:

/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-expectations-3.9.2/lib/rspec/matchers.rb:573:in `eql'"
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:46:in `require'

This indicates that we immediately go into rubygems internal require code, because RSpec matchers are autoloaded (lazily required). While doing its thing, rubygems ends up calling File.file?, causing the problem. This explains why bundler does not suffer from the issue, since it does not monkeypatch require, and thus it doesn't end up calling File.file?.

When File.file? gets called, rspec mocking code gets run, and a few lines above we can see something interesting:

/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib/rspec/mocks/proxy.rb:225:in `raise_missing_default_stub_error'

RSpec is detecting that File.file? is being used with a value different to the one being mocked, and it's about to give an error that reads like this:
#<File (class)> received :file? with unexpected arguments
expected: ("ruby")
got: ("/home/deivid/Code/tty-which/spec/rspec/matchers/built_in/eql")
Please stub a default value first if message might be received with other args as well.

This shows a potential resolution for this issue, probably the one that makes more sense: allow(File).to receive(:exist?).and_call_original, as suggested in https://github.com/chefspec/chefspec/issues/766#issuecomment-396631456.

But the error does not end up being displayed (to see it I manually added RSpec::Support.require_rspec_support "differ" on top of the error_generator.rb file). The problem is that the constant RSpec::Support::Differ, needed to show the error is also autoloaded, so we go to rubygems require again, call File.file? again, and so on. Rubygems detects this situation and fails.

So, maybe there's an issue in rspec, because it may be argued that RSpec should "undo" any mockings temporarily to run its own code (or any dependencies like rubygems in this case).
But probably the easiest way out is to use allow(File).to receive(:exist?).and_call_original before mocking calls with specific arguments?
The other failures should be similar.
Hope it helps!

@deivid-rodriguez Thank you for the quick reply and your investigation!
I have reservations about concluding that rubygems is not at fault here. I'm not saying it is but I also cannot see evidence that it isn't. I think this is a bug indicative of some significant regression someplace and it may be that it is manifested in RSpec failure but the root cause is someplace else.
Let's look at the provided example snippet and the comment:

RSpec is detecting that File.file? is being used with a value different to the one being mocked, and it's about to give an error that reads like this:

#<File (class)> received :file? with unexpected arguments
expected: ("ruby")
got: ("/home/deivid/Code/tty-which/spec/rspec/matchers/built_in/eql")
Please stub a default value first if message might be received with other args as well.

The method definition is as simple as delegating passed value:
def executable_file?(filename)
  ::File.file?(filename) && ::File.executable?(filename)
end

and the corresponding test looks like this:
allow(::File).to receive(:file?).with("ruby").and_return(true)
allow(::File).to receive(:executable?).with("ruby").and_return(true)

expect(TTY::Which.executable_file?("ruby")).to eql(true)

Can you please explain to me how ""/home/deivid/Code/tty-which/spec/rspec/matchers/built_in/eql"" ends up being a value passed into the ::File.file?? This doesn't make much sense. Where is this value from? Why RSpec matcher eql is used as an argument at all?
Using theallow(File).to receive(:exist?).and_call_original is not a solution. It doesn't solve the problem but 'papers over'  the bug. This is equivalent to mocking any exist? call with any arguments like so:
allow(File).to receive(:exist?)

Which matches the automated suggestion from the failure "Please stub a default value first if message might be received with other args as well. " But you shouldn't have to do it. The below passes:
# allow(::File).to receive(:file?).with("ruby").and_return(true)
# replaced with
allow(::File).to receive(:file?).with(any_args).and_return(true)
allow(::File).to receive(:executable?).with("ruby").and_return(true)

expect(TTY::Which.executable_file?("ruby")).to eql(true)

But we haven't solved the problem and made the test 'weaker' as it no longer tests that filename argument is passed correctly.
My other question is that, if this is rspec issue why does it work fine in other Rubies apart from Ruby 2.7.x? If this was autoloading bug I should've come across is much earlier since I've been using the allow(File).to receive(:file?).with(...).and_return(...) for many years without any issues?

Can you please explain to me how ""/home/deivid/Code/tty-which/spec/rspec/matchers/built_in/eql"" ends up being a value passed into the ::File.file?? This doesn't make much sense. Where is this value from? Why RSpec matcher eql is used as an argument at all?

It's rubygems making that call, it's not your code.

I honestly think there's no good solution here. Even if rubygems wouldn't monkeypatch require, end users mocking stdlib methods that RSpec uses internally would lead to a similar thing.

My other question is that, if this is rspec issue why does it work fine in other Rubies apart from Ruby 2.7.x? If this was autoloading bug I should've come across is much earlier since I've been using the allow(File).to receive(:file?).with(...).and_return(...) for many years without any issues?

I'm not saying that this is an RSpec issue, or that you should change your code, I'm just trying to explain the situation. The problem is actually not dependent on ruby 2.7, but on the rubygems version that you are running (which is higher on ruby 2.7). It also depends on the different gems you have installed on your system, since that triggers different code paths in rubygems that might or might not call File.file?. For example, using rubygems master version, and cleaning up my installed gems, the bug doesn't trigger for me anymore.

Hi @JonRowe - do you have any experience with the above? Does anything jump out for you as the cause of the issue? It's been suggested that rspec and its autoloading of dependencies may be a problem here.

If there are different versions of files being loaded its because the host system has more than one version of them installed and bundler is not being used, autoload and require being core Ruby methods (with or without rubygems) they depend on your path, so when you are not using bundler you are responsible for ensuring you have the correct version of gems in your path for your code.
Bundler solves many of these problems by manipulating the path to be correct for the current installed versions of dependencies. Without it you must do this manually. No other package manager is currently aware of Ruby dependencies to my knowledge.

Just to clarify my suggestion, I meant to add
allow(::File).to receive(:file?).and_call_original

on top of the other mocks, not instead of them, so in case other code external to your library, be it RSpec or rubygems, ends up calling File.file?, the original implementation gets called.
And I also meant it as a patch to be added by Debian to workaround the unfortunate situation, not as something to be changed upstream. The current implementation of the test looks just fine to me.

So, maybe there's an issue in rspec, because it may be argued that RSpec should "undo" any mockings temporarily to run its own code (or any dependencies like rubygems in this case).

Also, this was just a quick thought, but it sounds like something quite hard to implement, for little benefit.

If there are different versions of files being loaded its because the host system has more than one version of them installed and bundler is not being used, autoload and require being core Ruby methods (with or without rubygems) they depend on your path, so when you are not using bundler you are responsible for ensuring you have the correct version of gems in your path for your code.

Actually,  rubygems require has some basic support for this, that's why you can run require "rspec" in IRB and get the latest version of rspec installed on your system activated, together with its dependencies, without it being in the $LOAD_PATH.
irb(main):001:0> puts $LOAD_PATH
/home/deivid/.rbenv/rbenv.d/exec/gem-rehash
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/x86_64-linux
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/vendor_ruby/2.7.0
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/vendor_ruby/2.7.0/x86_64-linux
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/vendor_ruby
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/2.7.0
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/2.7.0/x86_64-linux
=> nil
irb(main):002:0> require 'rspec'
=> true
irb(main):003:0> puts $LOAD_PATH
/home/deivid/.rbenv/rbenv.d/exec/gem-rehash
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/diff-lcs-1.4.4/lib
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-mocks-3.9.1/lib
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-3.9.0/lib
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-support-3.9.3/lib
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/x86_64-linux
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/site_ruby
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/vendor_ruby/2.7.0
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/vendor_ruby/2.7.0/x86_64-linux
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/vendor_ruby
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/2.7.0
/home/deivid/.rbenv/versions/2.7.1/lib/ruby/2.7.0/x86_64-linux
=> nil

But unfortunately, it uses File.file? in several places to implement this.

@deivid-rodriguez Thanks a lot for explaining this. It makes a lot more sense now. I see also why you're suggesting to use:
allow(::File).to receive(:file?).and_call_original.
@JonRowe Would it make sense and is it even possible for rspec to limit its mock checks on things like allow(File).to receive(:method).with(...).and_return(...) to only file paths that relate to the namespace currently tested?

@deivid-rodriguez I read your first comment with all the backtraces in more detail and they make perfect sense now. It seems like mocking File.join and File.file? are best to be done cautiously in the future as they are used here: https://github.com/rubygems/rubygems/blob/master/lib/rubygems/core_ext/kernel_require.rb#L59-L60
It seems like rspec could/should call allow(File).to receive(:method).and_call_original by default when allow(File).to receive(:method).with(...).and_return(...) is invoked. That's probably the easiest and surest way to prevent this from happening again? I will add the and_call_original to the tests to ensure this issue doesn't come back again.

@deivid-rodriguez I read your first comment with all the backtraces in more detail and they make perfect sense now. It seems like mocking File.join and File.file? are best to be done cautiously in the future as they are used here: https://github.com/rubygems/rubygems/blob/master/lib/rubygems/core_ext/kernel_require.rb#L59-L60

Great! This should be a rare case, since it needs specs not be run with bundler, which most people do. But the number of linked tickets suprised me...

It seems like rspec could/should call allow(File).to receive(:method).and_call_original by default when allow(File).to receive(:method).with(...).and_return(...) is invoked. That's probably the easiest and surest way to prevent this from happening again?

But then users would miss actual unintended calls with other arguments to File.file?, right? I think RSpec is ok here. The only thing I believe RSpec could do is check whether the immediate caller location of a method double is a file inside RbConfig::CONFIG["rubylibdir"], RbConfig::CONFIG["rubysitedir"] or another "system file", and in that case use the original implementation. But I understand if they don't want to do it.

Actually, rubygems require has some basic support for this, that's why you can run require "rspec" in IRB and get the latest version of rspec installed on your system activated, together with its dependencies, without it being in the $LOAD_PATH.

Rubygems also manipulates your path... its also intertwined with bundler these days. There are still issues if it is not limited to the current Gemfile's dependencies. Using gems from linux package manages is known to be generally problematic.

Would it make sense and is it even possible for rspec to limit its mock checks on things like allow(File).to receive(:method).with(...).and_return(...) to only file paths that relate to the namespace currently tested?

No it is not possible, we don't know wether the code is intentionally called or unintentionally called. Stubs only exist for the lifecycle of your test, they are cleaned up automatically so they are directly or indirectly invoked by you.

It seems like rspec could/should call allow(File).to receive(:method).and_call_original by default when allow(File).to receive(:method).with(...).and_return(...) is invoked. That's probably the easiest and surest way to prevent this from happening again? I will add the and_call_original to the tests to ensure this issue doesn't come back again.

If you stub a core method you are responsible for restoring any behaviour you depend upon elsewhere in your tests. This can be as simple as:
allow(File).to receive(:method).and_call_original
expect(File).to receive(:method).and_return(value)

It is an impossible task to ask RSpec to special case all of the core library to ensure some special behaviour.

The only thing I believe RSpec could do is check whether the immediate caller location of a method double is a file inside RbConfig::CONFIG["rubylibdir"], RbConfig::CONFIG["rubysitedir"] or another "system file", and in that case use the original implementation. But I understand if they don't want to do it.

This is an interesting idea but in practise I don't think it would work, we only know wether code is in the current working directory, or not, so much core Ruby behaviour is now gem based we wouldn't be able to distinguish between core Ruby libraries and other gems, and most people stub gems not Ruby features.

This is an interesting idea but in practise I don't think it would work, we only know wether code is in the current working directory, or not, so much core Ruby behaviour is now gem based we wouldn't be able to distinguish between core Ruby libraries and other gems, and most people stub gems not Ruby features.

Yeah, even if the first caller_location was specially guarded to be a rubygems file, that would break people intentionally stubbing stuff in rubygems for some reason. And feels not right anyways.
I don't think there's anything actionable here other than people adding their workaround of choice like allow(File).to receive(:method).and_call_original, or eagerly loading the RSpec parts that need to be early loaded. Or if @piotrmurach wants to support running specs in this repo without Bundler, then he can also add allow(File).to receive(:method).and_call_original to the specs giving trouble.

Rubygems also manipulates your path... its also intertwined with bundler these days.

I don't think rubygems changes your PATH in any way, I'm not even sure if it uses it at all (definitely not for a lot of things). It's your ruby version manager making sure rspec is found at the proper path. And if running rspec from a Debian-installed gem, I assume it's already in your PATH. Not sure how it relates to this issue, but I think you're trying to say that you don't want to change anything in RSpec to support this case. That's fine to me :+1:.

There are still issues if it is not limited to the current Gemfile's dependencies. Using gems from linux package managers is known to be generally problematic.

Indeed, but it's getting better. The Debian ruby team has packaged hundreds of gems which are used by people. And they seem to work :laughing:. I think it's a great way to make general purpose ruby tools more widely available and I really appreciate the work they do :heart:.

I don't think rubygems changes your PATH in any way, I'm not even sure if it uses it at all (definitely not for a lot of things). It's your ruby version manager making sure rspec is found at the proper path. And if running rspec from a Debian-installed gem, I assume it's already in your PATH.

It's not just your ruby version manager doing it, its the entire system, at the end of the day the load path has usually been touched by several things and has an order. The first file matching your require in the load path will be the one that is loaded.

Not sure how it relates to this issue,

I was attempting (badly it seems) to explain why File was seeing different paths like /home/deivid/Code/tty-which/spec/rspec/matchers/built_in/eql being checked, its Ruby searching the load path for the file.

Sorry, I understand that you said "path" where you meant "load path". In that case, sure, I already mentioned this in previous comments.

I was attempting (badly it seems) to explain why File was seeing different paths like /home/deivid/Code/tty-which/spec/rspec/matchers/built_in/eql being checked, its Ruby searching the load path for the file.

I think we all got why this happens at this point :+1:

@deivid-rodriguez I decided to fix the failures to help make Debian releases easier. Thanks again for investigating and explaining the issue in such detail, that was mega useful.
@JonRowe Thanks for your perspective, I ensured that the original version of the mocked call is executed.

That's really kind of you @piotrmurach :purple_heart:!


Update rpc

Codecov Report

Merging #324 into develop will decrease coverage by 1.5%.
The diff coverage is 78.57%.

@@             Coverage Diff             @@
##           develop     #324      +/-   ##
===========================================
- Coverage    97.28%   95.78%   -1.51%
===========================================
  Files            9        9
  Lines          332      356      +24
  Branches       109      123      +14
===========================================
+ Hits           323      341      +18
- Misses           4       11       +7
+ Partials         5        4       -1

Impacted Files
Coverage Δ

packages/ckb-sdk-rpc/src/paramsFormatter.ts
84.61% <75%> (-11.39%)
:arrow_down:

packages/ckb-sdk-rpc/src/resultFormatter.ts
95.09% <83.33%> (-0.56%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ddff674...f9577aa. Read the comment docs.

Please have a look at the workflow of sending transactions in the example https://github.com/nervosnetwork/ckb-sdk-js/blob/8346ae422053e5ecc530be5ad465d048f759af28/packages/ckb-sdk-core/examples/sendTransaction.js#L203 @shaojunda

Updated the secp256k1 dep group for sending transactions

Add computeScriptHash RPC method and update the example

Hold for implementing the serialization

TODO: squash the commits


Chrome history still show the page being blocked

When navigating to to a blocked URL the url is blocked but still showing in chrome history log

@spacentime
thanks for your comments :)
do you said chrome history log, it is chrome://history ?
when blocked why you want to hide on chrome history log ?


New version: Recoupler.AudioBookConverter version 6.2.9

Pull request has been created with WinGet Automation :rocket:
Microsoft Reviewers: Open in CodeFlow

Manifest Error: Duplicate installer entry found.


Pop!_OS update (24.04 LTS)

Just to make sure that the script will run smoothly on the new upcoming 24.04 LTS release of Pop!_OS

First try running the script on 24.04 alpha

Creation of a new branch of the project
https://github.com/geraldohomero/post-install-pop-os/tree/24.04-test

Second time, on flatpaks:

#57 Update post-install.sh - remove gnome-tweaks and MEGA file manager


Wowza url signing

This is a feature to sign urls with Wowza.
To configure this, use the file configuration org.opencastproject.security.urlsigning.provider.impl.WowzaUrlSigningProvider.cfg
Each property has the form: key.keyId.propertyName
key.wowzatoken.organization is optional. By default is *, that means any organization.
key.wowzatoken.url is mandatory. describes the url prefix that urls must have.
key.wowzatoken.secret=myTokenPrefix:mySharedSecret is mandatory. It is a pair separated by :. First value, myTokenPrefix, means the prefix that all parameters of signed urls will have. Second value, mySharedSecret, defines a secret value, to secure urls.

So what's up with that? The code does look pretty similar.

@jchssystems ping, this has been open with questions for quite a while now.

Between the potential copyright issues here, and the lack of progress/response we're going to close this.


Graphical interface for catagory theory proofs

https://ncatlab.org/nlab/show/Globular
https://golem.ph.utexas.edu/category/2015/12/globular.html

http://opetopic.net
;-)

This paper is the original string diagrams paper where they establish that any topological deformation of a diagram represents the same morphism in a Monoidal category.
To do this they have to make precise what a string diagram is in a very explicit way so that could be helpful for designing the interface.


Fix target result channel close race condition

When running tests I noticed:
==================
WARNING: DATA RACE
Write at 0x00c000255348 by goroutine 95:
github.com/linuxboot/contest/pkg/runner.(*TestRunner).checkStepRunnersLocked()
/go/src/github.com/linuxboot/contest/pkg/runner/test_runner.go:737 +0x277
github.com/linuxboot/contest/pkg/runner.(*TestRunner).runMonitor()
/go/src/github.com/linuxboot/contest/pkg/runner/test_runner.go:788 +0x5ae
github.com/linuxboot/contest/pkg/runner.(*TestRunner).run()
/go/src/github.com/linuxboot/contest/pkg/runner/test_runner.go:226 +0xce4
github.com/linuxboot/contest/pkg/runner.(*TestRunner).Run()
/go/src/github.com/linuxboot/contest/pkg/runner/test_runner.go:134 +0x613
github.com/linuxboot/contest/pkg/runner.(*JobRunner).runTest()
/go/src/github.com/linuxboot/contest/pkg/runner/job_runner.go:378 +0xfd2
github.com/linuxboot/contest/pkg/runner.(*JobRunner).Run()
/go/src/github.com/linuxboot/contest/pkg/runner/job_runner.go:189 +0x93a
github.com/linuxboot/contest/pkg/jobmanager.(*JobManager).runJob()
/go/src/github.com/linuxboot/contest/pkg/jobmanager/start.go:98 +0x36d
Previous read at 0x00c000255348 by goroutine 91:
github.com/linuxboot/contest/pkg/runner.(*TestRunner).awaitTargetResult()
/go/src/github.com/linuxboot/contest/pkg/runner/test_runner.go:419 +0x5b
github.com/linuxboot/contest/pkg/runner.(*TestRunner).targetHandler()
/go/src/github.com/linuxboot/contest/pkg/runner/test_runner.go:507 +0xbb2
github.com/linuxboot/contest/pkg/runner.(*TestRunner).run.func1()
/go/src/github.com/linuxboot/contest/pkg/runner/test_runner.go:220 +0x56
Goroutine 95 (running) created at:
github.com/linuxboot/contest/pkg/jobmanager.(*JobManager).startJob()
/go/src/github.com/linuxboot/contest/pkg/jobmanager/start.go:80 +0x384
github.com/linuxboot/contest/pkg/jobmanager.(*JobManager).resumeJob()
/go/src/github.com/linuxboot/contest/pkg/jobmanager/resume.go:72 +0xaec
github.com/linuxboot/contest/pkg/jobmanager.(*JobManager).resumeJobs()
/go/src/github.com/linuxboot/contest/pkg/jobmanager/resume.go:26 +0x2ea
github.com/linuxboot/contest/pkg/jobmanager.(*JobManager).Run()
/go/src/github.com/linuxboot/contest/pkg/jobmanager/jobmanager.go:154 +0xc44
github.com/linuxboot/contest/cmds/contest/server.Main()
/go/src/github.com/linuxboot/contest/cmds/contest/server/server.go:315 +0x141e
github.com/linuxboot/contest/tests/e2e.(*E2ETestSuite).startServer.func1()
/go/src/github.com/linuxboot/contest/tests/e2e/e2e_test.go:125 +0x1f3
Goroutine 91 (running) created at:
github.com/linuxboot/contest/pkg/runner.(*TestRunner).run()
/go/src/github.com/linuxboot/contest/pkg/runner/test_runner.go:219 +0x959
github.com/linuxboot/contest/pkg/runner.(*TestRunner).Run()
/go/src/github.com/linuxboot/contest/pkg/runner/test_runner.go:134 +0x613
github.com/linuxboot/contest/pkg/runner.(*JobRunner).runTest()
/go/src/github.com/linuxboot/contest/pkg/runner/job_runner.go:378 +0xfd2
github.com/linuxboot/contest/pkg/runner.(*JobRunner).Run()
/go/src/github.com/linuxboot/contest/pkg/runner/job_runner.go:189 +0x93a
github.com/linuxboot/contest/pkg/jobmanager.(*JobManager).runJob()
/go/src/github.com/linuxboot/contest/pkg/jobmanager/start.go:98 +0x36d

Codecov Report

Merging #42 (ea9929b) into main (f6f482f) will decrease coverage by 0.03%.
The diff coverage is 57.14%.

@@            Coverage Diff             @@
##             main      #42      +/-   ##
==========================================
- Coverage   66.26%   66.22%   -0.04%
==========================================
  Files         155      155
  Lines        9268     9274       +6
==========================================
+ Hits         6141     6142       +1
- Misses       2475     2477       +2
- Partials      652      655       +3

Flag
Coverage Δ

e2e
50.34% <57.14%> (-0.01%)
:arrow_down:

integration
57.36% <57.14%> (-0.01%)
:arrow_down:

unittests
47.22% <57.14%> (+<0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/runner/test_runner.go
92.14% <57.14%> (-0.88%)
:arrow_down:

plugins/teststeps/waitport/waitport.go
68.93% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6f482f...ea9929b. Read the comment docs.


Update live with current master

Kicked off live branch build on OPS to see if the warnings go away.


auto-checking the latest version when install

CLI rg can be replaced by grep or other pattern search tool.

i've very intentionally hardcoded the version number in the install command; i want users to come to this document to get the updated install command, so they're not blindly receiving breaking changes.

Do you plan to change this in the future @ljharb? Say v1 gets released, will you still be hardcoding versions into the README?

@Zebiano the only time I’d maybe change that is when #400 is resolved, and possibly not even then.


chore(docs): update docs image

@scottire , i made the image shorter (closer to the height of the current image).  it is there to add a little visual interest to the page, but we don't want to push the cta and colab button below the fold.


Revert "Adds IngressClassName Validation (#388)"

This reverts commit 892c8c17bf0a066ea657b847bd65819f2d2e0117.
Fixes: https://github.com/projectcontour/contour-operator/issues/403

Friendly ping @skriss @youngnick  I am looking forward to this.

@skriss you might have to re-run the checks - I'm not able to do that

@dprotaso main is back to green, if you could rebase here we should be good to go

@skriss rebased

thanks @dprotaso!


Polish and cleanup in xi-trace

I know that @vlovich was already considering/planning some of these changes, but after actually using the tracing tools I wanted to note down some of my observations:

[ ] Add macros to wrap various tracing functions? currently even if tracing is disabled, the trace function is processed, which could include the allocation of a payload. This could be replaced with a macro that checks if tracing is enabled and noops if not. It's possible we would only want to do this for the _payload variants of the trace functions, since allocating payloads is the thing that could be potentially expensive in some cases.

[ ]  Add ability to specify process/thread names The chrome tracing format allows you to add certain special metadata traces, which can be used to associate a name with a process or thread. Adding support for this would make interpreting traces easier.

[ ] Settle on a payload type we'd talked about this at some point; the various payload types and related features is confusing to library consumers. My preference is once more for a serde_json::Value, since that's what chrome_trace will want, but a CowStr works too.

I have been in the process of cleaning it up a lot, just other things get
in the way.  Here are my thoughts on the points raised:
Adding macros: I’m not sure I understand this point.  While the function
jump happens, the first thing is the infrastructure checks an atomic
Boolean.  The overhead of this should be in the noise unless it somehow
ends up in a hot path loop, in which case your trace is probably in the
wrong spot anyway; the caveat is I could see this happening if you use the
infrastructure as a form of debugging/logging too.  For what it’s worth
memory isn’t allocated until tracing is enable (in Xi-core - I can’t recall
if I changed xi-mac to use that optimization).
Process/thread name.  Already working on it but it’s going to be
transparent to the API.  I’m hesistant to expose to much of the chrome
tracing API cause then it’s complex to use (too many options for the same
thing) but I am working on aligning the internal representation so it’s
possible to take advantage of things more easily that make sense.  I
wouldn’t be surprised if the spec has grown organically to encompass a wide
variety of uses specific to the needs of the many sub teams of chrome
rather than having any kind of opinionated overarching philosophy.
Payload type: I think here I would say we should pick what makes sense for
the developer using the API, not what the format wants because this is all
unstructured data that Chrome ignores.  It’s there for you only to dig in
extra to diagnose issues.  Also happy to remove it if there’s no use for it
since it borders on “this is a logging facility too” .
On Wed, Mar 28, 2018 at 9:05 AM Colin Rofls<EMAIL_ADDRESS>wrote:

I know that @vlovich https://github.com/vlovich was already
considering/planning some of these changes, but after actually using the
tracing tools I wanted to note down some of my observations:

Add macros to wrap various tracing functions? currently even if
tracing is disabled, the trace function is processed, which could include
the allocation of a payload. This could be replaced with a macro that
checks if tracing is enabled and noops if not. It's possible we would only
want to do this for the _payload variants of the trace functions,
since allocating payloads is the thing that could be potentially expensive
in some cases.
Add ability to specify process/thread names The chrome tracing
format allows you to add certain special metadata traces
https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview#bookmark=id.iycbnb4z7i9g,
which can be used to associate a name with a process or thread. Adding
support for this would make interpreting traces easier.
Settle on a payload type we'd talked about this at some point; the
various payload types and related features is confusing to library
consumers. My preference is once more for a serde_json::Value, since
that's what chrome_trace will want, but a CowStr works too.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/xi-editor/issues/581, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMSR1U47yVd2c0lHKSqGKKx_v8rLQ7Qks5ti7ThgaJpZM4S-7fl
.


I don't seem able to enable the "trace" feature

[dependencies]
nom-tracable = { git = "https://github.com/dalance/nom-tracable.git", features = ["trace"] }

I use this but I don't have any output. Am I missing something?
parser | histogram                                          | count
------ | -------------------------------------------------- | -----

parser | cumulative histogram                               | count
------ | -------------------------------------------------- | -----

I'm able to run the example in your git repo with cargo run --manifest-path=nom-tracable/Cargo.toml --example example --features trace.

Sorry. I missed this issue.
The dependencies seems to be correct.
Did you add #[tracable_parser] to your parser?

Sorry. I missed this issue.

No worries. Thanks for the reply.

Did you add #[tracable_parser] to your parser?

Yes. I have a minimal exemple. I'm probably doing something wrong.
[dependencies]
nom = "5.0"
nom-tracable = { git = "https://github.com/dalance/nom-tracable.git", features = ["trace"] }
nom_locate = "1.0.0"

use nom::bytes::complete::tag;
use nom::character::complete::digit1;
use nom::IResult;
use nom_locate::LocatedSpanEx;
use nom_tracable::{cumulative_histogram, histogram, tracable_parser, TracableInfo};

use std::io::Error;

type Span<'a> = LocatedSpanEx<&'a str, TracableInfo>;

#[tracable_parser]
pub fn parse(i: Span) -> IResult<Span, i32> {
    let (rest, _) = tag("test")(i)?;
    let (rest1, number) = digit1(rest)?;

    let number: i32 = number
        .fragment
        .parse()
        .map_err(|_e| nom::Err::Error((i, nom::error::ErrorKind::ParseTo)))?;

    Ok((rest1, number))
}

#[test]
fn test() -> Result<(), std::io::Error> {
    let info = TracableInfo::new().parser_width(64).fold("term");
    let (_rest, parsed) = parse(LocatedSpanEx::new_extra("test123", info)).unwrap();

    assert_eq!(parsed, 123);

    histogram();
    cumulative_histogram();

    assert!(false);

    Ok(())
}

❯ cargo test
   Compiling test-nom-trace v0.1.0 (/home/bbigras2/tmp/test-nom-trace)
[...]
    Finished dev [unoptimized + debuginfo] target(s) in 0.85s
     Running target/debug/deps/test_nom_trace-f4dcbabc0ed242d8

running 1 test
test test ... FAILED

failures:

---- test stdout ----

parser | histogram                                          | count
------ | -------------------------------------------------- | -----

parser | cumulative histogram                               | count
------ | -------------------------------------------------- | -----

thread 'test' panicked at 'assertion failed: false', src/lib.rs:34:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

This is documentation mistake.
trace feature must be included in the crate using nom-tracable.
Please try the following Cargo.toml and do cargo test --features trace.
[features]
default = []
trace   = ["nom-tracable/trace"]

[dependencies]
nom = "5.0"
nom-tracable = { git = "https://github.com/dalance/nom-tracable.git" }
nom_locate = "1.0.0"

It works. Thanks.


i have no idea what im doing

resolved all merge conflicts, waiting on @Bod9001 to review this and make sure I didn't accidentally yeet any of his code or any of the new changes breaks his current changes.


Use a single QueryEngine in the language server to share cache across large workspaces

I'm currently helping Fluid Protocol debug bad LSP performance on their workspace. The workspace had 20+ sway projects and 1 large "libraries" project that contain 20+ sway libraries.
When clicking through the various projects, I've noticed that the amount of RAM used can easily reach 20+ gig. It seems we are recompiling std / core and shared libraries for each project. So if we click through all 20 projects, we have compiled the std lib 20 times and store it in the servers memory. For example, if we put a print in the following query engine code
pub fn insert_programs_cache_entry(&self, entry: ProgramsCacheEntry) {
    eprintln!("👷 Instering programs cache entry: {:?}", entry.path);
    let mut cache = self.programs_cache.write().unwrap();
    cache.insert(entry.path.clone(), entry);
}

We get the below.

Instead, we should be having a single QueryEngine stored in ServerState that is used across each project. This way we will be able to drastically reduce the memory footprint for projects like this, and greatly speed up the time it takes for the server to initialise when a new project is navigated to.

related to #5645

Ok it's not as simple as just having the QueryEngine shared across workspace members. This will allow us to share the cached AST but the type and decl engines still need to be populated. What we probably want is to have the Engines and other items like the TokenMap stored in the ServerState rather than within each Session.
This is going to involve quite a bit or code re-arranging but should be feasible.


All  map not work in newest gvim(v8.2.5157)

Steps to reproduce

modify a file
press <C-s>

Expected behaviour
The key map
noremap <C-s> :w!<CR>

should be triggered, but it didn't.
Version of Vim
8.2.5157
Environment
Windows 10 64bit
gvim_8.2.5157_x64
minimal vimrc:
"Start (neo)vim with command: `vim -u min.vim`

set pyx=3
set langmenu=none
let $LANG = 'en'
syntax on
set nocompatible              " be iMproved, required
filetype off                  " required

" CTRL-X and SHIFT-Del are Cut
vnoremap <C-X> "+x
" CTRL-C and CTRL-Insert are Copy
vnoremap <C-C> "+y
" CTRL-V and SHIFT-Insert are Paste
noremap <C-V> "+gp
" CTRL-S are Save
noremap <C-s> :w!<CR>
vnoremap<C-r> :<C-U><C-R>=printf("%%s/%s/", expand("<cword>"))<CR>

Logs and stack traces
No response

Duplicate of #10615


Add component macro and example

The component macro helps to wrap the function body inside Cx::begin_loc and Cx::end to avoid key collision.

The name_of_cx function is very ugly but is simplest among the alternative I have tried out.


Ignore duplicates for IP guessing #1820

@juanignaciosl can you please review this one liner?
@iriberri, just FYI.

: +1:

:smile:


Arnold Threads

Added "ai:threads" to ArnoldOptions under the Performance section.
ArnoldRender uses ai:threads option to drive the kick -t command line arg.

I wasn't quite sure how to add tests for this. Maybe its alright without? The other ArnoldOptions don't seem to be tested anyways, and all current GafferArnoldTest and GafferArnoldUITests pass (at IE anyways).

Also, I wasn't sure if it was actually useful to add the -t option in Expand mode, but I did it anyways...

Nice one - thanks. I've done a manual merge with git rebase --whitespace=fix to fix the Codacy problem, and added a comment about the need for doing the whole -t N thing when you'd expect it to be unnecessary.


"Unexpected JSON token" when viewing execution history

The instance at saveourtool.com, deployed from 5776e4a3105d6f7da2c75c34f4bae670d04fd1a0:
Error when fetching data: Unexpected JSON token at offset 319: Encountered an unknown key 'contestName' at path: $[0].score Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys. JSON input: ....."kotlin-diktat","score":null,"contestName":null}]

Need to clean site's cache -- old version of frontend tries to communicate with new version of backend

Yes, this is the problem that your backend schema differs from frontend


Make segments optional on LineSets

If segments are not specified, vertices are implicitly connected in order.
This is equivalent to default segments value of [[0, 1], [1, 2], [2, 3], [3, 4]...] but saves storage space

Implementation notes:

In the code, this should be as simple as adding required=False as an additional argument to the segments property on linesets
The implicit default behaviour described in my previous comment also needs to be documented. Likely the best place for this is the docstring.


Add a function for points reprojection in projtools

If we don't, we should definitely have a small function for this type of operation in projtools !
Originally posted by @adehecq in https://github.com/GlacioHack/GeoUtils/pull/120#r594352768

See e.g. https://github.com/rhugonnet/GeoUtils/blob/a65d7c3ba2546d80e20c9535634d9f6e13d1e774/geoutils/georaster.py#L1287
And missing implementation here:
https://github.com/rhugonnet/GeoUtils/blob/a65d7c3ba2546d80e20c9535634d9f6e13d1e774/geoutils/georaster.py#L1083


subctl gather - discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+

When running in a deployment that has service discovery enabled, this warning is produced by the API:
Warning: discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
We should switch to using discovery.k8s.io/v1 once we drop support for K8s 1.20

What happened:
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:

Diagnose information (use subctl diagnose all):
Gather information (use subctl gather)
Cloud provider or hardware configuration:
Install tools:
Others:

This is a duplicate of https://github.com/submariner-io/lighthouse/issues/576.

No since this is about switching calls in subctl while the one you mentioned is about Lighthouse itself

No since this is about switching calls in subctl while the one you mentioned is about Lighthouse itself

Ah yes, there are separate references in subctl gather which aren’t transitive through Lighthouse. In any case
Depends on https://github.com/submariner-io/lighthouse/issues/576


Issue No #5

Description
For Text and URL I have made a common path so that it is choice of user to put text or URL once at a time
Issue Resolved
Fixes #5
Checklist
Please make sure to review the following before submitting your PR:

[ x] I have read the contribution guidelines.
[ x] I have read the code of conduct.
[ x] I have reviewed my submission in detail.

@Manobal-Singh-Bagady pls review the pr and suggest the change if needed in the concept

@Manobal-Singh-Bagady  since its a frontend issue I am unassigning myself thank u for your review

@Manobal-Singh-Bagady since its a frontend issue I am unassigning myself thank u for your review

Alright, @ugpec79, sorry for the unclear instructions, and thanks for showing interest in this project. You can work on the other issue, which is purely based on backend #4


cd and autojump failed reporting /usr/bin/env error

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Enable plugin '...'
Run command '...', or try to complete command '...', etc.
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots or recordings
If applicable, add screenshots or record an asciinema session to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. macOS]
Zsh version [e.g. 5.6]
Terminal emulator [e.g. iTerm2]

Additional context
Add any other context about the problem here.

You didn't fill out the questions? What does which -a cd show? Also post your zshrc file.


Non-descript RED Error message when admin password is too short

Rancher Server Setup

Rancher version: 2.6.3
Installation option (Docker install/Helm Chart): Helm Chart

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE1 v1.2.16

Proxy/Cert Details: self-signed

Information about the Cluster

Kubernetes version: local Rancher cluster
Cluster Type (Local/Downstream): N/A

If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider):

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)

If custom, define the set of permissions: N/A. Rancher initialization.

Describe the bug
Very poor error reporting of a basic issue during the first initialization of the Rancher UI.
Now when a password used is too short (less than 12 characters.  recently changed in [1]), a generic RED Error button is displayed [2] with no indication of the cause and no indications of how to proceed / avoid this simple problem.
[1] https://github.com/rancher/rancher/pull/34851
To Reproduce
Install Rancher v2.6.3 on RKE1 and attempt to initialize the Rancher UI.
Result
A generic non-descriptive RED Error message is displayed. See [2] screenshot.
Expected Result
The initialization screen should describe the current minimum admin password complexity requirements and not allow the user to click the "Continue" button until these are met. Similar to when the two password entries do not match. Either that or a better error indicating the problem.  For now, just addressing the error message would be sufficient.  However, ideally the admin password minimum complexity requirements should be more robust than just min length. i.e. mixed case, special characters, non trivial words like "Rancher" etc.
Screenshots
[2]
Additional context
Very poor experience for those new to Rancher of which there are many.

NOTE that code for catching this appears elsewhere in the UI.  When creating a new local user the same non-descript RED "Error" message appears but then:

Has this been addressed in the recent 2.6.4 release?


Disable Allow blob public access for load generator storage accounts

A couple of storage accounts are getting flagged. This contains the global storage account where this is needed as it hosts images, but also the storage accounts used by the loadgenerator. Is this needed or can we disable public blob access for the loadgenerator storage accounts?

In-Scope All Storage Account types, except for File Storage.
Out-of-Scope File Storage Account types.
Logic If Allow Blob public access is disabled, then the control will pass.Note: This control will only be evaluated if the provisioning state is marked as 'Succeeded'.

@sebader thoughts?

For the load generators I'm pretty sure that we can disable it without any issues

once this is merged, please redeplo the loadgenerator and lets make sure it runs

once this is merged, please redeploy the loadgenerator and lets make sure it runs

Yes, needed to check if INT completes successfully first.


BAM/BED to parquet

Hello,
Would it be possible to provide a minimal example be it in Scala/python/CLI, how to convert say BAM to an ADAMs parquet? Same with a canonical 6 columns BED.
DK

Command line
$ adam-submit transformAlignments sample.bam sample.alignments.adam
$ adam-submit transformFeatures annotation.bed annotation.features.adam

Scala
import org.bdgenomics.adam.ds.ADAMContext._

val alignments = sc.loadAlignments("sample.bam")
alignments.saveAsParquet("sample.alignments.adam")

val features = sc.loadFeatures("annotation.bed")
features.saveAsParquet("annotation.features.adam")

Python
from bdgenomics.adam.adamContext import ADAMContext
ac = ADAMContext(sc)

alignments = ac.loadAlignments("sample.bam")
alignments.saveAsParquet("sample.alignments.adam")

features = ac.loadFeatures("annotation.bed")
features.saveAsParquet("annotation.features.adam")

Hope this helps!

Thank you very much for such a quick answer.
Bit of a follow up:
the resulting .adam files are in a parquet format readable by say arrow?

Yes, I've never had any issues with Parquet in Apache Arrow.  There was a mis-specification between the JVM Parquet and the C++ Parquet with regards to LZ4 compression at some point, I don't know if that is still a problem.  Other compression algorithms should be fine.
I did have some issues with incomplete support for Parquet via DuckDB, details here
https://github.com/heuermh/bdg-formats-duckdb
As of that effort, DuckDB did not support Parquet enums or nested schema, both features that we use in bdg-formats/ADAM.

Hello,
I can confirm that so far I have no issues reading parquet files created by ADAM using python polars.
The only a bit confusing thing was with a test RNA-Seq BAM produced by STAR (2x 150bp reads) where somehow I got min insert size= -911256.0. Is it a true insert size or a location offset of a second read in the pair?
As for the .bed to adam/parquet, I noticed that the 6 column bed got transformed into 26 column parquet with obviously empty columns for values not in the input. Not a problem, just a note that the parquets created from BED files contain such extra slots.
Well, this should let me start experimenting with ADAM after getting back from vacations.
Many thanks for your help
Darek Kedra

As for the .bed to adam/parquet, I noticed that the 6 column bed got transformed into 26 column parquet with obviously empty columns for values not in the input. Not a problem, just a note that the parquets created from BED files contain such extra slots.

We use rather rich schema for all the various genomic data types, defined in Avro at
https://github.com/bigdatagenomics/bdg-formats
The Feature schema was designed to support all of GFF2/GTF, GFF3, BED, Genbank, NarrowPeak, and IntervalList formats.  A chart with attribute mappings can be found at
https://github.com/heuermh/bdg-formats/blob/docs/docs/source/features.md


Automatically install metering datasources and queries if metering is there.

As per title, detect if metering is installed and if so, automatically apply Datasources and Queries for to use with Knative.

Bogus.


Fixes status report when pipelinerun fails

if pipelinerun fails it checks the suceeded.status for true and if not
then report as pipelinerun was not created.
but when pipleinrun fails the succeded.status is false.
this fixes it.
Signed-off-by: Shivam Mukhade<EMAIL_ADDRESS>
Changes

Submitter Checklist

[ ] ♽  Run make test lint before submitting a PR (ie: with pre-commit, no need to waste CPU cycle on CI
[ ] 📖 If you are adding a user facing feature or make a change of the behavior, please verify that you have documented it
[ ] 🧪 100% coverage is not a target but most of the time we would rather have a unit test if you make a code change.
[ ] 🎁 If that's something that is possible to do please ensure to check if we can add a e2e test.
[ ] 🔎 If there is a flakiness in the CI tests then don't necessary ignore it, better get the flakyness fixed before merging or if that's not possible there is a good reason to bypass it. (token rate limitation may be a good reason to skip).

Does that fix the bug ? I guess you tested it ? any chances you can do a minor release with it ?

yeah it fixes the bug, tested it.
if pipelinerun completes successfully then everything works fine'
if it fails then ConditionSucceeded is false so the if check in the code use to block everything and return from there.
will do a release tomorrow morning.

💯  thanks!


Sketch stops updating AWS IoT after some period of time

I have been working with a sketch that reads from some different sensors and reports values to AWS using Thing Shadows. Things seem to work up until some point when the sketch seems to lose its ability to do updates. I have had a hard time pinpointing the issue, so I tried to use a slight modification of ThingShadowEcho example, where the loop function looks like this:
void loop() {

  if(success_connect) {

    if(myClient.yield()) {
      #ifdef AWS_IOT_DEBUG
        Serial.println(F("Yield failed."));
      #endif
    }

    char JSON_buf[150];
    String payload = "{\"state\":{\"reported\":{";
    float current_stage = (rand() % 150 + 1) / 10.0;
    payload += "\"stage\": " + String(current_stage, 1);
    payload += "}}}";
    payload.toCharArray(JSON_buf, 150);

    #ifdef AWS_IOT_DEBUG
      Serial.println("stage: " + String(current_stage, 1));
    #endif

    IoT_Error_t shadowUpdateError = myClient.shadow_update(AWS_IOT_MY_THING_NAME, JSON_buf, strlen(JSON_buf), NULL, 5);

    delay(1000);
  }
}

Things mostly work. Over the period of several hours I do get a handful (9) 'Yield failed.` errors. It is hard for me to tell from the logs, but I believe the shadow updates stopped working somewhere in the vicinity shown below.
Can you offer any pointers on why this would be happening? Am I just trying to do updates too often? If so, is there a recommended minimum interval?
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:AWSIoTPythonSDK.core.protocol.mqttCore:Try to put a publish request 41 in the TCP stack.
DEBUG:AWSIoTPythonSDK.core.protocol.mqttCore:Publish request 41 succeeded.
DEBUG:comm.serialCommunicationServer:Updated serialCommunicationServer internal protocolMessageQueue by inserting a new message. Size: 1
DEBUG:AWSIoTPythonSDK.core.shadow.deviceShadow:shadow message clientToken: centuryeng_headquarters_3_0_37_vnidc
DEBUG:comm.serialCommunicationServer:Send through serial to remote client: SU T Size: 4
DEBUG:AWSIoTPythonSDK.core.shadow.deviceShadow:Token is in the pool. Type: accepted
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: z
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:Updated serialCommunicationServer internal protocolMessageQueue by inserting a new message. Size: 1
DEBUG:comm.serialCommunicationServer:Send through serial to remote client: Z T Size: 3
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: y
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:No more messages for yield. Exiting writeToExternalYield.
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: z
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:Updated serialCommunicationServer internal protocolMessageQueue by inserting a new message. Size: 1
DEBUG:comm.serialCommunicationServer:Send through serial to remote client: Z T Size: 3
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: y
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:No more messages for yield. Exiting writeToExternalYield.
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: z
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:Updated serialCommunicationServer internal protocolMessageQueue by inserting a new message. Size: 1
DEBUG:comm.serialCommunicationServer:Send through serial to remote client: Z T Size: 3
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: y
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:No more messages for yield. Exiting writeToExternalYield.
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: z
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:Updated serialCommunicationServer internal protocolMessageQueue by inserting a new message. Size: 1
DEBUG:comm.serialCommunicationServer:Send through serial to remote client: Z T Size: 3
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: y
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:No more messages for yield. Exiting writeToExternalYield.
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: z
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:Updated serialCommunicationServer internal protocolMessageQueue by inserting a new message. Size: 1
DEBUG:comm.serialCommunicationServer:Send through serial to remote client: Z T Size: 3
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: y
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:No more messages for yield. Exiting writeToExternalYield.
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: z
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:Updated serialCommunicationServer internal protocolMessageQueue by inserting a new message. Size: 1
DEBUG:comm.serialCommunicationServer:Send through serial to remote client: Z T Size: 3
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: y
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:No more messages for yield. Exiting writeToExternalYield.
DEBUG:comm.serialCommunicationServer:Clear internal list. Size: 0
DEBUG:comm.serialCommunicationServer:Accept-timer starts, with acceptTimeout: 10 second(s).
DEBUG:comm.serialCommunicationServer:1 lines to be received. Loop begins.
DEBUG:comm.serialCommunicationServer:Received: 1/1 Message is: z
DEBUG:comm.serialCommunicationServer:Finish reading from remote client. Accept-timer ends.
DEBUG:comm.serialCommunicationServer:Updated serialCommunicationServer internal protocolMessageQueue by inserting a new message. Size: 1
DEBUG:comm.serialCommunicationServer:Send through serial to remote client: Z T Size: 3

Hi @eczajk1 ,
Thank you very much for providing the information.
From the log you provide, it seems that there is no new incoming messages for the yield call to dispatch, which means there is no incoming messages from the subscription (shadow/non-shadow). Can you print out the received response on 32u4 from AR9331 when you are experiencing the yield failure? I would like to exclude the possibility of a serial1 communication issue.
Thanks,
Liusu

I am not certain what you are asking for here. Here is a snippet from the Serial monitor:
stage: 7.1
stage: 1.9
stage: 1.1
stage: 1.5
stage: 2.8
Yield failed.
stage: 4.4
stage: 4.4
stage: 12.6
stage: 0.8
stage: 3.8

The AR9331 log output pasted previously was logged using this guidance this message.

Hi @eczajk1 ,
What I mean is the output from the Serial1 connection from AR9331 to ATmega32u4. You can find the details instructions here: https://github.com/aws/aws-iot-device-sdk-arduino-yun/issues/47

Add a Serail.println(rw_buf) in the SDK after this line and see if there is any extra output in your Arduino IDE Serial monitor.

Basically I would like to know what gets transmitted over the Serial1 connection to ATmega32u4 that results in a yield failure.
Thanks,
Liusu

Thanks! But, I'm not seeing anything extra:
Y F: No messages.
stage: 14.2

Z T

Y F: No messages.
stage: 5.7

Z T

Y F: No messages.
stage: 9.1

Z T

Y F: No messages.
stage: 0.4

Z
Yield failed.
stage: 14.5

Z T

Y F: No messages.
stage: 13.3

Z T

Y F: No messages.
stage: 10.9

Z T

Y F: No messages.
stage: 13.6

Z T

Y F: No messages.
stage: 3.2

Hi @eczajk1 ,
Thank you very much for providing the information.
If you take a look at the lines near where "Yield failed" happens, you can find the following:

Z
Yield failed.
stage: 14.5

The first inter-chip protocol seems to be broken, resulting in a yield failure. SDK logic details can be found here:
https://github.com/aws/aws-iot-device-sdk-arduino-yun/blob/master/AWS-IoT-Arduino-Yun-Library/aws_iot_mqtt.cpp#L373
This could be caused by byte loss in serial1 communication. Provided that yield is a frequently invoked call. What we can do here is just simply retry "contacting" the linux chip.
Apart from this, what is more interesting is that, on successful yield attempts, there is no further messages routed back to 32u4. From the logs on Python runtime that you previously attached, there is no explicit incoming messages from the cloud being captured as well. I would suspect that the client might get disconnect or the subscriptions some how get lost.
To further detect the issue, you can utilize the AWS IoT Lifecycle events to detect if the client is connected/disconnected/subscribed. I would like to confirm if there is any silent disconnect happened when you were experiencing the issue.
Thanks,
Liusu

What we can do here is just simply retry "contacting" the linux chip.

Are you saying that this is a technique that could be implemented in the Sketch itself?

I subscribed to the Lifecycle Events using the AWS IoT MQTT client. But, so far, I have not seen any unexpected disconnected events even while Yield failed errors are logged. I also saw this other one:
SU T

Z T

Y 1 0 JSON-267

Y F: No messages.
stage: 1.3

SUFF: Unknown error.

Z T

Y F: No
stage: 7.4
ssages.
u

SU T

Z T

Y F: No messages.
stage: 8.4

I have tried listening on on Lifecycle events several times in the last week. But I am still experiencing the same issue while not seeing ANY lifecycle events showing up. I only see them when the sketch is first started.
Any advice on actions I can take to further diagnose this issue?

@eczajk1  Are you still facing this issue?

Yes, I was not able to resolve it.


quant_aware_train_linear_quant doesn't work on resnet20_cifar

I tried the quant_aware_train_linear_quant.yaml on the resnet20_cifar model, the model seems to be messed up, and cannot get any reasonable prediction and also cannot train.
Is the quant_aware_train_linear_quant.yaml is only suitable for the resnet18 ?  It seems not, could anyone help ? Thanks very much.

Indeed I had a bug in there. I just committed a fix - please check if that works for you.

Excellent


Add The Beginning by Modestep

Song Name
The Beginning
Artist Name
Modestep
Source
Youtube
Youtube Link
I-fr9z8PrFo
Direct File Link
No response
Song ID
781102
Start Offset [ms]
0
Extra
No response

Accept

reject
popular songs (if used in rated levels) are too inconvenient to be replaced by smth unrelated and are therefore usually rejected.
Note that you can still use the song yourself, but it won't be public


[KOGITO-4333] - Hardening Kogito Operator permissions

See: https://issues.redhat.com/browse/KOGITO-4333
Signed-off-by: Tarun Khandelwal<EMAIL_ADDRESS>Many thanks for submitting your Pull Request :heart:!
Please make sure your PR meets the following requirements:

[x] You have read the contributors' guide
[x] Pull Request title is properly formatted: [KOGITO-XYZ] Subject
[x] Pull Request contains a link to the JIRA issue
[x] Pull Request contains a description of the issue
[x] Pull Request does not include fixes for issues other than the main ticket
[ ] Your feature/bug fix has a unit test that verifies it
[ ] You've tested the new feature/bug fix in an actual OpenShift cluster
[ ] You've added a RELEASE_NOTES.md entry regarding this change

/jenkins test

/jenkins test

/jenkins test

/jenkins test


zip: fix build with clang 16

Description of changes
Fix implicit declarations of memset and open/closedir resulting in incorrect feature detection and conflicts with libc headers.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[x] x86_64-darwin
[x] aarch64-darwin

[x] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

They’re pretty similar to the unzip changes. It’s a shame upstream seems basically dead.


Customizable records naming for external-dns

Today fqdn for external dns records is defined the following static way:
serviceName.stackName.environmentName.rootDomain
We should allow to define a custom format. Suggestion by @vincent99:
${service}.${stack}.${hostedzone}` (with a documented list of variables, or anything available in metadata?) and people can do whatever they want to it instead of us enforcing a certain scheme

@ibuildthecloud

Great  please include SRV records as well
--
Holger Winkelmann
Managing Director
Travelping GmbH
email<EMAIL_ADDRESS>mobil: +49-171-5594745
http://www.travelping.com
http://www.linkedin.com/in/hwinkel
Sent from a mobile device. Sorry for brevity and typos
On 9 Mar 2016 05:27, Alena Prokharchyk<EMAIL_ADDRESS>wrote:
Today fqdn for external dns records is defined the following static way:
serviceName.stackName.environmentName.rootDomain
We should allow to define a custom format. Suggestion by @vincent99:
${service}.${stack}.${hostedzone}` (with a documented list of variables, or anything available in metadata?) and people can do whatever they want to it instead of us enforcing a certain scheme
@ibuildthecloud
—
Reply to this email directly or view it on GitHub.

Yes, please! Having an uncustomizable fqdn forces me to create CNAMEs to the actual domains I want.

Realized in https://github.com/rancher/external-dns/commit/07b84160bbbcf878f6d0348c1f837082f5be2e6b


Add config for custom logout_path

Improvement for #30. Allow to define a custom logout_path.

Pull Request Test Coverage Report for Build 69

3 of 3 (100.0%) changed or added relevant lines in 1 file are covered.
576 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-3.5%) to 51.854%

Files with Coverage Reduction
New Missed Lines
%

/home/travis/build/nokia/kong-oidc/lua_install/share/lua/5.1/luaunit.lua
576
48.32%

Totals

Change from base Build 67:
-3.5%

Covered Lines:
1161

Relevant Lines:
2239

💛  - Coveralls


⚠ Bofrost.nl has degraded performance

In fa53c41, Bofrost.nl (https://www.bofrost.nl) experienced degraded performance:

HTTP code: 200
Response time: 1052 ms

Resolved: Bofrost.nl performance has improved in 5d567aa.


Blog Post Postman Requests

Add new request definitions to the Observatory Postman collection (.postman/Observatory.postman_collection.json) for the following API routes:

GET /api/posts
GET /api/posts/:id
POST /api/posts
PUT /api/posts/:id
PATCH /api/posts/:id
DELETE /api/posts/:id

Setup:

Download Postman
Import the Observatory.postman_collection.json and *.postman_environment.json files into Postman
Make your additions to the Observatory collection
Export the updated Collection and replace the old version in .postman/Observatory.postman_collection.json
Commit your changes, push to your dev branch, and open a pull request to the dev branch on rcos/Observatory3 (upstream).

Resolved


Release v0.9.6

Coverage increased (+0.3%) to 89.436% when pulling 614c435b082eefc9ca802374c8aad3ee7c4f793f on prepare_for_next_release into 1aaf5e9fe99d0daadb28ec90a9e37223974d0257 on master.