Split OSP services stopping into different roles (nova workloads adoption testing)

Split OSP services stopping into different roles

Split the osp services stopping role into a role that stops the
control plane services, and another role to stop the remaining
services.  Add up stopping Galera, RabbitMQ and HAProxy services
into the latter, because stopping those for the former appears to
be too early (we need to pull OSP configuration firstly).
Both roles to be integrated into EDPM roles in follow-ups.

Disable service units and do dev envs clean up for computes.

Provide extra commands for computes to cleanup tripleo-specific (DLRN)
repos and tripleo client packages, in case of a standalone dev setup,
and to install Antelope/current-podified RDO repos instead.
That is required for instaling decontainerized libvirt daemons
and related packages, like openstack-selinux, during EDPM adoption
bootstrap tasks execution without conflicts.
TODO: Provide RHEL steps with subscription-manager instead.

Document that additional requirement for libvirt repositories
reconfig in real adoption scenarios that will be using RHEL and
subscritption manager. The exact commands to be added later.

Related changes:

Document that EDPM adoption is "the point of no return"
Document that at the time we are stopping the OSP control plane
services, we must not stop the remaining infrastructure management,
and compute services yet.

Jira: OSPRH-338

recheck NODE_FAILURE

recheck NODE_FAILURE

blocked on https://logserver.rdoproject.org/25/225/7aa08b5b79fd2f0d13af665620bb3a98517756cf/github-check/data-plane-adoption-OSP-17-to-extracted-crc/1432ecf/controller/pod/download-cache-openstack-bgjcl-logs.txt
TASK [osp.edpm.edpm_bootstrap : Download updated packages] *********************
Friday 22 December 2023  14:52:49 +0000 (0:00:03.344)       0:00:24.910 *******
FAILED - RETRYING: [standalone]: Download updated packages (5 retries left).
FAILED - RETRYING: [standalone]: Download updated packages (4 retries left).
FAILED - RETRYING: [standalone]: Download updated packages (3 retries left).
FAILED - RETRYING: [standalone]: Download updated packages (2 retries left).
FAILED - RETRYING: [standalone]: Download updated packages (1 retries left).
fatal: [standalone]: FAILED! => {"attempts": 5, "changed": false, "failures": [], "msg": "Depsolve Error occurred: \n Problem: package rhosp-openvswitch-3.1-2.el9ost.noarch requires openvswitch3.1, but none of the providers can be installed\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-59.el9fdp.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-17.el9s.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-3.el9s.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-33.el9s.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-34.el9s.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-35.el9s.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-36.el9s.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-54.el9s.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-55.el9s.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-65.el9s.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-33.el9fdp.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-44.el9fdp.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-54.el9fdp.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-2.el9fdp.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-35.el9fdp.x86_64\n  - package rdo-openvswitch-2:3.2-1.el9s.noarch obsoletes openvswitch3.1 < 3.2 provided by openvswitch3.1-3.1.0-14.el9fdp.x86_64\n  - cannot install the best update candidate for package rhosp-openvswitch-3.1-2.el9ost.noarch\n  - cannot install the best update candidate for package openvswitch3.1-3.1.0-59.el9fdp.x86_64", "rc": 1, "results": []}

There seems to be a problem here - it seems like openvswitch 3.1 is being installed, but it should be 3.2. The expectation is that 3.2 is going to be used IIRC

There seems to be a problem here - it seems like openvswitch 3.1 is being installed, but it should be 3.2. The expectation is that 3.2 is going to be used IIRC

yes, my plan is to revert https://github.com/openstack-k8s-operators/edpm-ansible/pull/518 and see if that allows us to only upgrade openstack-selinux package from the antelope RDO repos, omitting the ovs rpms upgrades hard lifting for now

The revert hasn't been applied, I can see the removed task in ansible logs

note: this cannot be tested for adoption CI as we don't have edpm-ansible in the required-projects for that job.
And more importantly we don't build the operators in data-plane-adoption repo, so that change would not be applied anyway.
I'm going to test https://github.com/openstack-k8s-operators/edpm-ansible/pull/531 with my custom AEEE operator setup on my adoption dev env.

with the 'ensure: latest' pure revert approach, my dev setup with these patches, hits another selnux related issue (I ben I missing something obvious, like relabling things?)
[root@standalone ~]# dnf list openstack-selinux
Last metadata expiration check: 0:30:24 ago on Tue 09 Jan 2024 03:04:32 PM UTC.
Installed Packages
openstack-selinux.noarch                                              0.8.39-0.20240104134359.9856d9b.el9                                              @delorean-component-common
Available Packages
openstack-selinux.src                                                 0.8.39-0.20240104134359.9856d9b.el9                                              delorean-component-common
[root@standalone ~]# grep denied /var/log/audit/audit.log
type=AVC msg=audit(1704813742.678:196676): avc:  denied  { write } for  pid=12400 comm="virtlogd" name="virtlogd.pid" dev="tmpfs" ino=4188 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_run_t:s0 tclass=file permissive=0
type=AVC msg=audit(1704813749.497:196902): avc:  denied  { search } for  pid=13034 comm="systemd-machine" name="376191" dev="proc" ino=1587723 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:svirt_t:s0:c19,c855 tclass=dir permissive=0

This worked with https://github.com/openstack-k8s-operators/edpm-ansible/pull/532

@bogdando now that https://github.com/openstack-k8s-operators/data-plane-adoption/pull/234 is merged, if you rebase this PR against main the depends-on to https://github.com/openstack-k8s-operators/edpm-ansible/pull/532 should work

oh, sigh, could we keep the docs convertion to adoc in follow up...

I confirm this worked as the part of https://github.com/openstack-k8s-operators/data-plane-adoption/pull/193, with the aofrementioned edpm_ovs patch

I confirm this worked as the part of #193, with the aofrementioned edpm_ovs patch

so, that worked on my standalone centos stream 9 tripleo wallaby setup, but failed in CI (rhel/osp17.1):
+ update-crypto-policies --set FIPS:NO-ENFORCE-EMS
Warning: Using 'update-crypto-policies --set FIPS' is not sufficient for
         FIPS compliance.
         Use 'fips-mode-setup --enable' command instead.
Unknown policy `NO-ENFORCE-EMS`: file `NO-ENFORCE-EMS.pmod` not found in (., policies/modules, /etc/crypto-policies/policies/modules, /usr/share/crypto-policies/policies/modules
)",

recheck the most recent code change wasn't CI'ed properly

recheck dep

I confirm this worked as the part of #193, with the aofrementioned edpm_ovs patch

so, that worked on my standalone centos stream 9 tripleo wallaby setup, but failed in CI (rhel/osp17.1):
+ update-crypto-policies --set FIPS:NO-ENFORCE-EMS
Warning: Using 'update-crypto-policies --set FIPS' is not sufficient for
         FIPS compliance.
         Use 'fips-mode-setup --enable' command instead.
Unknown policy `NO-ENFORCE-EMS`: file `NO-ENFORCE-EMS.pmod` not found in (., policies/modules, /etc/crypto-policies/policies/modules, /usr/share/crypto-policies/policies/modules
)",

this keeps happening again and again, I can't see that change is picked by edpm ansible job log

recheck the most recent code change wasn't CI'ed properly

...

The next blocker identified (still ovs related, but another issue) https://logserver.rdoproject.org/25/225/830d6a00e39e87893eb25fa55749778be56f5688/github-check/data-plane-adoption-osp-17-to-extracted-crc/464e35a/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/pods/configure-network-openstack-62kw4/logs/openstackansibleee.log
TASK [osp.edpm.edpm_users : Ensure user is present on the host [ openvswitch ]] ***
Wednesday 17 January 2024  13:55:10 +0000 (0:00:00.771)       0:00:03.486 *****
fatal: [standalone]: FAILED! => {"changed": false, "msg": "usermod: user openvswitch is currently used by process 20786
",
"name": "openvswitch",
"rc": 8}

The next blocker identified (still ovs related, but another issue) https://logserver.rdoproject.org/25/225/830d6a00e39e87893eb25fa55749778be56f5688/github-check/data-plane-adoption-osp-17-to-extracted-crc/464e35a/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/pods/configure-network-openstack-62kw4/logs/openstackansibleee.log
TASK [osp.edpm.edpm_users : Ensure user is present on the host [ openvswitch ]] ***
Wednesday 17 January 2024  13:55:10 +0000 (0:00:00.771)       0:00:03.486 *****
fatal: [standalone]: FAILED! => {"changed": false, "msg": "usermod: user openvswitch is currently used by process 20786
",
"name": "openvswitch",
"rc": 8}

on my devsetup, the issue is also reproducable, and the process details are:
openvsw+   `<this_pid`>  0.2  0.0  18856  8156 ?        S<s  13:50   0:13 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach
openvsw+   38031  0.3  1.5<PHONE_NUMBER>04 ?       S<Lsl 13:50   0:16 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach

the quick fix is check if the user exists and only created it if it does not (tripleo created it for the source cloud)

just to capture this for others. the issue here is the task that is failing should be conditional and only create the user if it does not exist.
for greenfiled the user will not exist. for adoption it will have been created by tripleo already so we don't need to create it.

Removed dependency on ovs edpm changes. Added dependency on a clean revert

recheck

recheck dep changed

recheck no logs

FWIW the conversion of .md files to .adoc can be done by uncommenting this:
https://github.com/openstack-k8s-operators/data-plane-adoption/blob/9f711ccfbc260c69b00576ec960dc9eff078f034/Gemfile#L9C3-L9C26
then make docs-dependencies (or if you already installed the basic dependencies, just bundle install to re-apply the edited Gemfile) and bundle exec kramdoc <markdown file>. Let me know if you'd like me to help on that front.

Gemfile

thanks for the hints!
I thought I'll never need my local rvm/ruby setup again, apparently good tech never dies

recheck no logs

@bogdando please hold the rechecks, we have found an issue with the ci job, I'm close to a fix, will recheck when it's done

recheck

good to go, PTAL


Upgrade lemmy pipeline to spaCy v3

Adjusted architecture by adding the @Language.factory() registry 🎉

This looks like what I need. I am trying to use lemmy together with swedish spacy, since lemmy needs to POS tags to do lemmatization well. After loading the nlp pipeline from swedish spacy I simply do
nlp.add_pipe('lemmy', 'lemmy')

Is this the correct way to use the changes you've made here? Also, if you happen to have any other hints on how to effectively do Swedish lemmatization I would highly appreciate it 😄

With this PR you should be able to get lemmy working by:
    import spacy
    import lemmy.pipe

    nlp = spacy.load("da_core_news_sm")
    nlp.add_pipe("lemmy")

In terms of lemmatization hints, we just published a blog post of our new experimental edit-tree-lemmatizer 🤗 A machine learning-based lemmatizer that learns rules from data that makes it easier to create lemmatization for almost every language!

Thanks! Edit-tree lemmatization looks really interesting. When it comes to Swedish data I can only find the treebanks from universal dependencies, which means ~7.4k sentences and their lemmatized versions. Do I understand correctly that this is the type of training data I need? If so I might look into training one for Swedish.

Yes, exactly! It'd be interesting to see your results on the Swedish dataset. If you want, you can share your results in the spaCy discussion forum 😄

I'm running a library graph-tool through a docker container with arc-linux and am getting an error with spacy it seems.
I tried installing your version through
pip install git+https://github.com/thomashacker/lemmy.git
and then run your example:
import spacy
print(spacy.__version__)

import lemmy.pipe

nlp = spacy.load("da_core_news_sm")

nlp.add_pipe("lemmy")

Output:
3.2.0

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-3-abbe648ba30c> in <module>
      2 print(spacy.__version__)
      3
----> 4 import lemmy.pipe
      5
      6 nlp = spacy.load("da_core_news_sm")

/usr/lib/python3.9/site-packages/lemmy/pipe/__init__.py in <module>
      1 """Initialization code for Lemmy pipeline."""
----> 2 from lemmy.pipe.component import load

/usr/lib/python3.9/site-packages/lemmy/pipe/component.py in <module>
      1 # coding: utf-8
      2 """A spaCy pipeline component."""
----> 3 from spacy.symbols import PRON_LEMMA
      4 from spacy.tokens import Token
      5

ImportError: cannot import name 'PRON_LEMMA' from 'spacy.symbols' (/usr/lib/python3.9/site-packages/spacy/symbols.cpython-39-x86_64-linux-gnu.so)

Maybe it has something to do with the docker container I'm running it in ?


Could you provide more usage examples ?

Hi,
I couldn't find another way to contact you so I'm using this.
Could you be kind enough to share some usage examples ?
I've seen the tests so I've managed to understand  most of it, but I have problems trying to use a key longer then 16 bits.
Thank you

Hi AsharLohmar.  AES128 uses a 16bit key.  tiny-AES128 library only uses AES128 cipher, hence 16bit key.

[damn, now I look so stupid]
Hi,
yes, you are right (of course).
when I've wrote the question I was way too "focused" on the "AES" part and lost the "128" part and it's meaning.
Thank you

Hi
I wrote an android app with this library via ndk
Have a look! Enjoy


Nick Peer Review Ticket

[x] .1 - Frontend UI demo.   UI takes inputs and passes them to an API and returns data from the backend to frontend.
[x] .1 - Postman API Demo.  JSON Raw/Body formatted API passes inputs to backend and updates persistent table.
[x] .1 - Java Demo - Object Creation.  Tester method that shows creation of Objects and outputs to Terminal, illustrate printing objects (hopefully using toString().

.3/.3 Had all the current requirment and the website is well designed


initial outline for sprint #10

First stab at week 10 outline

Thanks @jedahan! I am going to close this in favor of #38. But thank you.


Update documentation for skipping lines

It seems that the skipLines option was replaced with from and fromLine options, however the documentation was not updated.

How to use these from and fromLine option.
I can not find please help me ❤

Nope looks like I was wrong. skipLines is the new way, and I guess the code matches the docs now.

Can you give me some idea about this
https://github.com/mafintosh/csv-parser/issues/185#issue-743889289


Try testing PGI compiler with Travis, fixes #3

Uncomment the lines for the PGI section and the script call, now that we figured out it may have been an indentation issue.

I must say it's annoying that the GCC-7 build takes so darn long...
I'm not sure which of the packages is so much larger in that version, but I don't know if there is a way around it.
At least the PGI install and build is quick.


Always a White page

Describe the bug
Is there a roulement whit you site
The only thing I see it's always a White page!!
I paid 500$ eand since 3 days this the only thing you site create White page!!!!!!
Is it me or there is a roulement whit your site??
Link to the Bolt URL that caused the error
https://bolt.new/~/sb1-tjdzon
Steps to reproduce

Expected behavior
I Try to carte a web page and I only see White page
Screen Recording / Screenshot

Platform
Browser name = Safari
Full version = 18.1
Major version = 18
navigator.appName = Netscape
navigator.userAgent = Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Safari/605.1.15
performance.memory = undefined
Username = MichelCardinal
Chat ID = c0e031b59ccf

Additional context
No response

hi there when you get the white screen could you send a screenshot of your browser devtools?
make sure this setting is enabled so we can see everything

Specifically do you see Minified React error #418 that'd be helpful to know

Thank you for your feedback and support.
I have finally managed to create my first project:
https://meilleurevie.investissementsante.com/quiz/diabete/index.html
It took me many trials and errors, but I eventually succeeded.
Thank you for your support. I believe I will continue using Bolt for other future projects.
I now better understand that what I initially perceived as bugs were not actually bugs. It was more about my understanding of what Bolt does and how I need to formulate my requests to Bolt.
Thank you again for your feedback and support.
Keep up the excellent work. This tool truly has tremendous potential.
Best regards,


GetNodeResources is no longer used

After #8164, nodecontroller no longer calls GetNodeResources.
I don't see a comment on this being deliberate.  @cjcullen ?
Nodes are still reporting memory correctly (I presume the kubelet is reporting it).  I don't know if cores are being reported correctly, but e.g. on AWS cores have very different sizes, so we can't rely on a simple core-count.

@justinsb what do you mean by "core have very different sizes"?

I think nodes have reported resources for a while. Before nodes self registered, the nodecontroller would get an initial resource estimate for the node but that should no longer be necessary since as soon as the node is registered it will have an accurate resource representation.

/cc @davidopp @dchen1107

On AWS, the core count is only a proxy for power.  AWS defines a different unit (called ECU) which they use to provide a way to normalize this.  These instance types all have two cores, but they have varying numbers of ECUs:
m3.large: 6.5
c4.large: 8
c3.large: 7
'legacy' instance types have more variation:
m1.large: 4
c1.medium: 5
m2.large: 6.5
It's not the end of the world if we simply use core-count; I was mostly surprised to see that GetNodeResources was no longer used.
Thinking about this more, I guess the point is that the CPU limits (at the cgroups level) will be in terms of cores, so nodes should report actual cores in these units.
I guess for v-next we should consider normalizing a container's CPU request by the power of a core.  And I shouldn't be reporting fictitious units in the aws implementation of GetNodeResources, although I'm saved because it is not currently used (correctly or incorrectly) :-)

Actually, justinsb, it appears you are the only one doing it right.  In docs/resources.md we state that cpu is actually in units of KCU, which is sort of like an ECU except that we haven't defined it yet :frowning:

Filed #10438 for the ECU/normalized-cpu issue, so that this Issue can just be about deleting dead code.


switch to 2018 edition

add rust_2018_preview lang feature
cargo fix --edition
cargo fix --edition-idioms

Since rav1e is trying to stay stable-ready, could you please split the stable-breaking changes from the stable-ok ones? (adding the dyn when needed should be acceptabled)

ok, I kept only the idiomatic part


Update CompatHelper.yml

Run CompatHelper only one day per week (Saturday UTC)

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (623c438) 98.68% compared to head (2a81786) 98.68%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #10   +/-   ##
=======================================
  Coverage   98.68%   98.68%
=======================================
  Files           2        2
  Lines          76       76
=======================================
  Hits           75       75
  Misses          1        1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add hasaccess check in forms dashboard widget

ATM forms widget shows an action Create a collection even if user doesn't have rights to create one.
PR adds similar check like how it's done in other modules widgets (for example in collections).

thanks!


Fix 'make gen-latest-csv' failure

'make gen-latest-csv' command fails when 'deploy/bundlemanifests/'
directory dont have any 'yaml' files in it. This is a fix for it.
Fix Issue: https://github.com/openshift/ocs-operator/issues/750
Signed-off-by: Arun Kumar Mohan<EMAIL_ADDRESS>

...this doesn't seem right. Why would we build the CSV with no role manifests?

I am noticing that you did not commit the deploy/csv-template changes in your previous PR. Can you add those?

I did in the commit mentioned above. However, after pulling 1c30d14be it seems to be working but two files named dummy.yaml  are getting created on their own. (Can be seen in https://github.com/openshift/ocs-operator/pull/754)

There has been no follow up on #750, so I can't accept this as a valid PR. It also still strikes me as wrong that we would even need to do this.

@jarrpa , the line we have changed here is a copy statement (which is for taking backup) and the script was failing, without any error message, as there were no files in it. Even though this may be a rare occurrence, IMHO, we should not be failing at a point which could be avoided.

There is a different PR up for this same problem at https://github.com/openshift/ocs-operator/pull/891


ztest: cleanup phase should reset the timeout queue.

Reusing the k_thread structure requires a cleanup of all essential
info. We need to remove the ztest_thread information from the timeout_q.
If left untouched, when a tick occurs the thread's delta_ticks_from_prev
would be corrupted. This inturn causes a chain reaction of problems.
Thus once the unit test is completed the timeout_q is scrubbed.
Signed-off-by: Adithya Baglody<EMAIL_ADDRESS>

A unit test creates a new thread with the k_tid_t  ztest_thread. From there are I create another new thread(say foo). Which does some tests and tells ztest_pass().  So by the time this ztest_thread comes back to the entry function and call the abort on itself we would have reinitialized the k_thread stuct. so an abort never gets called.
We cant handle this in any kernel function because it a problem of the z test implementation.
So I understand that I shouldn't touch the data structures from ztest. I will call k thread abort on the k_tid_t ztest_thread during cleanup.

Delta:
Handled Andrew's review comments.

LGTM


Build fails for nsp check

The build is failing due to the Node Security Platform service being depreciated.

"The Node Security Platform service has been depreciated since 30 September this year. Beginning with npm@6, a new command, npm audit, recursively analyzes your dependency trees to identify specifically what’s insecure, recommend a replacement, or fix it automatically with npm audit fix. Note that --audit-level high flag only works for NPM versions > v6.4.0"

:tada: This issue has been resolved in version 4.5.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


A new thread is being created for every OpenFgaClient call

Description
A new HttpClient is being created for each request, leading to an increasing number of threads. Specifying a thread pool alleviates the problem, but not completely. The issue happens because here a new HttpClient is built everytime.
Version of SDK

v0.3.1

Reproduction
Below is an example of how to reproduce the issue.

Initialize OpenFgaClient with any set of configs;
Invoke method OpenFgaClient#write(ClientWriteRequest request) multiple times;
Notice that for every write call, a new thread is created. If you have 300k write calls, 300k threads are created, eventually running out of memory.

Expected behavior

It should not create a new thread for every request.

Thanks for reporting, you are correct; we should not create a new http client for each request.

Thanks for the keen eye @Jacksongoulart 🕵🏽
The fix is now out in v0.3.2


Unknown linker flag 'noexecheap' for ld.gold

When trying to build this with ld.gold I get an error that noexecheap is not a valid option. Is this flag required and necessary when building?
I can get around this in two ways, by either forcing the build to use ld.bfd or to patch out the noexecheap option, but I'm not sure which one to go for.

@nordmoen Apologies for the lack of response.  Please check if you are still seeing this issue with the latest ROCm 6.2? If not, please close the ticket. Thanks!


hover-triggered popovers disappear after opening, or teleport to top-left of container

Using a value of hover for the trigger option with popovers causes strange behavior. Sometimes the popover opens in the correct position, and then disappears immediately. Other times the popover teleports to the top left of the viewport, but only after moving the mouse cursor.
View Codepen example
May be duplicate of #32897 and #32372.
32897 just says "not working", and is for beta1. 32372 is specifically about tooltips, not popovers.
OS: Windows 10
Browser: Chrome, Version 89.0.4389.82 (64-bit)

Edit: fixed link to codepen example

Popovers are based on the Tooltip component so this should be fixed with https://github.com/twbs/bootstrap/pull/33289.

Closing since https://github.com/twbs/bootstrap/pull/33289 was merged.


Missing text overlay in tesselation example

This is a very minor issue. By accident, I found out that the key s toggles the split screen. As this is currently not indicated, I added a corresponding text overlay.

Yes, I think there are a few other examples where some of the keys are not displayed in the text overlay. Will have to check that.
Thx for the PR.


Improve detection box filtering strategies

We can improve detections using:

Scene priors
Background/Foreground subtraction and modeling

@alpha-carinae29 & @mrn-mln I suggest you take a look at this.

@mhejrati please move this issue to fine-tuning and calibration milestone

I worked on this issue in last days.
I have implemented a background subtraction with cv.createBackgroundSubtractorMOG2 module.
here is a result on Oxford Town Center:

Now I am trying to filter object detection's output based on this information. To doing this I will filter boxes that the percentage of background pixels more than a threshold.

Also I am implementing a prior distribution of bounding boxes for each pixel in the image. (only mean and variance not the whole distribution.)
To this end I computed the average and variance of width and height of bounding boxes for the centroids of the boxes recurrently with Welford algorithm. After the training Phase, for each detected bounding box we compare its width and height with the prior mean and variance and if we detect an outlier we eliminate the bounding box.

@mhejrati please mark it as "in progress".


Automatic fill of commit comment in tfs commit tool

I don't know if it's possible but...
Is it possible to generate automaticly a commit message and fill the comment field of the tfs commit tool with all the git commit messages of all the commits made since the last commit (and that will be commited) ?

I use a git alias for that:
git config alias.ctc tfs checkintool --build-default-comment
Then whenever I go to commit I just use:
git ctc

It may be useful to make it a default. Part of the reason I like the rcheckin feature is that it automatically sets the commit message to whatever I wrote in the git commit.

the alias given by @mletterle works for me and do what I want. Thanks!
There is just a little mistake  (simple quotes are missing) :
git config alias.ctc 'tfs checkintool --build-default-comment'
And you're right, it will be really useful if it could be like that by default.

If someone would be kind enough to test the default_comment branch, and confirm the new behavior works as desired,
that would be great.


Fix flask-church link in docs

Current coverage is 98.46% (diff: 100%)

Merging #72 into master will not change coverage

@@             master        #72   diff @@
==========================================
  Files            28         28
  Lines          1494       1494
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           1471       1471
  Misses           23         23
  Partials          0          0

Powered by Codecov. Last update 4ee7810...d1d0f4f

@cl0ne Thank you!

I saw this, I think it's the only other mention of church in the program from a search


chore: fix spelling in upload signed url docs

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 84.515%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
6987

Relevant Lines:
8209

💛  - Coveralls


Layout within Scroll

I would like to have the child of a Scroll widget take up at least the visible area of the Scroll, but I can't figure out how using the current API.
As a motivating example, suppose I want to write a note-taking app with infinite scrolling. I try to do this by putting a Drawing widget inside a Scroll. Initially, I want the Drawing to take up the whole visible area of the Scroll but no more. As the drawing gets bigger, I want it to be scrollable. The issue is that there doesn't appear to be a way for Drawing's layout function to know how big the Scroll is: it just gets to see unconstrained bounds.
For example of what is done elsewhere, Flutter has a sort of custom layout protocol within scrollable widgets.

yes, I've been thinking about this in the past couple days; the current ways to achieve this are hacky, like having a parent widget of the Scroll send its size down to the inner widget in a Command. I'm not sure what the best approach is, will do some looking.

I've got a half baked scrollbar PR sitting around right now which allows scrollbars to be inlay instead of painting over the child content and this is very close to an issue I'm having. When adding content to a Scroll in the list example it to now require a vertical bar which shortens the child's paintable width so it also requires a horizontal scrollbar until the next time a layout occurs at which point the children will know the correct width. (I need to open this as a draft)
I also want to expand to the Scroll's size in my text editor project so any ideas on how to improve this would be awesome

yea, there are a few things here.
I think one simple improvement is to make Scroll pass its 'actual' size as the min constraint to its children; currently children have no constraints.
For this to make sense, we also want to make sure scroll has a clear size? Does it make sense for the size of the scroll widget to be determined by the size of the child? I think maybe not; maybe the scroll should always have a fixed size (and so be passed tight constraints?)

I think it would make sens for Scroll to pass down its own size as min constraints but still shrink itself to this size of its child. When we want the Scroll to take up the available space it could always be wrapped in a SizedBox using WidgetExt::expand right?

yes, I think that makes more sense.

I've been putting a bit of thought into this over the past few days as I continue to exercise the scroll widget in my text editor project and I've come to the conclusion that perhaps scrolling should be some kind of functionality that widgets should be able to embed in themselves somehow. Split up the functionality and let widgets pick and choose which parts to use. I'm not sure what level of abstraction is most suitable but some scroll building blocks would be handy.
This would allow the generic scroll widget to continue to exist (albeit calling out to these scroll building blocks) but there can also be more focused scrolling widgets re-using this same functionality (such as vertical lists). This is important as some widgets need a closer tie to their scrolling behaviour such as text edit widget which needs to know/manipulate the current scroll value or a tab bar which might want to reuse the scrollbar rendering but have different behaviour.
Perhaps I'm barking up the wrong tree with this idea but I think it has some merits and I'm interested to hear other's thoughts on it

I think something like this makes sense; we would break the scrolling stuff up into a "clipping widget" and then the mechanics of actually scrolling. the clipping widget would be some widget that shows an offset region of a child.

Closing in favour of https://github.com/linebender/druid/issues/73


Enable KafkaSource controller

Fixes #312
Proposed Changes

Add E2E tests
Enable KafkaSource controller (webhook and controller handlers)
Add KafkaSource manifests

Release Note

None

Docs

None

/unhold

/lgtm
/approve
Thanks, looks great! We now have a KafkaSource impl!

Thanks for your reviews!


All Multivariate Models (Except SOFTS) Cannot Be Ran

What happened + What you expected to happen
I'm seeing multiple issues (all related to matrix dimensions it seems) for all multivariate models (Except HINT because I could not determine S parameter from documentation and SOFT which seems to work). This is reproducible in both standard models and Auto models.
Errors presented are not the full stack but reduced for cleanliness:
TSMixer
  File "/burg/pmg/users/aec2244/mambaforge/torch/lib/python3.10/site-packages/neuralforecast/models/tsmixer.py", line 118, in forward
    x = x * self.weight
RuntimeError: The size of tensor a (4) must match the size of tensor b (7) at non-singleton dimension 2

TSMixerx
  File "/burg/pmg/users/aec2244/mambaforge/torch/lib/python3.10/site-packages/neuralforecast/models/tsmixerx.py", line 146, in forward
    x = x * self.weight
RuntimeError: The size of tensor a (4) must match the size of tensor b (7) at non-singleton dimension 3

TimeMixer
  File "/burg/pmg/users/aec2244/mambaforge/torch/lib/python3.10/site-packages/neuralforecast/models/timemixer.py", line 85, in _normalize
    x = x * self.affine_weight
RuntimeError: The size of tensor a (4) must match the size of tensor b (7) at non-singleton dimension 2

StemGNN
  File "/burg/pmg/users/aec2244/mambaforge/torch/lib/python3.10/site-packages/neuralforecast/models/stemgnn.py", line 339, in self_graph_attention
    key = torch.matmul(input, self.weight_key)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (175x4 and 7x1)

MLPMultivariate
  File "/burg/pmg/users/aec2244/mambaforge/torch/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 117, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (25x384 and 672x1024)

Versions / Dependencies
neuralforecast is 1.7.4
datasetsforecast is 0.0.8
pytorch_lightning is 2.3.0
torch is 2.4.0+cu121
Reproduction script
Reproduced the error with only Nixtla related packages, including NHITS as working example

import optuna
import pandas as pd

from neuralforecast import NeuralForecast
from neuralforecast.losses.pytorch import MAE
from neuralforecast.auto import AutoStemGNN, AutoTSMixer, AutoTSMixerx, AutoNHITS,AutoSOFTS, AutoHint, AutoTimeMixer, AutoMLPMultivariate
from neuralforecast.models import NHITS

from datasetsforecast.long_horizon import LongHorizon

# Change this to your own data to try the model
Y_df, _, _ = LongHorizon.load(directory='./', group='ETTm2')
Y_df['ds'] = pd.to_datetime(Y_df['ds'])
Y_df = Y_df[["ds", "unique_id","y"]]

#need to reduce this for memory reasons on the system im using (should not affect reproduction of issue)
Y_df = Y_df[Y_df.ds <= Y_df.ds.median()]
Y_df = Y_df[Y_df.ds <= Y_df.ds.median()]

H=96
num_samples=10
num_gpus=1

nhits_default_config = AutoNHITS.get_default_config(h=H, backend="optuna")
tsmixer_default_config = AutoTSMixer.get_default_config(h=H, backend="optuna", n_series=Y_df["unique_id"].nunique())
tsmixerx_default_config = AutoTSMixerx.get_default_config(h=H, backend="optuna", n_series=Y_df["unique_id"].nunique())
stemgnn_default_config = AutoStemGNN.get_default_config(h=H, backend="optuna", n_series=Y_df["unique_id"].nunique())
mlp_default_config = AutoMLPMultivariate.get_default_config(h=H, backend="optuna", n_series=Y_df["unique_id"].nunique())
timemix_default_config = AutoTimeMixer.get_default_config(h=H, backend="optuna", n_series=Y_df["unique_id"].nunique())

#these work
soft_default_config = AutoSOFTS.get_default_config(h=H, backend="optuna", n_series=Y_df["unique_id"].nunique())

models = [AutoNHITS(h=H,
                    config=nhits_default_config,
                    gpus=num_gpus,
                    valid_loss=MAE(),
                    search_alg=optuna.samplers.TPESampler(),
                    backend="optuna",
                    num_samples=num_samples),
        AutoTimeMixer(h=H,
                    n_series=Y_df["unique_id"].nunique(),
                    config=timemix_default_config,
                    gpus=num_gpus,
                    valid_loss=MAE(),
                    search_alg=optuna.samplers.TPESampler(),
                    backend='optuna',
                    num_samples=num_samples),
        AutoSOFTS(h=H,
                    n_series=Y_df["unique_id"].nunique(),
                    config=soft_default_config,
                    gpus=num_gpus,
                    valid_loss=MAE(),
                    search_alg=optuna.samplers.TPESampler(),
                    backend='optuna',
                    num_samples=num_samples),
        AutoTSMixer(h=H,
                    n_series=Y_df["unique_id"].nunique(),
                    config=tsmixer_default_config,
                    gpus=num_gpus,
                    valid_loss=MAE(),
                    search_alg=optuna.samplers.TPESampler(),
                    backend='optuna',
                    num_samples=num_samples),
        AutoTSMixerx(h=H,
                    n_series=Y_df["unique_id"].nunique(),
                    config=tsmixerx_default_config,
                    gpus=num_gpus,
                    valid_loss=MAE(),
                    search_alg=optuna.samplers.TPESampler(),
                    backend='optuna',
                    num_samples=num_samples),
          AutoStemGNN(h=H,
                    n_series=Y_df["unique_id"].nunique(),
                    config=stemgnn_default_config,
                    gpus=num_gpus,
                    valid_loss=MAE(),
                    search_alg=optuna.samplers.TPESampler(),
                    backend='optuna',
                    num_samples=num_samples),
          AutoMLPMultivariate(h=H,
                    n_series=Y_df["unique_id"].nunique(),
                    config=mlp_default_config,
                    gpus=num_gpus,
                    valid_loss=MAE(),
                    search_alg=optuna.samplers.TPESampler(),
                    backend='optuna',
                    num_samples=num_samples)
    ]

nf = NeuralForecast(models=[models[1]], freq="15min")

Y_df.sort_values("ds", inplace=True)
nf.fit(df=Y_df)

I also reproduced these errors with the base models.

from neuralforecast.models import StemGNN, TSMixer, HINT, TSMixerx, NHITS, MLPMultivariate, SOFTS, TimeMixer

models = [NHITS(h=H,
                input_size=H,
                windows_batch_size = 25,
                max_steps = 100,
                loss=MAE()),
          TimeMixer(h=H,
                input_size=H,
                batch_size = 25,
                max_steps = 100,
                n_series=Y_df["unique_id"].nunique(),
                loss=MAE()),
          SOFTS(h=H,
                input_size=H,
                batch_size = 25,
                max_steps = 100,
                n_series=Y_df["unique_id"].nunique(),
                loss=MAE()),
          TSMixer(h=H,
                  input_size=H,
                  batch_size = 25,
                  max_steps = 100,
                  n_series=Y_df["unique_id"].nunique(),
                  loss=MAE()),
          TSMixerx(h=H,
                  input_size=H,
                  batch_size = 25,
                  max_steps = 100,
                  n_series=Y_df["unique_id"].nunique(),
                  loss=MAE()),
          StemGNN(h=H,
                  input_size=H,
                  batch_size = 25,
                  max_steps = 100,
                  n_series=Y_df["unique_id"].nunique(),
                  loss=MAE()),
         MLPMultivariate(h=H,
                input_size=H,
                n_series=Y_df["unique_id"].nunique(),
                batch_size = 25,
                max_steps = 100,
                loss=MAE())]

nf = NeuralForecast(models=[models[1]], freq="15min")

Y_df.sort_values("ds", inplace=True)
nf.fit(df=Y_df)

Issue Severity
High: It blocks me from completing my task.

I think i've isolated the bug further.
It seems as if it's an issue between torch for GPU vs torch for CPU. as the models run with no issue for torch 2.4.0+cpu . I also tried all torch versions 2.3.0 - 2.4.0 and all GPU compliant versions fail.

That's good to know, because I was not able to reproduce the errors at all! I'll see if we can have a fix for that on our end, or if it's out of our control.

I found another important wrinkle, seems that the bug not only has to due with torch for GPUs but regarding the amount of GPUs available.
The system im using is managed by slurm and a session with the following parameters works
srun --pty -A pmg --time 0-04:00 --gpus 1 --cpus-per-task 5 --mem-per-cpu 5G /bin/bash

but a session with the below parameters fail
 srun --pty -A pmg --time 0-04:00 --gpus 2 --cpus-per-task 5 --mem-per-cpu 5G /bin/bash

My assumption is that there is a mismatch somewhere between the number of GPUs indicated in the model definition to use vs the number models are actually using causing some dimension mismatch. Although I'm more of a JAX guy so I have no further insights into what torch/pytorch_lightning may be doing.
So I guess the issue is not as catastrophic as it might have been but this is going to seriously limit the scalability of these models, and with the recent Neurips 2024 workshop Time Series in the Age of Large Models it would be nice to have a more scalable package to use for submission (as I intend to do)

pytorch lightning uses all available GPUs by default (you should see this in the logs as the training starts) by using data parallelism and I think this doesn't play well with multivariate models because they require the batches to have all series. A possible solution would be to set the batch size to n_series * n_gpus so that each GPU gets n_series, however I think we have checks in place that won't allow this.
In the meantime the only solution is to limit those models to one GPU, which you can do by setting devices=1 or devices=[DEVICE_ID].

FYI I was just trying to run some larger LLMs (specifically Gemma2-2B) with time-LLM and this issue seems to appear here as well when i try to shard the model across GPUs.  RuntimeError: mat1 and mat2 shapes cannot be multiplied (1024x2304 and 768x1024)


proposal: net/http: expose status code via optional interface on http.ResponseWriter

Proposal Details
Expose status code of http response
In this proposal I want to discuss the possibility of exposing the http status code of an HTTP response of an HTTP server.
This is mostly useful for logging and telemetry purposes.
Current situation
Currently, to capture the status code of a request you have to something like this:
type statusRecorder struct {
	writer            http.ResponseWriter
	writeHeaderCalled bool

	StatusCode int
}

var _ http.ResponseWriter = &statusRecorder{}

func (s *statusRecorder) Header() http.Header {
	return s.writer.Header()
}

func (s *statusRecorder) Write(data []byte) (int, error) {
	if !s.writeHeaderCalled {
		// Replicate behaviour from http.ResponseWriter.
		// When Write() is called before WriteHeader(), a 200 OK is returned.
		s.WriteHeader(http.StatusOK)
	}
	return s.writer.Write(data)
}

func (s *statusRecorder) WriteHeader(statusCode int) {
	s.writeHeaderCalled = true
	s.StatusCode = statusCode
	s.writer.WriteHeader(statusCode)
}

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("GET /", func(w http.ResponseWriter, _ *http.Request) {
		w.WriteHeader(http.StatusOK)
	})

	_ = http.Server{
		Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			st := &statusRecorder{writer: w}
			mux.ServeHTTP(st, r)
			fmt.Printf("Request completed with code %d\n", st.StatusCode)
		}),
	}
}

This works but has a major issue in which it does not expose any optional interfaces implemented by the internal http.response struct.
Things like http.Flusher, http.Hijacker, etc... are lost.
We could implement these extra optional interfaces in the statusRecorder described above but what if the original http.ResponseWriter did not implement this interface?
Code using the wrapped request might incorrectly think that extra functionality is available even though it isn't.
My proposal
Since we can not wrap the http.ResponseWriter without losing these optional interfaces the only real option that we have left is to add another optional interface.
// Ignore function / interface names for now, feedback on this is much appreciated.
type StatusExposer interface {
	StatusCode() int
}

The http.request struct would then implement this extra interface. This would allow you to do the following:
func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("GET /", func(w http.ResponseWriter, _ *http.Request) {
		w.WriteHeader(http.StatusOK)
	})

	_ = http.Server{
		Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			// We are not wrapping w, so we do not lose optional interfaces implemented by w.
			mux.ServeHTTP(w, r)

			var code int
			if st, ok := w.(http.StatusExposer); ok {
				code = st.StatusCode()
			}
			fmt.Printf("Request completed with code %d\n", code)
		}),
	}
}

Most structs inside net/http that implement http.ResponseWriter already have easy access to the returned HTTP status code or require minimal changes to save this.
An example implementation of the StatusCode() function could be as simple as adding:
// in src/net/http/server.go
func (w *response) StatusCode() int {
	return w.status
}

The thing that might be a bit tricky with implementation is how to deal with websockets, etc... but I think this is easily handled by tests.

Related Issues

net/http/httptrace: add ServerTrace hooks #18997
proposal: net/http: add helper funcs for HTTP status codes #29652 (closed)
net/http: http.ServeFile and http.ServeContent Explicitly Set http.StatusOK #11269 (closed)

Related Code Changes

net/http: add ResponseStater interface, ResponseState struct

Related Documentation

Package httptest > type ResponseRecorder 				¶

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Looks like a similar approach was taken (but abandoned in 2017) in: https://go-review.googlesource.com/c/go/+/36647#related-content
The original author mentions trying a different approach via httptrace but as far as I know the current implementation of httptrace does not provide the response status code. So I think this proposal is still relevant.

we still recommend wrapping. See ResponseController and the Unwrap method https://pkg.go.dev/net/http#NewResponseController

Thanks for the response. In the codebase that I'm currently working on I still see a lot of code that depends on direct interface checks but if wrapping is the recommended way of doing this via an Unwrap method then I think migrating the legacy code is the way forward.


improve skills catalog scalability

currently skills catalog replicates events and achievements across all the imported skills (user_points, user_performed_skill and user_events row);
import a skill requires to copy events and achievements - a very expensive operation

as a part of this ticket

do not replicate events and achievements but rather re-use
update /api and metrics endpoints to load the data from the original events and achievements tables

PR: https://github.com/NationalSecurityAgency/skills-service/pull/1159


the admin panel of packages does not work or not good enough

[ ] start/stop did not work on packages
[ ] same for install
[ ] packages like alerta and myjobs_ui are obsolete, we have our admin interface
[ ] should there be no detail page of a package, job, ... it does not open
[ ] bad formatting of message in alerts

alert message badly done:


🕵 find a way to test monaco editor better

Background
We're using the monaco editor in two features already. It would help is to find a way to test it reasonably on Jest level (proper testing for this probably works best in a browser environment!).
Currently we use a mock that's used in setupFilesAfterEnv ("<rootDir>/test-setup/mock-monaco-editor.tsx",)
It's not critical (yet), because we are testing manually, but it would be nice to have for the future!

I actually ended up investigating this quite a lot and trying different solutions on Aiven console side. In the end there are too many browser features (canvas, service workers etc, ...) that the editor requires to make it work making it unsuitable to be tested in node environment.
As long as we don't do anything special with the editor features, mocking out the editor is fine (for example validation).
But once we do, the best way to test the editor IMO is to set up e2e test to test the editor features. I.e run it in browser environment.
I then learned about component testing and it seems like the perfect testing environment for testing browser code. I tried the Playwright BETA for component testing and i felt like it was not ready yet, however looking into cypress could be worth it.

Thanks for the feedback. It seems like that is really hard.
I'm not sure how feasible real e2e would be for us, but we could set up integration tests with mocked apis using cypress or playwright -> or use component tests! I've had a really good experience with using a UI testing framework for doing the UI heavy integration tests while using mocked apis instead of backend -> it's of course not giving the same level of confidence as an E2E tests, but these test can be easier to write then the jest/testing-library test on big components (e.g filling out complex forms etc.)
I'm a bit more inclined to Playwright tbh. I worked with both in my past job and while both worked well, Playwright turned out to be  (a lot) faster and therefore more stable. On the other hand I already did a whole setup for integrating cypress in an app and pipeline and all, so have more experience in that area 🤔
I think we could maybe close this specific ticket and create a new housekeeping ticket to investigate UI tests a bit more for Coral later? 🤔


the way of loading data about train_softmax.py

As I know, the script load data by reading these image files.
what I need to do is transform these files to a TFRecord file and read it when training.
I have read this code
and know how to transform image files to TFRecord files and read as bacth
But I don't know how to modify the train_softmax.py
Looking forward your reply sincerely
Thx

@limicala ，你最后找到办法了吗


docs(transformer): add README including style guide

Add a README to oxc_transformer crate, including a "style guide" for writing transforms. As discussed in #4881.

#4899  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on  Graphite

@Boshen and @Dunqing please could you review as closely as you can and suggest any changes?
I am not at all tied to the content of this document, very happy to make changes if either of you would prefer some things were different.
This is quite boring, but there would not be much point in adopting a style guide if we don't all agree with it, and consequently ignore it / diverge from it! It'd be preferable to work out any differences of opinion upfront.


No support for config reload with stdin input plugin

I am using config reloading for testing my configuration. When I start up I get the message:
[2017-02-02T11:33:09,198][ERROR][logstash.agent           ] Logstash is not able to start since configuration auto reloading was enabled but the configuration contains plugins that don't support it. Quitting... {:pipeline_id=>"main", :plugins=>[LogStash::Inputs::Stdin]}

I am using logstash 5.2.0.

I have same problem with my test logstash box
in my logstash.yml file, plugin part is commented, it should be adopted as the defult setting,,

Autoreload is disabled on purpose for the stdin input plugin.
# When a configuration is using this plugin
# We are defining a blocking pipeline which cannot be reloaded
def self.reloadable?
  false
end

Autoreload is disabled on purpose for the stdin input plugin.

Well, yes it is, but that sucks: I'd most use stdin for testing changes to my config. And reload is most useful for the same purpose -- with reload, I can make config changes and easily iterate on the result.
Perhaps a workaround can be documented, like using a file input plugin pointing to a named pipe (I haven't tried this, but I don't see why it wouldn't work)?

Same here. A possible solution is to create dedicated plugin for testing purpose with reload support

Unfortunately that's just how java works, the stdin channel can't be closed and reopened. We already have extra code that allows logstash to interrupt the read of the stdin's underlying channel so that logstash can terminate gracefully.
The way the interrupt happens is by calling close and catching the exception, but once that is done, it's not possible (from what we know) to reopen stdin until the JVM is restarted.
Patches, suggestions or new ideas are more than welcome. The requirement is simple: it must be possible for a thread to read stdin, be told to stop reading, and be able to read again (this last one is what we don't support).

I'm not sure why you would need support for rereading standard input. In most cases a reconfiguration of the logstash pipeline doesn't affect the standard input read in any way.
There are three cases to consider:

configuration on other plugins changes -- stdin remains the same other than perhaps the reference to the next plugin in the the pipe. Don't see any complexity here.

configuration on stdin plugin changes if that config was used as part of creating the input stream. Fine, warn on that case and ignore the change or bail.

configuration on stdin plugin changes if that config is unrelated to the read. Choose whether to support this or not. I'm fine with ignoring or bailing out here, for now.

Correct, as of now pipelines are reloaded as a whole, we don't support partial reloading of the filter or output sections only.
We aim to have that in the future but it is outside the scope of this plugin.
At the moment the only work around is having a separate pipeline sending from stdin to..lumberjack or TCP for example, and reloading a second pipeline that uses beats input or TCP input with the filters you want to test.

Also, just for reference, there is an issue open for adding a simulate API to logstash https://github.com/elastic/logstash/issues/7832 , that you allow you to test events against the filter section of a pipeline configuration


Shader resources in shader nodes from the shader node factory only work with the "stage" keyword

The shader keys or object pin updaters aren't working for them. This is a problem because it prevents shader parts from being used multiple times in the same shader.

Any chance we can have more detailed descriptions, repro case for this?


Support delta & size thresholds for database_size

This pull request tries to implement what has been discussed in issue #98.

@ioguix: any chance that you update this? Or shall I try? The easiest fix is probably to make critical and warning compulsory.

Fixed in my branch. @Krysztophe, let me know if it is ok for you! thanks,

I hope to test this before next year.

As discussed, your tests were good enough and this only bug has been fixed and tested on my side. So I merged the branch.
Thanks!

I was just making a last check while you were merging...
These may be problems:

fails to detect that critical size < warning size (that could be a feature to avoid any critical, as both thresholds are compulsory) ;
delta is computed as an absolute value, but it is not specified in the comments
delta does not accept negative values, that could be useful but was not discussed before.

And I've just found that '100B' is not a valid size value but this is more general (theoretical?) problem.

I was just making a last check while you were merging...

oops, sorry :(

fails to detect that critical size < warning size (that could be a feature to avoid any critical, as both thresholds are compulsory) ;

It seems to me this is a common problem with all other services. so I choose to ignore this for this PR.

delta is computed as an absolute value, but it is not specified in the comments

I don't understand what you mean

delta does not accept negative values, that could be useful but was not discussed before.

No need to allow negative value, we compare the absolute value of the database size variation with the given threshold: if abs($delta) >= $limit

And I've just found that '100B' is not a valid size value but this is more general (theoretical?) problem.

Could you file a bug report with details please?


Store stacking context mask as a display list instead of bitmap

Display list item PushStackingContext currently uses following structure to store a mask:
struct StackingContextMask {
    NonnullRefPtr<Gfx::Bitmap> mask_bitmap;
    Gfx::Bitmap::MaskKind mask_kind;
};

Storing a bitmap inside a display list item means following problems:

It's painted during display list building = layering violation.
Painted always using Gfx::Painter even when Skia painter is enabled.
It's always painted regardless of whether it's visible in a viewport.

Instead of a bitmap we should change PushStackingContext to store nested DisplayList with commands required to paint a mask.

Fixed by #1143?


rust-enable-format-on-save + compile = annoying jump to end of buffer

This is bug is similar to #231, but instead of begin of buffer,
at now it jumps to the end of buffer with rust code.
I have applied fix from #251
Steps to reproduce:

Split window by two and run compile command in the second one:
C-x 2 C-x O M-x compile choose and run compilation command
Return back to code window and choose code at the middle of buffer and change it's
formating in so way that rustfmt should modify it (Do not save buffer)
Jump to compilation window C-x O and press g

as result the position of cursor in buffer with rust code will change to the end of buffer.
/cc @brotzeit

Can you do me a favor and try this with my fork. I have added a custom process and it's possible that this issue doesn't occur with it. If the problem still exists I will try to fix it.

@brotzeit
I tried. For real project is does not work at all.
error[E0583]: file not found for module `cpp_code`
 --> /home/evgeniy/projects/rust-infra/swig/macroslib/src/cpp/mod.rs:1:5
  |
1 | mod cpp_code;
  |     ^^^^^^^^
  |
  = help: name the file either cpp_code.rs or cpp_code/mod.rs inside the directory "/tmp"

looks like it copy just one file into '/tmp' and run rustfmt for it?
But for dummy test project created with cargo --new bin it looks like work.
I wonder how scheme with copying will work for real projects, with tree of modules,
plus part of code is generated by build.rs and lay in target and included via include!?

Thanks for taking a look at it!
Actually you wouldn't even get this error with the current solution in rust-mode, because it is ignored when running rustfmt with stdin. I've tried it with a temp file as there are several issues with stdin, but I didn't think of this error.
What do you think would be the best solution. Switch to stdin again, try to ignore this error with the current solution or create a temp file in the project directory ?
I don't like the solution with the temp file, but I think stdin is currently not really usable.

@brotzeit

What do you think would be the best solution. Switch to stdin again, try to ignore this >error with the current solution or create a temp file in the project directory ?
I don't like the solution with the temp file, but I think stdin is currently not really usable.

If I edit foo.rs why not just run rustfmt against foo.rs without stdin/temp etc?
emacs reload changed on disk file without problems and it saves position of cursor.
The only problem is time of reload, I think default timings like 5 seconds or so,
so may be force reload of current buffer after run of rustfmt.

I wouldn't have a problem with that. The best way would be reverting I guess, but I will check if this is practicable.
But probably there will be complaints about it as it changes the user experience.

@brotzeit
May be time  to ask emacs-devel for general infrastructure for that? gofmt/clang-format-buffer and so on all of them win from that too.

Maybe, but I don't think it would change anything for this issue. rustfmt just isn't ready.
And even if they would add something we could use it in 5 ? years.

It should work, but I just fixed it for now.

Maybe, but I don't think it would change anything for this issue. rustfmt just isn't ready.
And even if they would add something we could use it in 5 ? years.

Looks like function already exists in emacs 26: https://github.com/emacs-lsp/lsp-mode/pull/258#issuecomment-360667059 , also in lsp-mode#258 was implemented some work around for emacs 25.

Hi,
not sure if relevant: I experience a slight variation of this issue: when I save the code is formatted and the cursor goes to the lower part of the buffer (as if running executing command recenter-top-bottom i.e. C-l).
Example:
https://asciinema.org/a/kDrKAA9n6OFkDKYEa4xSfYBbH
I'm using Emacs 25.2.2 on Linux and rust-mode 20180215.210

Hi @zippy, I've opened a pull request #269 (no feedback so far) but it works for me (© 2018). Could you try it and see if it works for you, too?

I added save-excursion to the function so it should work now(if it didn't already before).


Introduction and Attribution sections of CoC carry redundant info

Both sections have essentially the same info.

Motion to strike the Introduction and start with the Pledge.

Alright, that makes sense.
Let's start with the pledge.

Closing since #12 fixes this


Incorrect path during upload of .hart package.

I have a build in Jenkins using habitat plugin that succeeded about 2 weeks ago, but as of this morning, it's failing here:
[... snip ...]
Setting Directory: /hab/svc/jenkins-slave/data/workspace/workspace/uepipeline-vault-policies_master/results/last_build.env
Artifact bluepipeline-vault-policies-0.1.1-20190103170144-x86_64-linux.hart found in: /hab/svc/jenkins-slave/data/workspace/workspace/uepipeline-vault-policies_master/results/last_build.env
Last Package: /hab/svc/jenkins-slave/data/workspace/workspace/uepipeline-vault-policies_master/results/last_build.env/bluepipeline-vault-policies-0.1.1-20190103170144-x86_64-linux.hart
Could not find hart /hab/svc/jenkins-slave/data/workspace/workspace/uepipeline-vault-policies_master/results/last_build.env/bluepipeline-vault-policies-0.1.1-20190103170144-x86_64-linux.hart

Note that it doesn't seem to be picking up the correct path to upload the .hart file from: uepipeline-vault-policies_master seems to be chomped at the beginning of the directory path.
My plugin version is: habitat-executor v0.5.3

I believe this was due to a rework of how we're validating running on windows vs linux.
I've related a new version with some changes that will hopefully fix this.
https://github.com/jenkinsci/habitat-plugin/releases/tag/habitat-0.6.1
I'll check on the status of the builds and validate.


Deprecate xr.cftime_range in favor of xr.date_range

What is your issue?
xr.date_range automatically chooses a return type based on calendar and can take use_cftime=True to always return CFTimeIndex, so I don't see why we need xr.cftime_range any more

+1, indeed we can make cftime_range private.


Store partition synopses in separate files

:notebook_with_decorative_cover: Description
Introduce a new PartitionSynopsis top-level flatbuffer and use that to split out the meta index state into separate files.
Note that the idea is to keep this as a backwards-compatible change, so no versions are bumped.
:memo: Checklist

[x] All user-facing changes have changelog entries.
[x] The changes are reflected on docs.tenzir.com/vast, if necessary.
[x] The PR description contains instructions for the reviewer, if necessary.

:dart: Review Instructions

Review commit-by-commit.

I tested this by creating a database with many small partitions and timing startups of the tagged release build and 2 subsequent startups of a build from this branch.
For this test, the final startup time was reduced by about 56% compared to the first startup.
A snippet from my terminal to reproduce:
zstdcat ~/t/data/vastingest.json.zst | ~/t/build/vast/release/bin/vast --max-partition-size=128 -d mdx.db
-N import --batch-size=64 suricata --simdjson
source.hpp:201 json-reader produced 1477056 events at a rate of 36300 events/sec in 40.69s
...
source.hpp:201 json-reader produced 10399 events at a rate of 46935 events/sec in 221.56ms
...
xo ~/t/vast (topic/separate-midx-file|…67415) $                                              vast-dev-gcc
xo ~/t/vast (topic/separate-midx-file|…67415) $
time ~/t/build/vast/release/bin/vast --max-partition-size=128 -d mdx.db -N status
...
________________________________________________________
Executed in   17.27 secs   fish           external
   usr time    7.55 secs    0.00 micros    7.55 secs
   sys time    7.05 secs  777.00 micros    7.05 secs

xo ~/t/vast (topic/separate-midx-file|…67415) $
time ~/t/build/vast/mdx/bin/vast --max-partition-size=128 -d mdx.db -N status
....
________________________________________________________
Executed in   18.96 secs   fish           external
   usr time   13.21 secs  687.00 micros   13.21 secs
   sys time   11.22 secs  165.00 micros   11.22 secs

xo ~/t/vast (topic/separate-midx-file|…134810) $
time ~/t/build/vast/mdx/bin/vast --max-partition-size=128 -d mdx.db -N status
...
________________________________________________________
Executed in    7.62 secs   fish           external
   usr time    6.63 secs  572.00 micros    6.63 secs
   sys time    3.78 secs  138.00 micros    3.78 secs

Before merging this should probably still be rebased onto master.


Cannot Sign PSBT

I set up a multi-sig wallet requiring 1 of 2 keys but the second key was not finalized.  Cannot sign the PSBT.

Is this a bug?  Even when I change the multi-sig wallet to 1of1 I cannot sign the PBST.

It's unclear what you are doing. Your screenshot shows an incomplete multisig wallet. Then, you mention a PSBT, presumably created from somewhere else. You will need the same wallet configuration in Sparrow as that "somewhere else" that created the PSBT.

Hello,So I didn’t realize that using two same signers (cold card and ledger) would recreate the same wallet I already have.  I decided to remove the ledger in the 1 of 2 multisig wallet setup, and sent the btc to the wallet.  Today, I tried to send the btc out of the wallet but the cold card wouldn’t sign the PSBT. When I change the multisig setup to 1 of 1 (cold card only) the btc is visible but still would sign the PSBT. If I use a different signer there is an error stating the transaction will be wiped.On Apr 30, 2024, at 12:36 AM, craigraw @.***> wrote:
It's unclear what you are doing. Your screenshot shows an incomplete multisig wallet. Then, you mention a PSBT, presumably created from somewhere else. You will need the same wallet configuration in Sparrow as that "somewhere else" that created the PSBT.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

I decided to remove the ledger in the 1 of 2 multisig wallet setup

This creates an entirely different wallet, with different addresses.

When I change the multisig setup to 1 of 1

Again, this creates an entirely different wallet.
You need to use (or recreate) the original wallet into which you received the funds, in order to spend it.

If the multisig wallet is incomplete, does that mean the btc is inaccessible even though I have the cold card  signer? On Apr 30, 2024, at 1:00 AM, Daniel Kwon @.> wrote:Hello,So I didn’t realize that using two same signers (cold card and ledger) would recreate the same wallet I already have.  I decided to remove the ledger in the 1 of 2 multisig wallet setup, and sent the btc to the wallet.  Today, I tried to send the btc out of the wallet but the cold card wouldn’t sign the PSBT. When I change the multisig setup to 1 of 1 (cold card only) the btc is visible but still would sign the PSBT. If I use a different signer there is an error stating the transaction will be wiped.<image0.jpeg><image1.jpeg>On Apr 30, 2024, at 12:36 AM, craigraw @.> wrote:
It's unclear what you are doing. Your screenshot shows an incomplete multisig wallet. Then, you mention a PSBT, presumably created from somewhere else. You will need the same wallet configuration in Sparrow as that "somewhere else" that created the PSBT.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

That is this wallet.  1 of 2 wallet with only one signer established.  I guess the there is no hope to access the funds. noThanks,DanielOn Apr 30, 2024, at 1:04 AM, craigraw @.***> wrote:
I decided to remove the ledger in the 1 of 2 multisig wallet setup
This creates an entirely different wallet, with different addresses.
When I change the multisig setup to 1 of 1
Again, this creates an entirely different wallet.
You need to use (or recreate) the original wallet into which you received the funds, in order to spend it.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

The wallet address I sent btc to with 1 of 2 signors established was  bc1qnskfxeahrseh0fcqw5lwwgfsyuea877x2z852q654lzfuapsnrmsrfavvr.   I still have this wallet and when I realized that it wasn't a new wallet, I removed the ledger and sent the funds to  bc1q9wrq2rj8dr6gmlfl7f87tdp7t0s3tzexlxadjz9znf8wtgyfad5sxpq5tq . BTC is now stuck there and cannot sign the PSBT.
On Tuesday, April 30, 2024 at 01:36:08 AM PDT, craigraw<EMAIL_ADDRESS>wrote:

That is this wallet. 1 of 2 wallet with only one signer established.
If you never completed the wallet setup, you never received an address from Sparrow to deposit funds. It is technically impossible to do so.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

The funds are not stuck - you just need to recreate the wallets that you sent funds to. You have all of the information necessary to do this.
I suggest using File > New Wallet to create new wallets to do so.
Closing this off as user support.


Using UnauthenticatedError with Bun causes Segmentation Fault

Describe the bug
When calling new UnauthenticatedError bun crashes with Segmentation Fault. My current workaround is to use the "base" GraphQLError from graphql.
To Reproduce Steps to reproduce the behavior:

Use bun (I'm running 1.0.3).
Call new UnauthenticatedError("Some error")

Expected behavior
I expect it not to crash :)
Environment:

OS: MacOS
Bun 1.0.3
@envelop/core 4.0.1
@envelop/generic-auth 6.0.1

@EmrysMyrddin Sounds like a good idea, sorry for clogging :)


Option to specify output filename

Thanks for this nice tool !
Currently the output filename is non-deterministic which makes it hard to use in batch scripts.
Please provide a parameter like --out FILENAME or sth like this.

Added (--output-path):
https://github.com/artiomn/markdown_articles_tool/commit/138499a126fe884ba96de606084d31ab9fb58bbf
https://github.com/artiomn/markdown_articles_tool/commit/dc7e143bafe1f258a4c2772fb9c7e83480f402ee

Awesome, thanks !

You're welcome.


customization of the tagged language strings

I wonder if it would be possible to define some kind of mechanism that lets a user define what the language of the embeddedLanguages should map to. I ask because there are existing VSCode tooling extensions around the postgresql language for example that might be able to be lit up if the text was tagged as postgres language instead of just sql.  Are you aware of any ways to do this?

For what I've seen, the way to "customize" this is to have different patterns in a comment preceding the literal, like (* sql *) or  (* postgresql *). I used the function to make it less verbose, but we could check for a comment for extra fine-tuning. Like this extension that I was using at the beginning (but had to uninstall because it was messing with other code many times).
BTW, about embeddedLanguages, they don't really seem to work. The highlighting is just activated when defining the content of the match as text.html.derivative or source.sql.

I'm not sure to what extent embedded languages could work with interpolated strings anyway. Their tooling wouldn't know how to deal with the holes.
As for custom languages would let postgresql = sql work? Unfortunately due to a limitation of TextMate language grammars postgresql would still need added to the syntax definition though. Close?


defineVars variables are not picked up

With type: haste variable names are not matching:

@layer priority1, priority2, priority3, priority4;
@layer priority1{
:root{--x1t7srzs:#e67e22;--x1s93pyq:#34495e;}
}
@layer priority2{
.x1ghz6dp{margin:0}
.x1717udv{padding:0}
}
@layer priority3{
.x6s0dn4{align-items:center}
.x1lah574{background-color:var(--x1vswi0k)}
.xm5gh2h{color:var(--x1k3o04s)}
.x78zum5{display:flex}
.xdt5ytf{flex-direction:column}
.xif65rj{font-size:14px}
.x1pvqxga{font-size:24px}
.xo1l8bm{font-weight:400}
.xk50ysn{font-weight:500}
.xl56j7k{justify-content:center}
}
@layer priority4{
.x1dr59a3{height:100vh}
}

With CommonJS resolution, the variables are not even computed.

found the blocking issue: https://github.com/oven-sh/bun/issues/4511
the file path resolver for stylex vars relies on require.resolve, and currently the path option is not implemented on Bun

I released a new version that polyfills the require.resolve method required in the StyleX babel plugin. should work as a temporary fix, will attempt to fix the issue within Bun runtime itself.


Fix Copyright template

The template is so 2017
https://github.com/jcustenborder/connector-operator/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/operator/ConnectOperatorService.java#L2

hah. I just grabbed it off another pom. Thanks buddy!
On Sun, May 3, 2020 at 1:16 AM Jordan Moore<EMAIL_ADDRESS>wrote:

The template is so 2017
https://github.com/jcustenborder/connector-operator/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/operator/ConnectOperatorService.java#L2
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jcustenborder/connector-operator/issues/8, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AABTHABFTMDDKP3DPUY7N4DRPUD5PANCNFSM4MX7WWKQ
.


[HELP] 1 node(s) didn't have free ports for the requested pod ports

I'm trying to install istio in k3d cluster, but one of istio component(service load balancer) is failing to start with below error.
 Warning  FailedScheduling  42s (x6 over 2m59s)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

NAME                                      READY   STATUS      RESTARTS   AGE
grafana-6fb9f8c5c7-mr7vb                  1/1     Running     0          6m24s
istio-citadel-5cf47dbf7c-jxc4w            1/1     Running     0          6m24s
istio-galley-7898b587db-8jrpq             1/1     Running     0          6m25s
istio-ingressgateway-7c6f8fd795-wl6fn     1/1     Running     0          6m24s
istio-init-crd-10-8qh2j                   0/1     Completed   0          26m
istio-init-crd-11-j7glh                   0/1     Completed   0          26m
istio-init-crd-12-gvsg6                   0/1     Completed   0          26m
istio-nodeagent-clvkf                     1/1     Running     0          6m25s
istio-pilot-5c4b6f576b-2b5zf              2/2     Running     0          6m24s
istio-policy-769664fcf7-hj6bn             2/2     Running     3          6m24s
istio-sidecar-injector-677bd5ccc5-wj9zb   1/1     Running     0          6m24s
istio-telemetry-577c6f5b8c-j9dxn          2/2     Running     3          6m24s
istio-tracing-5d8f57c8ff-t7mm4            1/1     Running     0          6m24s
kiali-7d749f9dcb-w7qxr                    1/1     Running     0          6m24s
prometheus-776fdf7479-gznbs               1/1     Running     0          6m24s
svclb-istio-ingressgateway-4znth          0/9     Pending     0          6m25s

Please help me in fixing this issue.

Hey there, thanks for filing this issue.
Can you paste the full output of kubectl describe for the failing pod/deployment?

Hey @iwilltry42
Here is the output of the failing pod.

~ k describe pods svclb-istio-ingressgateway-92p9s -n istio-system
Name:           svclb-istio-ingressgateway-92p9s
Namespace:      istio-system
Priority:       0
Node:
Labels:         app=svclb-istio-ingressgateway
controller-revision-hash=597bd7b896
pod-template-generation=1
svccontroller.k3s.cattle.io/svcname=istio-ingressgateway
Annotations:
Status:         Pending
IP:
Controlled By:  DaemonSet/svclb-istio-ingressgateway
Containers:
lb-port-15020:
Image:      rancher/klipper-lb:v0.1.1
Port:       15020/TCP
Host Port:  15020/TCP
Environment:
SRC_PORT:    15020
DEST_PROTO:  TCP
DEST_PORT:   15020
DEST_IP:     <IP_ADDRESS>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f5w67 (ro)
lb-port-80:
Image:      rancher/klipper-lb:v0.1.1
Port:       80/TCP
Host Port:  80/TCP
Environment:
SRC_PORT:    80
DEST_PROTO:  TCP
DEST_PORT:   80
DEST_IP:     <IP_ADDRESS>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f5w67 (ro)
lb-port-443:
Image:      rancher/klipper-lb:v0.1.1
Port:       443/TCP
Host Port:  443/TCP
Environment:
SRC_PORT:    443
DEST_PROTO:  TCP
DEST_PORT:   443
DEST_IP:     <IP_ADDRESS>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f5w67 (ro)
lb-port-31400:
Image:      rancher/klipper-lb:v0.1.1
Port:       31400/TCP
Host Port:  31400/TCP
Environment:
SRC_PORT:    31400
DEST_PROTO:  TCP
DEST_PORT:   31400
DEST_IP:     <IP_ADDRESS>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f5w67 (ro)
lb-port-15029:
Image:      rancher/klipper-lb:v0.1.1
Port:       15029/TCP
Host Port:  15029/TCP
Environment:
SRC_PORT:    15029
DEST_PROTO:  TCP
DEST_PORT:   15029
DEST_IP:     <IP_ADDRESS>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f5w67 (ro)
lb-port-15030:
Image:      rancher/klipper-lb:v0.1.1
Port:       15030/TCP
Host Port:  15030/TCP
Environment:
SRC_PORT:    15030
DEST_PROTO:  TCP
DEST_PORT:   15030
DEST_IP:     <IP_ADDRESS>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f5w67 (ro)
lb-port-15031:
Image:      rancher/klipper-lb:v0.1.1
Port:       15031/TCP
Host Port:  15031/TCP
Environment:
SRC_PORT:    15031
DEST_PROTO:  TCP
DEST_PORT:   15031
DEST_IP:     <IP_ADDRESS>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f5w67 (ro)
lb-port-15032:
Image:      rancher/klipper-lb:v0.1.1
Port:       15032/TCP
Host Port:  15032/TCP
Environment:
SRC_PORT:    15032
DEST_PROTO:  TCP
DEST_PORT:   15032
DEST_IP:     <IP_ADDRESS>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f5w67 (ro)
lb-port-15443:
Image:      rancher/klipper-lb:v0.1.1
Port:       15443/TCP
Host Port:  15443/TCP
Environment:
SRC_PORT:    15443
DEST_PROTO:  TCP
DEST_PORT:   15443
DEST_IP:     <IP_ADDRESS>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f5w67 (ro)
Conditions:
Type           Status
PodScheduled   False
Volumes:
default-token-f5w67:
Type:        Secret (a volume populated by a Secret)
SecretName:  default-token-f5w67
Optional:    false
QoS Class:       BestEffort
Node-Selectors:
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/pid-pressure:NoSchedule
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type     Reason            Age                  From               Message

Warning  FailedScheduling  82s (x7 over 5m18s)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

Alright, so it seems like one of the Host Port's is already blocked by another pod.
Anyway, this is not a component of istio but an automatic deployment coming from k3s.

Can you provide more details on
a) how you created the cluster (k3d command)
b) how you deployed istio
please? Just so I can replicate it :+1:

Any news on this @harshavardhanc ?

experiencing same issue:
mac-os: 10.4.6
docker-desktop: <IP_ADDRESS>
wget -q -O - https://raw.githubusercontent.com/rancher/k3d/master/install.sh | bash
export KUBECONFIG=$(k3d get-kubeconfig)

curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.3.0 sh -
cd istio-1.3.0/
export PATH=$PWD/bin:$PATH
for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done
kubectl apply -f install/kubernetes/istio-demo-auth.yaml

bash-4.4$ kubectl get po -n istio-system
NAME                                      READY   STATUS      RESTARTS   AGE
grafana-6fc987bd95-pvg9j                  1/1     Running     1          6h57m
istio-citadel-679b7c9b5b-rmqt6            1/1     Running     1          6h57m
istio-cleanup-secrets-1.3.0-wwnfr         0/1     Completed   0          6h57m
istio-egressgateway-5db67796d5-msz5n      1/1     Running     1          6h57m
istio-galley-7ff97f98b5-n5zng             1/1     Running     1          6h57m
istio-grafana-post-install-1.3.0-mfbnm    0/1     Completed   0          6h57m
istio-ingressgateway-859bb7b4-24l9p       1/1     Running     1          6h57m
istio-pilot-9b9f7f5c8-99mj9               2/2     Running     2          6h57m
istio-policy-754cbf67fb-6x9dl             2/2     Running     7          6h57m
istio-security-post-install-1.3.0-7bh9n   0/1     Completed   0          6h57m
istio-sidecar-injector-68f4668959-274mv   1/1     Running     1          6h57m
istio-telemetry-7cf8dcfd54-tnnbq          2/2     Running     8          6h57m
istio-tracing-669fd4b9f8-gsqm5            1/1     Running     1          6h57m
kiali-94f8cbd99-gfgzl                     1/1     Running     1          6h57m
prometheus-776fdf7479-kv95j               1/1     Running     1          6h57m
svclb-istio-ingressgateway-bkpw8          0/9     Pending     0          6h57m

bash-4.4$ kubectl describe pod svclb-istio-ingressgateway-bkpw8 -n istio-system
Name:               svclb-istio-ingressgateway-bkpw8
Namespace:          istio-system
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             app=svclb-istio-ingressgateway
                    controller-revision-hash=688bbd58b
                    pod-template-generation=1
                    svccontroller.k3s.cattle.io/svcname=istio-ingressgateway
Annotations:        <none>
Status:             Pending
IP:
Controlled By:      DaemonSet/svclb-istio-ingressgateway
Containers:
  lb-port-15020:
    Image:      rancher/klipper-lb:v0.1.1
    Port:       15020/TCP
    Host Port:  15020/TCP
    Environment:
      SRC_PORT:    15020
      DEST_PROTO:  TCP
      DEST_PORT:   15020
      DEST_IP:     <IP_ADDRESS>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z58mp (ro)
  lb-port-80:
    Image:      rancher/klipper-lb:v0.1.1
    Port:       80/TCP
    Host Port:  80/TCP
    Environment:
      SRC_PORT:    80
      DEST_PROTO:  TCP
      DEST_PORT:   80
      DEST_IP:     <IP_ADDRESS>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z58mp (ro)
  lb-port-443:
    Image:      rancher/klipper-lb:v0.1.1
    Port:       443/TCP
    Host Port:  443/TCP
    Environment:
      SRC_PORT:    443
      DEST_PROTO:  TCP
      DEST_PORT:   443
      DEST_IP:     <IP_ADDRESS>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z58mp (ro)
  lb-port-31400:
    Image:      rancher/klipper-lb:v0.1.1
    Port:       31400/TCP
    Host Port:  31400/TCP
    Environment:
      SRC_PORT:    31400
      DEST_PROTO:  TCP
      DEST_PORT:   31400
      DEST_IP:     <IP_ADDRESS>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z58mp (ro)
  lb-port-15029:
    Image:      rancher/klipper-lb:v0.1.1
    Port:       15029/TCP
    Host Port:  15029/TCP
    Environment:
      SRC_PORT:    15029
      DEST_PROTO:  TCP
      DEST_PORT:   15029
      DEST_IP:     <IP_ADDRESS>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z58mp (ro)
  lb-port-15030:
    Image:      rancher/klipper-lb:v0.1.1
    Port:       15030/TCP
    Host Port:  15030/TCP
    Environment:
      SRC_PORT:    15030
      DEST_PROTO:  TCP
      DEST_PORT:   15030
      DEST_IP:     <IP_ADDRESS>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z58mp (ro)
  lb-port-15031:
    Image:      rancher/klipper-lb:v0.1.1
    Port:       15031/TCP
    Host Port:  15031/TCP
    Environment:
      SRC_PORT:    15031
      DEST_PROTO:  TCP
      DEST_PORT:   15031
      DEST_IP:     <IP_ADDRESS>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z58mp (ro)
  lb-port-15032:
    Image:      rancher/klipper-lb:v0.1.1
    Port:       15032/TCP
    Host Port:  15032/TCP
    Environment:
      SRC_PORT:    15032
      DEST_PROTO:  TCP
      DEST_PORT:   15032
      DEST_IP:     <IP_ADDRESS>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z58mp (ro)
  lb-port-15443:
    Image:      rancher/klipper-lb:v0.1.1
    Port:       15443/TCP
    Host Port:  15443/TCP
    Environment:
      SRC_PORT:    15443
      DEST_PROTO:  TCP
      DEST_PORT:   15443
      DEST_IP:     <IP_ADDRESS>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z58mp (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  default-token-z58mp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-z58mp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason            Age                     From               Message
  ----     ------            ----                    ----               -------
  Warning  FailedScheduling  5m12s (x96 over 6h57m)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
  Warning  FailedScheduling  26s (x6 over 4m54s)     default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

I'm pretty sure, that ports 80 and 443 are already taken by traefik.

Sorry for late reply @iwilltry42 was OOO, I was using k3d create --name .
You are right @iwilltry42, then I used this command without traefik k3d create --server-arg --no-deploy --server-arg traefik --name

So it works without traefik? Can I go ahead and close this issue then @harshavardhanc ? :+1:

@iwilltry42 Any idea why this happens? I think it'll be good to have this in the doc, in case somebody get blocked.

yes it works without traefik @iwilltry42

+1 @rjshrjndrn

@rjshrjndrn yep, it's because of the Service Load Balancer, which reacts to services of type: LoadBalancer.
See the related k3s documentation: https://rancher.com/docs/k3s/latest/en/configuration/#service-load-balancer
If you don't need or want this feature, start the cluster with --server-arg '--no-deploy servicelb'.
Note: the pod that stays in Pending state is part of the k3s infrastructure, not part of the istio manifests/chart which you deployed.

@iwilltry42 I always run a k3d k3d with one node and without traefik.
For type: LoadBalancer, I always gets an ip. Usually I install istio for ingress and tinker over it.

Addition: Do both of you have a k3d cluster created with only a single node? (since the the controller tries to find a node, where the ports are free and obviously, there's none in a single node cluster, where traefik is already running and has the ports occupied)

Why can't traefik run in the same node. I don't thinks there's any toleration for traefik to run in the master itself.
Note: I tired the cluster with traefik and in one node, for me it works perfectly fine.

@rjshrjndrn , I'm not sure, I understand you correctly there.
But the issue, shown in the outputs posted here, say that there's a pod svclb-istio-ingressgateway-abcd stuck in Pending state.
This pod is not spawned by the istio manifests, it's spawned by a controller that is part of k3s. This controller is similar to MetalLB and for every kind: Service of type: LoadBalancer that you create in the cluster, it tries to find a node, where it can map the requested port from the node to the pod.
Now, if you create a cluster without the --no-deploy=traefik flag, you'll already have a pod svclb-traefik-abcd with two containers, which use hostPort: 80 and hostPort: 443 (meaning, that ports 80 and 443 on the node are in use).
Unfortunately, the svclb-istio-ingressgateway now needs exactly the same ports, but since those are already taken, it's stuck in Pending state.

Okay. Thank you @iwilltry42 for clarification. Makes sense now.
Basically it's clash of ingress, right !!!

Are there any questions left here or can I close this issue? :+1:

You can close this issue @iwilltry42

anybody here. I have an issue and want discuss it

hi @Umair841 is it something related to port mapping ?

yes sir. actually, I am using host port mapping with the container port in pods so when my new pod spins up it does not get in a ready state because that port is already taken by another port .. so is there any way so that my new pod gets the host port dynamically while getting up ?? can we add a list of ports in PORT value in service ???

this is the error actually """"1 node(s) didn't have free ports for the requested pod ports, 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
"""

can you gimme the command you use to create the cluster.
Note for code block use backquotes three times.

```
This is my code.
```

Sir, actually I am new to k8s and this is the first take I have given to troubleshot the error and resolve this.
I am creating a cluster with kops on aws .
command is
kops create a cluster --name umair.k8s.local --zones eu-central-1a --master-size t2.micro --master-count 1 --node-count 1  --node-size t2.micro --Kubernetes-version 1.15.0
and here is the deployment YAML file .

apiVersion: v1
kind: Service
metadata:
name: compliancex-rabbitmq
spec:
selector:
app: compliancex-rabbitmq
ports:
- name: tcp
port: 80
port: 90
targetPort: 80
apiVersion: apps/v1
kind: Deployment
metadata:
name: compliancex-rabbitmq
spec:
selector:
matchLabels:
app: compliancex-rabbitmq
replicas: 1
template:
metadata:
labels:
app: compliancex-rabbitmq
spec:
hostNetwork: true
containers:
- name: compliancex-rabbitmq
image: nginx:stable-alpine
ports:
- containerPort: 80

now I copied this file in the same directory with another name to create a conflict of host ports. now is there any way so that both of the pods gets ready by assigning different host port to both of them .
thanks in advance

this is the deployment YAML actually.
apiVersion: v1
kind: Service
metadata:
name: compliancex-rabbitmq
spec:
selector:
app: compliancex-rabbitmq
ports:

name: tcp
port: 80
targetPort: 80
apiVersion: apps/v1
kind: Deployment
metadata:
name: compliancex-rabbitmq
spec:
selector:
matchLabels:
app: compliancex-rabbitmq
replicas: 1
template:
metadata:
labels:
app: compliancex-rabbitmq
spec:
hostNetwork: true
containers:
name: compliancex-rabbitmq
image: nginx:stable-alpine
ports:
containerPort: 80

is there any way to assign random host port automatically to the pod while it is getting up or can we give a list of ports as a value to the PORT in service ??

won't able to help you unless you provide proper yaml. And I asked the command which you used to create the k3d cluster.

is there any way to assign random host port automatically to the pod while it is getting up or can we give a list of ports as a value to the PORT in service ??

Nope. If you're using linux, you should be able to use the loadbalancer ip from the host machine, as the docker network is shared.
Means when you do kubectl get service -n namespace the ip address you get, try with that ip:port combo

what thoughts you have sir on "dynamic port binding" ??

I don't think that's possible.

Hi @Umair841 , I moved your question to the discussions feature of this repo, please continue over there to add more details: https://github.com/rancher/k3d/discussions/551
@rjshrjndrn , thanks for helping out here :)
If I understood it correctly, you're trying to use hostNetwork: true with a dynamically assigned containerPort, which won't work anyway, as due to the hostNetwork: true stanza, all of the ports that your application exposes are opened on the host. So you would have to reconfigure that app to listen on a different port.
In your case it's most probably not working, because it's listening on port 80 (or at least, that's what's specified in the containerPort), which is like the most common port and most probably already used by the Ingress controller.
Please also note, that we cannot give support for things you try with kops, as this is the k3d repo :wink:
Answers in the linked discussion please :+1:

with the new version of k3d, argument name changed - was able to make istio's ingressgateway working by passing --k3s-server-arg '--no-deploy=traefik'

On k3d version 5 flag is --k3s-arg '--no-deploy=traefik@server:*'


basic/Tutorial dependencies

The pip command used in the first basic/Tutorial notebook is malformed. See image below.

I can submit a PR fixing this, however, I can take the opportunity to improve it a little.

Using --upgrade when installing multiple packages together can lead to long install times because of pip backtracking. I stopped it after ~30 minutes.

I would suggest talking out --upgrade and using version numbers instead. These are the latest version numbers, and I've tested them in my local environment and SageMaker Notebooks too.

pip3 install boto3>=1.18.42 pandas>=1.3.3 s3fs>=0.4.2 matplotlib>=3.4.3 ipywidgets>=7.6.5

Also, it's not advisable to use !pip install <package> in Jupyter notebooks as it might not install packages in the same execution environment used by the notebook. For example, I have to use pip3 instead for this to work.

The code snippet below is how you can work around the different execution environments (along with version numbers).

import sys
!{sys.executable} -m pip install boto3>=1.18.42 pandas>=1.3.3 s3fs>=0.4.2 matplotlib>=3.4.3 ipywidgets>=7.6.5

I've tested it on my local environment, I'll run it on a SageMaker Notebook too to verify the changes are good. If you're happy with the changes, I can submit a PR. :slightly_smiling_face:

Good catches, if validate the functionality in SM it will definitely be merged in. That's the default assumption for an environment given the educational use of these notebooks in labs and workshops.
Thanks!