Get and implement windows code signing certificate

From @rfay on October 4, 2017 23:16
What happened (or feature request):
To send out trusted install packages for Windows we'll need a code signing certificate which must be purchased.  Link to that page on https://www.electron.build/code-signing#where-to-buy-code-signing-certificate
There seem to be a variety of prices from the various vendors.
What you expected to happen:
How to reproduce this:
Version: Please include the output of ddev version, docker version and the project's .ddev/config.yaml.
Anything else do we need to know:
Related source links or issues:
Copied from original issue: drud/ddev-ui#34

From @andrew-c-tran on October 5, 2017 23:23
According to electron-builder documentation (https://www.electron.build/code-signing#windows) we need either:

EV Code Signing Certificate
Code Signing Certificate

a code signing certificate will display a warning to the user until enough users have installed the app to build trust, at which point it'll stop showing to new installs. an EV code signing cert is more expensive, but will not show a warning ever. The drawback of the EV code signing cert though is that it's bound to a physical USB dongle, and thus will need to be physically attached to the machine that's doing the signing. This means no circle builds, so i'm not sure this route will work.
According to Microsoft, the certificate can be bought from a few vendors, such as DigiCert and Symantec:
https://www.digicert.com/friends/sysdev/
https://products.websecurity.symantec.com/orders/enrollment/winqualOrder.do
the EV Certificate can be bought from them as well:
https://trustcenter.websecurity.symantec.com/process/trust/productOptions?productType=EVCodeSigning&flavor=MSAuthentiCode&application_locale=VRSN_US&newLanguage=en
https://www.digicert.com/friends/sysdev/
We'll need to make a decision on the following points:

EV vs Standard Code Signing Certificate
Which Vendor to buy from (Symantec, Certum, Entrust, GlobalSign, DigiCert)

From @andrew-c-tran on October 10, 2017 14:38
any input @rickmanelius / @cyberswat ? The sooner we get an application in, the sooner we can get signing. I'm not sure how long the approval process is, but on mac it's quite the wait from what i've read.

From @rickmanelius on October 10, 2017 14:58
No input other than Kevin and I had overlapping PTO and I'm still catching up. Tagging as a blocker once https://github.com/drud/general/issues/137 gets in and we can triage/prioritize from there.

From @cyberswat on October 10, 2017 15:51
@rickmanelius just need to add this to the budget and proceed.  It's $449 per year if you buy it through digicert and pay 1 year at a time. $399x2 if you do it for 2 years, and $331x3 if you do it for 3 years.  The process for this is lengthy and will take a couple of weeks.  Let me know when we should start this process.

From @rickmanelius on October 10, 2017 15:55
Let's do one year!

From @andrew-c-tran on October 10, 2017 15:57
@cyberswat which cert are you looking at? The standard Code Signing Certificate is showing up at $267 for 3 years ($89 a year) and the EV Certificate at $497 for 3 years ($165.67 a year) through digicert for me?

From @cyberswat on October 10, 2017 16:4
https://www.digicert.com/code-signing/ev-code-signing/ ... we started the order focusing on the per year cost.  Hopefully let's encrypt let's us sign these by the time next year rolls around.

From @rickmanelius on October 10, 2017 16:33
Had a quick call w/@andrew-c-tran and @cyberswat. We'll just eat the cost of the EV and get the standard. Kevin may be following up with the need to get additional information.

From @andrew-c-tran on October 10, 2017 16:16
We might not want the EV cert. The EV cert is on a physical dongle and that means we will need a dedicated machine to sign our installers.
Also, MS directly linked me to https://www.digicert.com/friends/sysdev/ , which gets us 50% off as a developer. Not sure who would be buying a cert besides a developer, ha.

From @andrew-c-tran on December 5, 2017 22:42
@cyberswat did we ever get this cert?

From @cyberswat on January 31, 2018 23:59
Placed an order for this. They are validating the address of the company.

From @cyberswat on February 1, 2018 20:44
@rickmanelius We changed our phone number which is leading to this response from them.  I'm not quite sure how to deal with this right now so I'm recording it here.
I am trying to finish the validation of your Code Signing certificate order for DRUD Technology, LLC and I need your help with just one thing:
To verify your phone number, please send a document such as a utility bill (telephone, gas, electric, water, or internet), bank statement, rental agreement, or any government issued document. You may block out any sensitive information as long as it shows your organization name (DRUD Technology, LLC) and phone number<PHONE_NUMBER>).
Send this document to<EMAIL_ADDRESS>or fax it to +1-801-705-0481. Write "Validation #02558320" in the subject line.
Once we verify this, we will move to the next step of the validation process.

From @cyberswat on February 1, 2018 21:35
The address is on the bank statements, but not the phone number!

From @cyberswat on February 5, 2018 19:12
@pgalligan80126 Would our phone bill satisfy the conditions outlined above? Can you send me a copy outside of this issue if so?

@cyberswat Could you please let me know how this ended up? We need this for ddev Windows installer.

Moved this to ddev project, it's critical for our windows installer (although we can tell users about the workaround in the meantime). I scanned the issue for sensitive info and found none, so think it's ok to be in this public repo.

A TYPO3 user pointed out https://www.certum.eu/certum/cert,offer_en_open_source_cs.xml - Special cert offer for open-source projects.

That link is about $100. I will plan to take that path and expense it if we don't have progress on the previous approaches during this release cycle.

Honestly, taking that approach will involve the least number of moving parts and just get it done... I'm fine with it.

@rfay you should do that if it moves this forward.  If that doesn't work I'll change the number so that I can answer it and then resubmit the order.

The open source cert above says "OPEN SOURCE CODE SIGNING CERTIFICATES ARE ISSUED ONLY TO NATURAL PERSONS!", so that means it couldn't be signed by DRUD Tech. Also, with courier cost, this ends up being about $142. Also, this is a not-too-fancy outfit in Poland that's issuing.
Do we want to go for it and just see how it comes out? I'd have to provide my own authenticating info like passport and stuff, and it would then be signed by me, as I understand it.
There's also no complete assurance that I'll be able to figure out how to sign the nsis installer.

Buying this would also not be useful for ddev-ui assuming its eventual release, if it's not open-sourced.

I think we have the correct certificate in Randy's hands.  It is a Microsoft Authenticode Code Signing Certificate. It looks like these instructions provide a command line example of using it with an additional tool.

Yay, thanks @cyberswat ! Will be trying it over in https://github.com/drud/ddev/issues/840


Add docs for kapp-controller config and startup

Please don't merge until we release v0.14.0 of kapp-controller.

Please don't merge until we release v0.14.0 of kapp-controller.

Please don't merge until we release v0.14.0 of kapp-controller.

merging since docs clarify which version this feature is available in.

Please don't merge until we release v0.14.0 of kapp-controller.

merging since docs clarify which version this feature is available in.


panic: runtime error: slice bounds out of range [:13] with capacity 1

Describe the bug
Recently we started see this kind of issues:
2022-07-08 01:31:42  Initializing Defect Types...
2022-07-08 01:31:43  panic: runtime error: slice bounds out of range [:13] with capacity 1
2022-07-08 01:31:43
2022-07-08 01:31:43  goroutine 27 [running]:
2022-07-08 01:31:43  github.com/RedHatQE/tfacon/connectors.processREReturnedText(0x0, 0x0, 0x8db5c3, 0x6)
2022-07-08 01:31:43  	/home/runner/work/tfacon/tfacon/connectors/rp_connector.go:342 +0x42f
2022-07-08 01:31:43  github.com/RedHatQE/tfacon/connectors.(*RPConnector).GetREResult(0xc0000ee400, 0xc00037794f, 0x26, 0xc00037794f, 0x26)
2022-07-08 01:31:43  	/home/runner/work/tfacon/tfacon/connectors/rp_connector.go:333 +0x271
2022-07-08 01:31:43  github.com/RedHatQE/tfacon/connectors.(*RPConnector).BuildIssueItemHelper(0xc0000ee400, 0xc0001df312, 0x6, 0x101, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
2022-07-08 01:31:43  	/home/runner/work/tfacon/tfacon/connectors/rp_connector.go:298 +0x6fd
2022-07-08 01:31:43  github.com/RedHatQE/tfacon/connectors.(*RPConnector).BuildIssueItemConcurrent(0xc0000ee400, 0xc00036c0c0, 0xc00036c120, 0xc000370150, 0xc000320101)
2022-07-08 01:31:43  	/home/runner/work/tfacon/tfacon/connectors/rp_connector.go:369 +0x72
2022-07-08 01:31:43  created by github.com/RedHatQE/tfacon/connectors.(*RPConnector).BuildIssuesConcurrent
2022-07-08 01:31:43  	/home/runner/work/tfacon/tfacon/connectors/rp_connector.go:256 +0x16b

I see there is big timeframe for those error, so not sure if it might be related to model restart?
Few links:
https://url.corp.redhat.com/91e61ed
https://url.corp.redhat.com/45daed0
https://url.corp.redhat.com/e403ff3
To Reproduce
When I tried to re-produce now by re-triggering one of the failed job it passed ok. E.g. here:
https://url.corp.redhat.com/9235eae
https://url.corp.redhat.com/2e07e7c
Expected behavior
To update the predictions and recommendations
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

looking at it

I hit this same issue if run tfacon with -r:
Initializing Defect Types...
panic: runtime error: slice bounds out of range [:13] with capacity 1

goroutine 13 [running]:
github.com/RedHatQE/tfacon/connectors.processREReturnedText(0x0, 0x0, 0x8db5c3, 0x6)
	/home/runner/work/tfacon/tfacon/connectors/rp_connector.go:342 +0x42f
github.com/RedHatQE/tfacon/connectors.(*RPConnector).GetREResult(0xc000192400, 0xc00032f24d, 0xc, 0xc00032f24d, 0xc)
	/home/runner/work/tfacon/tfacon/connectors/rp_connector.go:333 +0x271
github.com/RedHatQE/tfacon/connectors.(*RPConnector).BuildIssueItemHelper(0xc000192400, 0xc0000b46f2, 0x4, 0x101, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/runner/work/tfacon/tfacon/connectors/rp_connector.go:298 +0x6fd
github.com/RedHatQE/tfacon/connectors.(*RPConnector).BuildIssueItemConcurrent(0xc000192400, 0xc00007f200, 0xc00007f260, 0xc000364cb0, 0xc000400101)
	/home/runner/work/tfacon/tfacon/connectors/rp_connector.go:369 +0x72
created by github.com/RedHatQE/tfacon/connectors.(*RPConnector).BuildIssuesConcurrent
	/home/runner/work/tfacon/tfacon/connectors/rp_connector.go:256 +0x16b

Run tfacon without -r looks normal.

Hit again here:
https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-tfac/1630/console
@JunqiZhang0 any update here please?

Next reproduce:
https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-tfac/1646//console
:(

We are facing this problem as well.

Fixing this

fixed by the latest release


Update COBOL Programming with VSCode.md

2nd attempt at fixing links

First, thank you for changing it into a markdown! this is easier for me to read.
In my opinion, maybe it is just me, it would be easier to name the chapters as
VSCode with Z Open Editor and ZOWE extensions rather than
Chapter 2. VSCode with Z Open Editor and ZOWE extensions
Because if, for example, I would insert a new Chapter (e.g. Zowe Explorer Basics) then I need to revise all the Chapter numbers.
The Parts are good. I think they separate the document well but the chapters are tricky if I'm an editor

Thank you @jellypuno yes we are looking into the best way to handle chapter numbering in markdown.


Can we change the tableName for tests to lowercase only?

Hi there,
The reason I am asking is because I am writing an adapter for couchdb, and there is no collection in couchdb (not like mongo). So in order to write an adapter for couchdb, I have to create databases to use for table/collection. But currently, the rule for creating databases of couchdb is "Only lowercase characters (a-z), digits (0-9), and any of the characters _, $, (, ), +, -, and / are allowed.
Must begin with a letter.". So I have difficulty to run integration tests, becuase the tableName is in carmelcase, ex: userTable, userTable1, ...
So my question is can we change the tableName to lowercase only?

@Powpow-Shen, is this still an issue? https://github.com/tjwebb/sails-couchdb-orm/commit/39a34386ed5f00c8a43d0e70241370768d1b2bb4 looks like a good solution for this.

Closing this as unresponsive, please reopen if you want to resume the issue/discussion. Thanks


Show detail table when clicking a bar/dot in the charts

Add the ability to click on a bar (or dot) to open a detailed listing of that subset of data records.

This is a fairly complex change, but I think it's worth doing. Inclined to hit the low-hanging fruit in v1.1.0 first, and this do this, so going to put this by itself in v1.2 for now.
It might be worth thinking about it in conjunction with #59 since both likely involve a new "highlighted" data layer.

Might need to make a highlighted data object for this.
Disable filters in some way.
Need to figure out how to allow highlighting.  Sliders?  Mock-up and discuss as a team.
High priority because DMEDS folks mentioned this being a solution for issues they are having.

Technical Specs
When a mark is clicked do the following:

Give the mark the 'highlighed' class. (Clear the highlighed class from all other marks.)
Assign the mark's raw data to codebook.data.highlighted. (Overwrite existing data if any)
Show a "clear highlights" button to the controls followed by a note defining the current highlighed population ("[X] records with [Variable X] values of [X to Y] highlighted"). When clicked the button should trigger a 'clearHighlights' method that does the following:

clear codebook.data.highlighted
clear the highlighed class from amrks
delete the note and the button
clear highlights whenever the group control or a filter change

Consider implementing with the following new methods:

codebook.controls.highlight.init(codebook) – creates hidden “clear highlight” button and a null codebook.data.highlight object
codebook.controls.highlight.update(codebook) – initializes highlights. expects a non-null codebook.data.highlight object.
codebook.controls.highlight.clear(codebook) – Clears highlights
chart.{chartType}.highlight() - assigns the 'highlighted' class to a mark and calls codebook.controls.highlight.update(). Triggered via new event listeners initialized in the onResize() for each of the individual charts.

Test Notes: Combined test notes for #97, #59 and #142.
Clicking a bar in the codebook bar chart should trigger the following:

The number of highlighted records should be noted in the controls title next to the number of rows selected.
The bar should become orange indicating that it is selected.
Orange bars corresponding to data in the selected bar should be drawn in all other bar charts. So, if a large bar is selected, more data will be highlighted in the other charts.
The rows for the highlighted records should have an orange background in the data listing view, and they should be shown first in the listing
Sorting the data listing should work as expected (rows should still be orange, but won't neccesarily be first)
Clicking a second bar should override the original selection and change the highlighted data
Clicking the "Clear Highlighting" button should remove all highlight functionality.
Filtering the data, selecting a grouping variable or changing a setting on the settings page should   reset the highlight

The functionality described above should work for all bar chart types, with and without group by variables selected. Test by clicking bars from all of the following:

Histogram, grouped
Histogram, ungrouped
Vertical bar chart, grouped
Vertical bar chart, ungrouped
Horizontal bar chart, grouped
Horizontal bar chart, ungrouped

Pass


[Configuration] Filtering registers based on resources attributes

Is your feature request related to a problem? Please describe.

When a CHSS logs into their application, they should only have access to data that is from the CHAs under them. They should also be able to view a list of only their CHAs on the CHA register.

Describe the solution you'd like

[ ] When a CHSS opens the patient data registers from the CHA profile the records displayed should be filtered by the. CHA (Practitioner ID)

[ ] All the records created by the app are tagged by the practitioner Id on the meta.tag attribute.
[ ] Write a filter that can accept expressions (probably) FHIR path to filter the resources displayed on the register by the clicked practitioner Id.

This work by @SebaMutuku filters resources in a LIST view by specific FHIR path expressions. I think we can extend it to filter all the registers.

Describe alternatives you've considered
N/A
Additional context

A CHSS acts as a supervisor to the CHA. A CHSS is normally assigned 5-10 CHAs under them. CHSS is responsible for reviewing the CHA work and thus needs to see each CHA's updated data after P2P or a sync.

@dubdabasoduba @AnnieMungai
since CHSS should only see the work from CHAs, is it possible to sync based on the related CHAs?

Couldn't we handle this by using CareTeam sync as is? If we assign the CHSS to all of the CareTeams any of its CHAs are in, wouldn't that handle this?

Couldn't we handle this by using CareTeam sync as is? If we assign the CHSS to all of the CareTeams any of its CHAs are in, wouldn't that handle this?

@pld @FikriMilano The CHSS device is used as a data store. These devices hold the data from all the CHAs supervised by the CHSS. It will occasion also hold data for other CHAs not supervised by the CHSS. This is because the CHSS is allowed to P2P sync will other CHAs in order to transmit their data to the server.
With the above context, a CHSS is allowed to view a replica of the registers on the CHAs device.  This issue is to make sure the data shown on these CHA replica registers is data that belongs to the CHA in question.
e.g if the CHSS selects CHA 1 and tries to view the Household register for CHA 1 then only families tied to CHA 1 should show up as much as the CHSS device has household data from other CHAs.


Angular cli compile (ng serve) time very slow (278309 ms)

And recompile  time takes  143555 ms

Thanks for reporting this issue. However, you didn't provide sufficient information for us to understand and reproduce the problem. Please check out our submission guidelines to understand why we can't act on issues that are lacking important information.
If the problem persists, please file a new issue and ensure you provide all of the required information when filling out the issue template.


Suggestion: Manual titan digger checkbox

As i'm pushing through the Titan 6 versions it occurred to me a quick toggle box on the UI would be appreciated to ensure you have Exp, PP, Adv, and DC diggers active while manually fighting a titan (or i guess in ITOPOD for the first 3).
I often forget the DC digger when i go to fight it manually >.>

Addendum to previous idea:
Label box as "Manual Titan Settings"
Respects regular titan script, and if selected for that particular titan will swap to gear listed in titan tab

Even an auto kill titan custom digger setup would be very necessary if I am still farming some gear from the titan. I want DC digger to be active just before the kill. If I am doing ITOPOD the rest of the time, I don't need DC then.

Even an auto kill titan custom digger setup would be very necessary if I am still farming some gear from the titan. I want DC digger to be active just before the kill. If I am doing ITOPOD the rest of the time, I don't need DC then.

Swapping to DC digger is part of the titan swap process already. @trulypoison you could consider using the Quick Loadout/Quick Digger option which is in the settings.json file. F8 swaps to the config you specify, and then F8 swaps back after

Swapping to DC digger is part of the titan swap process already. @trulypoison you could consider using the Quick Loadout/Quick Digger option which is in the settings.json file. F8 swaps to the config you specify, and then F8 swaps back after

@rvazarkar Thanks. I found where it's hardcoded in the source. I think it's fine as it is as those 4 diggers are the only mandatory ones for Titan auto-kill.

@rvazarkar Thanks. I found where it's hardcoded in the source. I think it's fine as it is as those 4 diggers are the only mandatory ones for Titan auto-kill.


Fix dashboard copying

Issue
When skeletonizing the dashboard copying, I got sidetracked with log labeling and forgot to go back to adjust the formatting.
Use json.dumps, not yaml.dump. Dashboards are JSON.
Give the dashboards the correct extension, and remove parens around the string format so the filename doesn't have oddly escaped characters.

Closed without merge?

Duplicate PR. @Abuelodelanada merged the same thing in #139 -- we just ended up writing PRs at exactly the same time, and I closed this one because it's basically a NOOP now.


elasticsearch_data will not start on v206.x

Hey
When updating to v206.x we see the following error on the elasticsearch_data nodes:
2017-04-19 09:12:44,311 main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBeanTrustPermission" "register")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:585)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanTrustPermission(DefaultMBeanServerInterceptor.java:1848)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:322)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
        at org.apache.logging.log4j.core.jmx.Server.register(Server.java:389)
        at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:167)
        at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:140)
        at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:541)
        at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:258)
        at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:117)
        at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:84)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
        at org.elasticsearch.cli.Command.main(Command.java:88)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84)

The ctl file looks like this: https://gist.github.com/janaurka/598c3e6a833e8d4d03803534ed45ade7
The manifest like this:
- azs:
  - az1
  - az2
  - az3
  instances: 3
  jobs:
  - name: elasticsearch
    properties:
      elasticsearch:
        cluster_name: logsearch
        drain: false
        health:
          disable_post_start: true
        heap_size: 30G
        master_hosts:
        - <ip>
        - <ip>
        - <ip>
        node:
          allow_data: true
          allow_master: false
    release: logsearch
  - name: syslog_forwarder
    properties:
      syslog_forwarder:
        config:
        - file: "/var/vcap/sys/log/elasticsearch/elasticsearch.stdout.log"
          service: elasticsearch
        - file: "/var/vcap/sys/log/elasticsearch/elasticsearch.stderr.log"
          service: elasticsearch
    release: logsearch
  name: elasticsearch_data
  networks:
  - name: dynamic
  persistent_disk_type: 400gb
  stemcell: trusty
  update:
    max_in_flight: 3
  vm_type: 8cpu_32gbram_20gbdis

This does not happen on v205.x.
We run on bosh Version    261.4.0 (00000000) and use the latest vsphere_cpi.

I was able to debug this.
The problem is that /var/vcap/packages/elasticsearch belongs to root and the user vcap cannot access some data.
I’ll create a PR with a fix in some minutes.

Hi @janaurka, i guess that Elasticsearch trying to use jvm.options config which stored in Elastcsearch package dir, we can add jvm.options file to job config and set ES_JVM_OPTIONS to the path of the file.

@Infra-Red: yeah, I guess you are correct.
I’ve worked-around with a chown -R vcap:vcap /var/vcap/packages/elasticsearch/ but a ps then looks like:
elasticsearch_data/aef1a704-d826-4273-9c8a-4e0df8b29a17:~# ps aux |grep java
vcap     22151 10.6 98.3 38370568 32415064 ?   S<Ll 11:48   6:26 /var/vcap/packages/java8/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Xms30G -Xmx30G -Djava.io.tmpdir=/var/vcap/sys/tmp/elasticsearch -Des.path.home=/var/vcap/packages/elasticsearch -cp /var/vcap/packages/elasticsearch/lib/elasticsearch-5.3.0.jar:/var/vcap/packages/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/vcap/sys/run/elasticsearch/elasticsearch.pid

check the -Xms and the -Xmx which shows up twice. Once the setting from the jvm.options in the packages dir and once the setting I’ve provided.

@Infra-Red: My experience with those jvm.options is relatively small. Is this something you can do easily? I would have to look up every setting since I am really no JVM specialst...

@jabrena Yes, i can handle it. Thanks for submitting the issue!

Fixed https://github.com/cloudfoundry-community/logsearch-boshrelease/commit/4508d9acab0dfe3a43af57cfa4fe3d885f70e47b.

fyi: I’ve applied the patch and looks nice. great work, thx!


DRAFT: refactor(entities,resources): Decouple parsing from entities hook into a separate serializer class

create CamelResourceSerializer interface and YamlCamelResourceSerializer implementantion
create CamelResourceFactory and CamelKResourceFactory to creating new resources
move parsing out of the entities hook

Thanks @mmelko for putting this together. I'm turning this PR into a draft for now, so we don't consume Chromatic credits until we're ready to merge :smile:

@lordrip thanks for the review! Updated code coming soon.


Error when combining multiple one-qubit gates into a single one-qubit gate

Information

Qiskit Terra version: 0.7.3
Python version: 3.7
Operating system: Linux/MacOSX

What is the current behavior?
The circuit of the shor algorithm in the tutorial cannot be compiled for real-device backends because of error in combining multiple one-qubit gates into a single one-qubit gate.
See the details here:
https://github.com/Qiskit/qiskit-tutorials/issues/536
It can be fixed if we turn off the pass for optimizing one-qubit gates.
Steps to reproduce the problem
Run the circuits in this tutorial on real devices:
https://github.com/Qiskit/qiskit-tutorials/blob/master/community/algorithms/shor_algorithm.ipynb
What is the expected behavior?
Compilation succeeds without error.
Suggested solutions
Turn off optimizing one-qubit gates if it produces error.

I'll look into it but that circuit won't run on a device anyway. It has a reset.


cmd: fix help doc and run bug

fix help doc
fix compiler's complain: components/package/main.go:189:10: if block ends with a return statement, so drop this else and outdent its block
fix tiup run playground

Codecov Report

Merging #54 into master will increase coverage by 5.91%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##           master      #54      +/-   ##
==========================================
+ Coverage   29.33%   35.24%   +5.91%
==========================================
  Files          17       21       +4
  Lines         958     1044      +86
==========================================
+ Hits          281      368      +87
+ Misses        625      617       -8
- Partials       52       59       +7

Impacted Files
Coverage Δ

cmd/help.go
57.62% <0.00%> (+47.45%)
:arrow_up:

cmd/run.go
0.00% <0.00%> (ø)

cmd/update.go
29.21% <0.00%> (-1.38%)
:arrow_down:

cmd/root.go
59.63% <85.71%> (+25.68%)
:arrow_up:

pkg/localdata/profile.go
17.97% <100.00%> (ø)

pkg/version/version.go
40.00% <0.00%> (ø)

pkg/tui/tui.go
0.00% <0.00%> (ø)

main.go
0.00% <0.00%> (ø)

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e798398...672dc3f. Read the comment docs.


goTemplateRendering env function not working

Before proceeding

[X] I didn't find a similar issue

Version
1.2.303
How to reproduce

set werf.yaml

project: test
configVersion: 1
---
image: test
dockerfile: Dockerfile
staged: true
args:
  CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX: '{{ env "CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX" "docker.io" }}'

Allow env function in werf-giterminism

giterminismConfigVersion: 1
config:
  dockerfile:
    allowContextAddFiles: [".npmrc"]
  goTemplateRendering:
    allowEnvVariables: ["CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX"]

Set .gitlab-ci.yaml

 script:
    - source $(werf ci-env gitlab --as-file)
    - export
    - CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX=$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX werf build --repo $WERF_REPO --add-custom-tag=%image%-$CI_COMMIT_REF_SLUG

Result
$ CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX=$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX werf build --repo $WERF_REPO --add-custom-tag=%image%-$CI_COMMIT_REF_SLUG
Version: v1.2.303
Using project dir: /builds/platform/test
Using werf config render file: /tmp/werf-config-render-2365218990
⛵ image test
00:00
┌ ⛵ image test
└ ⛵ image test (0.00 seconds) FAILED
Running time 0.36 seconds
Error: phase build before image test stages handler failed: unable to setup base image: unable to get base image "/node:21.1.0-slim" manifest: error getting buildah inspect of "/node:21.1.0-slim": error doing inspect: error getting builder from image "/node:21.1.0-slim": importing settings: invalid reference format

Expected result
Sucсessfull build
Additional information
No response

Thank you for reporting this issue!
In the current logic, the env function uses the default value only when the corresponding environment variable is absent. If the variable exists but is empty, the empty value is considered valid and is used as is.

Variable exists, has value and exported. Why does it substitute empty value?

I reproduced the situation, and here's what I got:
$ CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX="" werf config render
project: test
configVersion: 1
---
image: test
dockerfile: Dockerfile
staged: true
args:
  CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX: 'docker.io'

$ CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX="any" werf config render
project: test
configVersion: 1
---
image: test
dockerfile: Dockerfile
staged: true
args:
  CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX: 'any'

$ werf config render
project: test
configVersion: 1
---
image: test
dockerfile: Dockerfile
staged: true
args:
  CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX: 'docker.io'

$ werf version
v2.4.1

This is the expected behavior. The CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX environment variable is correctly used in the werf.yaml.
Do you expect different behavior?

I finally understand the issue you're facing. The problem lies not in how the env function works but in how you are using ARG in your Dockerfile. To use a build argument (build-arg) when specifying the image name in the FROM statement, you need to declare the ARG at the very beginning of your Dockerfile (before any FROM statements).
Here’s how you can modify your Dockerfile:
# Declare the ARG at the beginning
ARG BASE_IMAGE

# Then use the ARG in the FROM statement
FROM ${BASE_IMAGE}/node:21.1.0-slim

# The rest of your Dockerfile...

This is a characteristic of Dockerfiles and is not related to werf specifically.
If you have any further questions or run into additional issues, feel free to ask. We are here to help!

Seems like my bad. I have:
# Declare the ARG at the beginning
ARG ${BASE_IMAGE}

# Then use the ARG in the FROM statement
FROM ${BASE_IMAGE}/node:21.1.0-slim

Instead of the correct version that you offered:
# Declare the ARG at the beginning
ARG BASE_IMAGE

# Then use the ARG in the FROM statement
FROM ${BASE_IMAGE}/node:21.1.0-slim

Thank you for help!


Add serve command line options to list all supported model-ids

Could you add a test for this in the pytest?

Could you add a test for this in the pytest?

add new test in test_serve.py


How can I make a custom button?

I'd like to make a custom indent button which increases a tag's inline 'margin-left' value.
(ex.<p> → <p style="margin-left:20px;">) I can refer to existing files (Bold.js.. etc) and documentation, but things are mostly guess work. Two main things I'm curious about are

How to detect node based on tag attribute not tag
How to manipulate that tag attribute

Thank you for this amazing project :)

I am also keen on making a custom button/module.
I want to be able to highlight some text then press a button and it will save it into the VueX store to use that piece of text else where


[DONOTMERGE]Jenkins tester

jenkins: retest this please

jenkins: retest this please

jenkins: retest this please


Remove unused localstack file

/test jvm-build-hacbs-jdk11-builder-on-pull-request

appstudio bootstrap flake
/test jvm-build-service-in-repo-with-pruning-e2e

OK this PaC bug affecting our PR status is really getting annoying
I'm assuming the removal of the localstack file has no bearing on the builder images, but we've been bitten a couple of times now assuming that, merging PRs, and then bugs getting introduced.
And of course the pipelineruns have been pruned and I can no longer check to see they have passed.
Going to to try a
/retest
and see if that launches a new one.  Otherwise, may close/reopen PR.
If anything, just trying to establish a protocol on how to get a PR to look at if this PaC bug hits us.

retest did restart them

the cache/sidecar PasC jobs are still spining, but their PRs are marked complete on the staging cluster
merging


Macro: Provide save/open ket and molv3000 ketcher API for macromolecules

Background
It needs to have an ability to save/open ket and molv3000 files through ketcher API. It should be available in global "ketcher" variable similar as for micromolecules.
The following API methods should be changed/implemented:

[x] Implement getKet for Macromolecules
[x] Implement getMol for Macromolecules
[x] Implement setMolecule for Macromolecules
[x] Implement addFragment for Macromolecules
[x] layout should throw an Error 'layout is not available' in macro mode
[x] recognize should throw an Error 'recognize is not available' in macro mode
[x] generateImage should work with macromolecules; Most likely, output will be different from molecules drawn in Ketcher
[x] setSettings – when executed does nothing, as there are no settings for macro mode yet (may affect preview for monomers)
[x] containsReaction – no actual changes in functionality; returns true if structure has a reaction in micro mode
[x] getSmiles, getCdx, ... and other formats, except getKet and getMol – throw Error

Manual tests added to automation table


maxTimeMS during save is not supported

We used AdvancedDatastore to save the entity into MongoDB.
http://docs.mongodb.org/manual/reference/operator/meta/maxTimeMS/
How do you apply maxTimeMS parameter to this operation?
Can you please suggest.

MongoDB java-driver does not support on insert/update/delete. So closing the issue.


error!

Yes, the same problem happened to me!

Did not find a solution?
вт, 13 нояб. 2018 г. в 21:47, abrshdir<EMAIL_ADDRESS>
Yes, the same problem happened to me!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/NarimanMov/FaceApp-PHP/issues/10#issuecomment-438390456,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APiagl-RzvWCgzNtv8-2zhWrwHln7ngvks5uuxO8gaJpZM4YZ4CM
.

@burov0798 I will fix it in 4 days...
Please wait

I'm waiting...

I'm still trying. It has got a little complicated in this version.

:(

end?

end?

Похоже, что end :(

end?

Похоже, что end :(

вроде другие нашли способ обхода

Hey Are you fix it?

How to fixed it??


build_docs try to parse the .ipynb_checkpoints files.

I get FileNotFoundError when running nbdev_build_docs because it is trying to convert the files on the folder .ipynb_checkpoints
File "/home/tcapelle/Apps/nbdev/nbdev/export2html.py", line 699, in _get_title
    with open(fname, 'r') as f: code = f.read()
FileNotFoundError: [Errno 2] No such file or directory: '/home/tcapelle/SteadySun/nbdev_image_lib/docs/core-checkpoint.html'

I will try to propose a fix if this is an issue.

The problem appears to be this line:
def _create_default_sidebar():
    "Create the default sidebar for the docs website"
    dic = {"Overview": "/"}
    files = [f for f in Config().nbs_path.glob('**/*.ipynb') if not f.name.startswith('_')]

glob is very ambitious, an will iterate over the tree.

I made a PR to fix this.

Looks like this was fixed in https://github.com/fastai/nbdev/pull/273


Incompatible install in (summarize your environment)

Prerequisites

[ ] Read Python 2 Sunset docs.
[ ] Python 2 is required for this application.
[ ] I maintain the software that installs Setuptools (if not, please contact that project).
[x] Setuptools installed with pip 9 or later.
[ ] Pinning Setuptools to setuptools<45 in the environment was unsuccessful.

Environment Details

Operating System and version:
Python version:
Python installed how:
Virtualenv version (if using virtualenv): n/a

Command(s) that triggered the warning/error (and output):

Command(s) used to install setuptools (and output):

Output of pip --version when installing setuptools:

Other notes

Reports like this without the necessary context are unactionable.


Updates and Includes Example Resources in Install

Previously, the example resources failed to install. This PR updates the example to reflect the latest api changes. Additionally, the example resources are included in make install.
/assign @bowei
/cc @Miciah

Examples in this PR need to be updated if https://github.com/kubernetes-sigs/service-apis/pull/111 merges first.

/ok-to-test
/lgtm
/approve


[Enhancement] Remove unnecessary code

Description
As in the title, I removed unnecessary code, this variables aren't used anywhere in the code.

Thanks!


Notice of intention to change licence type from BSD 3 to MIT (Scalm)

Dear former contributors to Scalm!
(According to github, that's: @julienrf, @salc2, and @battermann.)
As mentioned on the Scalm gitter account, I've forked the excellent (but seemingly abandoned) Scalm as I hope to use it in conjunction with Indigo. I've brought it up to date, converted it to Scala 3, and I've started changing the apis and internals in to suit my own purposes. I do plan to release it (why not?).
The current licence agreement specifically mentions Julien Richard-Foy and EPFL, and I'd like to switch across to an MIT licence and change the named people to be consistent with Indigo's licence agreement and representative of who is looking after this version of the project (the original will remain as-is).
I would appreciate it if you could respond with your approval, or indeed your objections.
I realise most of you haven't looked at Scalm for 3 years or so (which is part of the reason for the fork!) so if I don't hear from you within 30 days, I'll go ahead and assume I have your consent.
Thank you for building the project in the first place!
All the best,
Dave

Hi David,
First of all I think it's great you folks have plan to integrate Scalm with
Indigo, I always have thought Elm like architecture applied to games have
lot of potential, really looking forward to seeing this.
About the fork and the licence, besides I know nothing about licences, I
think it's Julien and EPFL decision, I will support any.
I felt I needed to come with a comment because I was mentioned in the
thread but my contribution was really minimal actually, I think Julien's
work in Scalm was really great, neat and polished and I would love to see
more of this project, like ported to Scala 3 and with different views other
than vdom but Canvas or webgl.
Thanks,
Javier.
On Sat, 5 Jun 2021, 23:05 Dave Smith, @.***> wrote:

Dear former contributors to Scalm https://github.com/julienrf/scalm!
(According to github, that's: @julienrf https://github.com/julienrf,
@salc2 https://github.com/salc2, and @battermann
https://github.com/battermann.)
As mentioned on the Scalm gitter account, I've forked the excellent (but
seemingly abandoned) Scalm as I hope to use it in conjunction with Indigo
https://github.com/PurpleKingdomGames/indigo. I've brought it up to
date, converted it to Scala 3, and I've started changing the apis and
internals in to suit my own purposes. I do plan to release it (why not?).
The current licence agreement specifically mentions Julien Richard-Foy and
EPFL, and I'd like to switch across to an MIT licence and change the named
people to be consistent with Indigo's licence agreement
https://github.com/PurpleKingdomGames/indigo/blob/master/LICENCE.md and
representative of who is looking after this version of the project (the
original will remain as-is).
I would appreciate it if you could respond with your approval, or indeed
your objections.
I realise most of you haven't looked at Scalm for 3 years or so (which is
part of the reason for the fork!) so if I don't hear from you within 30
days, I'll go ahead and assume I have your consent.
Thank you for building the project in the first place!
All the best,
Dave
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PurpleKingdomGames/tyrian/issues/1, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABGSOIBGLY3XEKLMZV2JCWDTRKNTLANCNFSM46FCHSTQ
.

Hi @salc2,
Thanks! When you're moving licence you can only really move with the blessing of all former contributors - however many commits they added. :-)
I agree, lots of potential for games but also this particular architecture is in my opinion, the best one out there for web dev in general, and it really doesn't exist in Scala at the moment.
Indigo use the Elm Architecture specifically for WebGL (2.0) and game development, so my interest in Scalm/Tyrian is mostly Web dev (and games that use web tech) and Canvas. I will probably add WebGL support in places too, but it's not the primary focus as I have Indigo for that.
I've already upgraded to fully Scala 3 now and it makes the syntax even nicer to work with. All the readme examples, and example apps are Scala 3 now if you're interested.
The main use cases I have are for housing Indigo games and building / rebuilding supporting tooling such as simple image editors and game asset conversion tools.
Thank you for your support! :-)
Dave

Hello!
I’m super happy to see that you are interested in this project!
Feel free to change the license to whatever suits you!

Hi @julienrf,
Excellent, many thanks! I shall give the final contributor a chance to respond but I really appreciate your endorsement. :-)
All the best,
Dave

Thanks all, switched to MIT licence as discussed.


feat(function): add least function

Which issue does this PR close?
Closes #6531
Rationale for this change
adding more expressions support, and I already added greatest
What changes are included in this PR?
copied the greatest function I did in #12474 and modified.
I choose coping over macro for the following reasons:

Easier debugging and maintainability
Least has custom logic regarding scalars

Are these changes tested?
yes
Are there any user-facing changes?
yes

LGTM. thanks!

Thank you @rluvaton
Taking into account that the least/greatest implementations are almost identical (the way how these functions handle input arguments / constant folding optimization etc) and the major difference is sort options for the comparator and the method called on the comparator, perhaps it is possible to make these functions share the same parameterized implementation? I suppose, in this case it would be easier to fix/update functions behavior if required (since it'll require only one implementation modification instead of two).

Thank you, I did not have time to update based on the review, I'll do it in the coming days

So I merged both implementations into one, I don't really like it, the naming I choose is off, I need a better name but all comments and resolved

Thanks again @rluvaton @Omega359 and @2010YOUY01 :rockeet


[Feature]: Support monorepo

What problem does this feature solve?
Currently, rspress could not support perfectly, when we write docs in packages, and defined sidebar using _meta.json, it's not working.
The monorepo dir structure is usually as below:
- docs      # top level docs
   - _meta.json
- packages
  - pkg1
     - docs
     - _meta.json
  - pkg2
     - docs
     - _meta.json
  - pkg3
     - docs
     - _meta.json

What does the proposed API look like?
Just add monorepo supporting, and better provide plugin API at the same time.

In default mode all the docs should be put in the same direcotry(That is docs in your case). And if you want to add some docs in other places, we recommend to use plugin and addPages hook to register the route of docs. Details: https://rspress.dev/plugin/system/plugin-api.html#addpages
And in this way, you should configure the sidebar and nav bar in themeConfig.nav/sidebar of Rspress instead of using _meta.json. https://rspress.dev/api/config/config-theme.html#nav

Should we support using glob patterns to match documents? It can be easier than the addPages API.

It is currently support by route.include config.https://rspress.dev/api/config/config-basic.html#routeinclude cc @jas0ncn

Should we support using glob patterns to match documents? It can be easier than the addPages API.

It is currently support by route.include config.https://rspress.dev/api/config/config-basic.html#routeinclude cc @jas0ncn

Should we support using glob patterns to match documents? It can be easier than the addPages API.

Thanks, but even if using the route.include to match ducuments, we still need to configure sidebar manually, _meta.json is easier to use obviously😭

It is currently support by route.include config.https://rspress.dev/api/config/config-basic.html#routeinclude cc @jas0ncn

Should we support using glob patterns to match documents? It can be easier than the addPages API.

Thanks, but even if using the route.include to match ducuments, we still need to configure sidebar manually, _meta.json is easier to use obviously😭

我通过这样的配置

没有生成对应的可访问路径
检查方式是，写了一个测试插件，通过日志打印

发现没有路由信息
而我的项目种确实有对应的md文件，这是什么问题呢？


Gravitite ore can be mined by hand, does not require iron level pickaxe

how
https://user-images.githubusercontent.com/53229958/125366498-c43f7200-e33b-11eb-968b-9a3c02ad6221.mp4

was missing .requirestool() oops


Test tests.integration.cmor._fixes.CMIP5.test_CESM1_BGC.TestNbp.test_fix_data failing

The test is failing after unpinning the libgdal library, could you please have a look? I suspect the code does not work with newer versions of the netcdf library.
See https://circleci.com/gh/ESMValGroup/ESMValTool/2774

By the way, maybe that test should be renamed from test_fix_data to test_fix_file, as it appears to be testing the fix_file method.

Soory for the delay. I will take a look now

Ok. They fixed a bug I was exploiting to overwrite the fill value of a variable: they were not allowing to change it individually but not when setting all attributes using a dictionary.
I will need to complicate a bit the fix, but is not really difficult


[FEATURE REQUEST] - Option to auto minimize incoming messages

Describe the solution you'd like
Ability to toggle an option to automatically minimize (shrink) incoming messages.
Helps avoid screen clutter when being spammed, or helps visibility when mapping.

Added an option to hide messages until next one will come.
Later on I will add option in settings for it to not be shown if hidden manually even if new message will come


Size 9 and 10 are not working

I was testing the font size and I saw that size 9 and 10 were not working properly on my instance of etherpad.
Then I went to test on Etherpad's sandstorm demo and was not working there either.
Both 9 and 10 have the same size as 15

You need to change the first two lines in iframe.css file like this:
fs8{font-size:8px;}
fs9{font-size:9px;}

The the semicolons were wrongly placed.

Pull request welcome!


simple theme matching RedHat bash looks

most zsh themes are well departed from the basic bash themes.  This adds a minimal theme that tries to mimic the bash theme used on most RHEL based distros

Hi @hellozt, we have lots of themes already. You can add yours as an external theme in the wiki.


Some issues with PVN3D

Hi@yzqin,
Thanks for your great work!
I have encountered some problems in the process of using PVN3D. I would like to know if you used the original parameters of PVN3D, or did further training.
Thanks in advance!

Hi @weixzhang ,
We do not use the segmentation module provided by PVN3D but directly feed some external segmentation. To do this, we per-process with some segmentation network to get the segmentation mask. Since the segmentation is well-solved nowadays, you can use some recent SOTA segmentation network to output the mask. In this way, you can get better performance with the per-trained pose estimation network with PVN3D.

Thank you very much for your reply, it has been very helpful to me！

I have another question. Did you do the segmentation on the point cloud or on the image? Looking forward to your reply.

I have another question. Did you do the segmentation on the point cloud or on the image?What is the final input form into PVN3D?
Can you please explain it to me? Looking forward to your reply.

Segmentation on the image. If you have aligned RGB and depth, you can easily lift the 2D segmentation mask to point cloud. The final input can be the segmented point cloud.

Hello, we also use advanced image segmentation methods (which have been verified to be accurate) on the test image (our own scene), but the final object pose effect is improved very little. I would like to ask you how to further process it to get It's the exact pose, thank you very much!

Hi @xsaxasni
You can also apply some post processing for the estimated pose, like ICP (I believe that there will be better alternatives for ICP) and low-pass-filter. Normally the original pose estimated from video will not be so good, like in this project. But we found that even with pose errors in the human demonstration, it still helps for the downstream imitation learning side. So do you worry too much if you do not get a very good pose estimation results.

Hi @xsaxasni
You can also apply some post processing for the estimated pose, like ICP (I believe that there will be better alternatives for ICP) and low-pass-filter. Normally the original pose estimated from video will not be so good, like in this project. But we found that even with pose errors in the human demonstration, it still helps for the downstream imitation learning side. So do you worry too much if you do not get a very good pose estimation results.

Thank you very much for your reply. The scene we use is not imitation learning, so the pose estimation results required should be better. We will try the method you mentioned, and thank you for your timely reply again!


Update arrow.py

Fix relativedelta subtraction from arrow

Current coverage is 100% (diff: 100%)

Merging #375 into master will not change coverage

@@           master   #375   diff @@
====================================
  Files           8      8
  Lines        1246   1246
  Methods         0      0
  Messages        0      0
  Branches      194    194
====================================
  Hits         1246   1246
  Misses          0      0
  Partials        0      0

Powered by Codecov. Last update 4d31d9a...0e2bb03


Investigate failure in osm_to_pq.py for asia-latest OSM

Some of the road network linestring data in the OSM subset asia-latest seems to be malformed, and when shapely tries to handle it in osm_to_pq.py it dies.
To reproduce:
snakemake --cores 10 --rerun-incomplete results/asia-latest_filter-highway-tertiary/road_network_map_by_component.png
Traceback (most recent call last):
  File "shapely/speedups/_speedups.pyx", line 90, in shapely.speedups._speedups.geos_linestring_from_py
AttributeError: 'list' object has no attribute '__array_interface__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fred/open-gira/.snakemake/scripts/tmp7pi4kmsb.osm_to_pq.py", line 197, in <module>
    h.apply_file(pbf_path, locations=True)
  File "/home/fred/open-gira/.snakemake/scripts/tmp7pi4kmsb.osm_to_pq.py", line 92, in way
    linestring = shape.linestring.LineString(locations)
  File "/home/fred/anaconda3/envs/open-gira/lib/python3.9/site-packages/shapely/geometry/linestring.py", line 46, in __init__
    ret = geos_linestring_from_py(coordinates)
  File "shapely/speedups/_speedups.pyx", line 156, in shapely.speedups._speedups.geos_linestring_from_py
ValueError: LineStrings must have at least 2 coordinate tuples
[Wed Jul 13 15:47:56 2022]
Error in rule convert_to_geoparquet:
    jobid: 28
    output: results/geoparquet/asia-latest_filter-highway-tertiary/slice-6.geoparquet

RuleException:
CalledProcessError in line 20 of /home/fred/open-gira/workflow/rules/convert_to_geoparquet.smk:
Command 'set -euo pipefail;  /home/fred/anaconda3/envs/open-gira/bin/python /home/fred/open-gira/.snakemake/scripts/tmp7pi4kmsb.osm_to_pq.py' returned non-zero exit status 1.
  File "/home/fred/open-gira/workflow/rules/convert_to_geoparquet.smk", line 20, in __rule_convert_to_geoparquet
  File "/home/fred/anaconda3/envs/open-gira/lib/python3.9/concurrent/futures/thread.py", line 58, in run

Closed by #85


ambient: "failed to add pod to ztunnel" on k3s cluster

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description
When attempting to run Istio ambient mode on k3s cluster (arm64) using Calico, istio-cni reports the following error when attempting to add pods to the ambient mesh (adding ambient label to namespace):
2024-03-24T23:28:47.457136Z     error   Command error output: xtables other problem: line 2 failed
2024-03-24T23:28:47.457224Z     error   iptables        failed to restore iptables rules: exit status 1
2024-03-24T23:28:47.457330Z     error   ambient failed to update POD inpod: default/details-v1-698d88b-n42tz exit status 1
2024-03-24T23:28:47.457353Z     error   ambient failed to add pod to ztunnel: exit status 1

And the following error when attempting to remove a pod from the mesh (removing ambient label from namespace):
2024-03-24T23:29:01.150318Z     warn    ambient failed to find pod netns        ns=default name=details-v1-698d88b-n42tz
2024-03-24T23:29:01.150381Z     error   ambient failed to remove pod to ztunnel: failed to find pod netns       ns=default name=details-v1-698d88b-n42tz
2024-03-24T23:29:01.156068Z     warn    ambient failed to find pod netns        ns=default name=reviews-v3-5b9bd44f4-tmml8
2024-03-24T23:29:01.156133Z     error   ambient failed to remove pod to ztunnel: failed to find pod netns       ns=default name=reviews-v3-5b9bd44f4-tmml8

Full logs of each event attached, bug-report output attached below.
add-pods.log
remove-pods.log
I see mention of a similar issue in this PR: https://github.com/istio/istio/pull/49231
Istio installed in this example with istioctl install, same issue observed with helm install
Version
$ istioctl version
client version: 1.21.0
control plane version: 1.21.0
data plane version: 1.21.0 (6 proxies)
$ kubectl version
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.1+k3s2

Additional Information
bug-report.tar.gz

`istioctl bug-report` output
$ istioctl bug-report

Target cluster context: kubernetes-admin-pico@pico

Running with the following config:

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: {  }
exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage }
end-time: 2024-03-24 19:30:22.269187089 -0400 EDT

Cluster endpoint: https://pi01.pico.home.arpa:6443
CLI version:
version.BuildInfo{Version:"1.21.0", GitRevision:"Homebrew", GolangVersion:"go1.22.1", BuildStatus:"Homebrew", GitTag:"1.21.0"}

The following Istio control plane revisions/versions were found in the cluster:
Revision default:
&version.MeshInfo{
    {
        Component: "pilot",
        Revision:  "default",
        Info:      version.BuildInfo{Version:"1.21.0", GitRevision:"2ca3e986a683fbdb82dffcd7b2e8d02076a42468", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.21.0"},
    },
}

The following proxy revisions/versions were found in the cluster:
Revision default: Versions {1.21.0}

Fetching logs for the following containers:

calico-apiserver/calico-apiserver/calico-apiserver-7d78f674d6-9gxvb/calico-apiserver
calico-apiserver/calico-apiserver/calico-apiserver-7d78f674d6-nmr2b/calico-apiserver
calico-system/calico-kube-controllers/calico-kube-controllers-7978f6997b-bqs6w/calico-kube-controllers
calico-system/calico-node/calico-node-b5mq9/calico-node
calico-system/calico-node/calico-node-nkkpv/calico-node
calico-system/calico-node/calico-node-qrmxf/calico-node
calico-system/calico-node/calico-node-shzl8/calico-node
calico-system/calico-node/calico-node-v8kbf/calico-node
calico-system/calico-typha/calico-typha-86d697f799-8664w/calico-typha
calico-system/calico-typha/calico-typha-86d697f799-msdg4/calico-typha
calico-system/calico-typha/calico-typha-86d697f799-t7x4x/calico-typha
calico-system/csi-node-driver/csi-node-driver-gjdrm/calico-csi
calico-system/csi-node-driver/csi-node-driver-gjdrm/csi-node-driver-registrar
calico-system/csi-node-driver/csi-node-driver-mdcc9/calico-csi
calico-system/csi-node-driver/csi-node-driver-mdcc9/csi-node-driver-registrar
calico-system/csi-node-driver/csi-node-driver-tx2nk/calico-csi
calico-system/csi-node-driver/csi-node-driver-tx2nk/csi-node-driver-registrar
calico-system/csi-node-driver/csi-node-driver-w576d/calico-csi
calico-system/csi-node-driver/csi-node-driver-w576d/csi-node-driver-registrar
calico-system/csi-node-driver/csi-node-driver-wxvc2/calico-csi
calico-system/csi-node-driver/csi-node-driver-wxvc2/csi-node-driver-registrar
default/details-v1/details-v1-698d88b-n42tz/details
default/notsleep/notsleep-5c785bc478-wcnlk/notsleep
default/productpage-v1/productpage-v1-675fc69cf-p7t5z/productpage
default/ratings-v1/ratings-v1-6484c4d9bb-f6nhz/ratings
default/reviews-v1/reviews-v1-5b5d6494f4-6dfqg/reviews
default/reviews-v2/reviews-v2-5b667bcbf8-6m2xh/reviews
default/reviews-v3/reviews-v3-5b9bd44f4-tmml8/reviews
default/sleep/sleep-7656cf8794-cj8sc/sleep
istio-system/bookinfo-gateway-istio/bookinfo-gateway-istio-7755d4897c-lbk5l/istio-proxy
istio-system/istio-cni-node/istio-cni-node-9smjx/install-cni
istio-system/istio-cni-node/istio-cni-node-q99bw/install-cni
istio-system/istio-cni-node/istio-cni-node-wtpdn/install-cni
istio-system/istio-cni-node/istio-cni-node-wwzx8/install-cni
istio-system/istio-cni-node/istio-cni-node-xwgs7/install-cni
istio-system/istiod/istiod-7879855784-9tj5h/discovery
istio-system/ztunnel/ztunnel-4gsgd/istio-proxy
istio-system/ztunnel/ztunnel-5m78r/istio-proxy
istio-system/ztunnel/ztunnel-vwndw/istio-proxy
istio-system/ztunnel/ztunnel-x6xj5/istio-proxy
istio-system/ztunnel/ztunnel-xh6qz/istio-proxy
metallb-system/metallb-controller/metallb-controller-5bfcc7df68-27c6j/controller
metallb-system/metallb-speaker/metallb-speaker-2chn6/speaker
metallb-system/metallb-speaker/metallb-speaker-46zpg/speaker
metallb-system/metallb-speaker/metallb-speaker-fqw8l/speaker
metallb-system/metallb-speaker/metallb-speaker-rh9fm/speaker
metallb-system/metallb-speaker/metallb-speaker-xhlh6/speaker
tigera-operator/tigera-operator/tigera-operator-6d7bbc7b65-gpk6t/tigera-operator

Fetching Istio control plane information from cluster.

Fetching CNI logs from cluster.

Running Istio analyze on all namespaces and report as below:
Analysis Report:
Info [IST0102] (Namespace calico-apiserver) The namespace is not enabled for Istio injection. Run 'kubectl label namespace calico-apiserver istio-injection=enabled' to enable it, or 'kubectl label namespace calico-apiserver istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace calico-system) The namespace is not enabled for Istio injection. Run 'kubectl label namespace calico-system istio-injection=enabled' to enable it, or 'kubectl label namespace calico-system istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run 'kubectl label namespace default istio-injection=enabled' to enable it, or 'kubectl label namespace default istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace metallb-system) The namespace is not enabled for Istio injection. Run 'kubectl label namespace metallb-system istio-injection=enabled' to enable it, or 'kubectl label namespace metallb-system istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace monitoring) The namespace is not enabled for Istio injection. Run 'kubectl label namespace monitoring istio-injection=enabled' to enable it, or 'kubectl label namespace monitoring istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace tigera-operator) The namespace is not enabled for Istio injection. Run 'kubectl label namespace tigera-operator istio-injection=enabled' to enable it, or 'kubectl label namespace tigera-operator istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0118] (Service calico-apiserver/calico-api) Port name apiserver (port: 443, targetPort: 5443) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service calico-system/calico-kube-controllers-metrics) Port name metrics-port (port: 9094, targetPort: 9094) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service calico-system/calico-typha) Port name calico-typha (port: 5473, targetPort: calico-typha) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service metallb-system/metallb-webhook-service) Port name  (port: 443, targetPort: 9443) doesn't follow the naming convention of Istio port.
Creating an archive at /home/gclawes/repos/github.com/istio/istio/bug-report.tar.gz.
Time used for creating the tar file is 429.97106ms.
Cleaning up temporary files in /tmp/bug-report.
Done.

Affected product area

[X] Ambient
[ ] Docs
[ ] Dual Stack
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Control Plane Revisions

cc @bleggett

Someone else mentioned to me they had issues with k3d specifically as well - I'll take a look at that.
We have integ tests that validate calico with ambient, so I don't think calico is the problem, specifically.

FYI, this is regular k3s on a set of Raspberry Pi 4s, not k3d (in docker).  But it could be a k3s issue.  I can try to test w/ kubespray instead of k3s on this cluster, may have some time to test that in a week or so.

Fyi: When I install k3s with this command:
curl -fL https://get.k3s.io | INSTALL_K3S_VERSION=v1.29.3+k3s1 \
    sh -s - server \
    --disable traefik

I need to install istio-cni with these values included to startup successfully:
helm upgrade --install istio-base istio/base -n istio-system --create-namespace
helm upgrade --install istio-cni istio/cni -n istio-system \
  --set profile=ambient \
  --set cni.cniBinDir=/var/lib/rancher/k3s/data/current/bin/ \
  --set cni.cniConfDir=/var/lib/rancher/k3s/agent/etc/cni/net.d

Ah perfect. Thank you for digging. That tracks. Microk8s and minikube also use nonstandard cniConf/BinDirs.
We already have some platform notes for those two here: https://istio.io/latest/docs/ops/ambient/install/platform-prerequisites/
but not one for k3s - if you would like to PR into istio/istio.io to add a small k3s section with what you've found, please do, otherwise I'll do it later.
(Given that so many of the "mini" variants use nonstandard paths, I wonder if we should make the code smarter and do our own path search, rather than having minor flag overrides for every local k8s variant? something to consider I guess)

I run Istio ambient mode on k3s cluster (amd64) using Calico, It works fine. The commands are：
curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn \
  INSTALL_K3S_EXEC='--flannel-backend=none --cluster-cidr=<IP_ADDRESS>/16' \
  K3S_KUBECONFIG_OUTPUT=/root/.kube/config sh -s - \
  --disable-network-policy \
  --disable "traefik"
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml
istioctl install --set profile=ambient --set "components.ingressGateways[0].enabled=true" \
--set "components.ingressGateways[0].name=istio-ingressgateway" \
--skip-confirmation

I run Istio ambient mode on k3s cluster (amd64) using Calico, It works fine. The commands are：
curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn \
  INSTALL_K3S_EXEC='--flannel-backend=none --cluster-cidr=<IP_ADDRESS>/16' \
  K3S_KUBECONFIG_OUTPUT=/root/.kube/config sh -s - \
  --disable-network-policy \
  --disable "traefik"
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml
istioctl install --set profile=ambient --set "components.ingressGateways[0].enabled=true" \
--set "components.ingressGateways[0].name=istio-ingressgateway" \
--skip-confirmation

I believe this works because you are not installing K3S with any CNI, and then installing Calico on top with defaults, which would use its own, more standard default CNI directory.
If you let K3S manage the CNI, you need the overrides, as it uses nonstandard paths. If you do not, it happens to work, because the K3S nonstandard path is being ignored by the primary CNI you have installed.
I'll enhance the docs and we can close this out.

My original example was a k3s cluster with an external CNI (calico), so I don't think the k3s-specific CNI dir was the issue.  I do not disable network-policy at the moment (oversight on my part), I'll try that and report back.

My original example was a k3s cluster with an external CNI (calico), so I don't think the k3s-specific CNI dir was the issue. I do not disable network-policy at the moment (oversight on my part), I'll try that and report back.

Ok, let us know. We run integ tests with calico, so if there's an incompatibility, it's k3s-specific, and it's probably the CNI netdir.

I was mistaken, my k3s config.yaml does in fact have disable-network-policy: true, so that's not the issue:
ubuntu@pi01:~$ sudo cat /etc/rancher/k3s/config.yaml
---

cluster-cidr: <IP_ADDRESS>/16
disable:
- servicelb
- traefik
- local-storage
disable-cloud-controller: true
disable-helm-controller: true
disable-network-policy: true
flannel-backend: none
node-external-ip: <IP_ADDRESS>
node-name: pi01.pico.home.arpa
node-taint:
- node-role.kubernetes.io/control-plane=:NoSchedule
service-cidr: <IP_ADDRESS>/16

I am using calico in k3s as an external CNI, and I confirmed that calico is using the standard CNI dirs (so I shouldn't need to set the k3s-specific ones):
ubuntu@pi01:~$ sudo ls -lah /etc/cni/net.d/
total 20K
drwx------ 2 root root 4.0K Apr 17 03:22 .
drwxr-xr-x 3 root root 4.0K Apr 17 02:17 ..
-rw-r--r-- 1 root root 1.4K Apr 17 02:33 10-calico.conflist
-rw------- 1 root root 2.0K Apr 17 02:23 calico-kubeconfig
-rw------- 1 root root 2.1K Apr 17 03:22 ZZZ-istio-cni-kubeconfig
ubuntu@pi01:~$ sudo ls -lah /opt/cni/bin/
total 183M
drwxr-xr-x 2 root root 4.0K Apr 17 03:22 .
drwxr-xr-x 3 root root 4.0K Apr 17 02:21 ..
-rwxr-xr-x 1 root root 3.9M Apr 17 02:21 bandwidth
-rwxr-xr-x 1 root root  58M Apr 17 02:21 calico
-rwxr-xr-x 1 root root  58M Apr 17 02:21 calico-ipam
-rwxrwxr-x 1 root root 2.5M Apr 17 02:21 flannel
-rwxr-xr-x 1 root root 3.4M Apr 17 02:21 host-local
-rwxr-xr-x 1 root root  48M Apr 17 03:22 istio-cni
-rwxr-xr-x 1 root root 3.5M Apr 17 02:21 loopback
-rwxr-xr-x 1 root root 3.9M Apr 17 02:21 portmap
-rwxr-xr-x 1 root root 3.6M Apr 17 02:21 tuning

K3s does populate the directory /var/lib/rancher/k3s/data/current/bin, but as far as I can tell calico is using the standard ones:
ubuntu@pi01:~$ ls -lah /var/lib/rancher/k3s/data/current/bin/ | head -8
total 189M
drwxr-xr-x 3 root root  12K Apr 17 03:41 .
drwxr-xr-x 4 root root 4.0K Apr 17 03:41 ..
lrwxrwxrwx 1 root root    9 Apr 17 03:41 [ -> coreutils
lrwxrwxrwx 1 root root    7 Apr 17 03:41 [[ -> busybox
lrwxrwxrwx 1 root root    7 Apr 17 03:41 addgroup -> busybox
lrwxrwxrwx 1 root root    7 Apr 17 03:41 adduser -> busybox
lrwxrwxrwx 1 root root    7 Apr 17 03:41 ar -> busybox

/var/lib/rancher/k3s/agent/etc/cni/net.d is not created:
ubuntu@pi01:~$ sudo ls -lah /var/lib/rancher/k3s/agent/etc/cni/net.d
ls: cannot access '/var/lib/rancher/k3s/agent/etc/cni/net.d': No such file or directory

I'm still getting this issue:
2024-04-17T04:07:37.083680Z     info    Running command (without lock): iptables-restore --noflush -v
2024-04-17T04:07:37.091592Z     error   Command error output: xtables other problem: line 2 failed
2024-04-17T04:07:37.091664Z     error   iptables        failed to restore iptables rules: exit status 1
2024-04-17T04:07:37.091759Z     error   ambient failed to update POD inpod: default/notsleep-5c785bc478-62bxl exit status 1
2024-04-17T04:07:37.091784Z     error   ambient failed to add pod to ztunnel: exit status 1

I've tried this with container_settings.allow_ip_forwarding set to both true and false as recommended in the k8s docs linked in #14880  (my initial cluster setup when I opened the issue had it as false).
This cluster is running on ubuntu 23.10:
ubuntu@pi01:~$ head -3 /etc/os-release
PRETTY_NAME="Ubuntu 23.10"
NAME="Ubuntu"
VERSION_ID="23.10

From the logs, istio-cni can't run the iptables binary it ships with there, which isn't related to calico.
If it can't run the iptables binary, that means

The iptables binary isn't inside the istio-cni container (might happen if you are using an older 1.21 build: https://github.com/istio/istio/pull/50223)

Your istio-cni container is not running as a privileged container, or your container runtime is not configured to correctly support privileged containers.

You can check this by kubectl exec ing into the istio-cni image and trying to run iptables yourself.

$ kubectl -n istio-system exec -it istio-cni-node-xjqj7 -- bash
root@pi02:/opt/cni/bin# which -a iptables
/usr/sbin/iptables
/sbin/iptables
root@pi02:/opt/cni/bin# ls -lah /usr/sbin/iptables /sbin/iptables
lrwxrwxrwx 1 root root 26 Mar 28 19:05 /sbin/iptables -> /etc/alternatives/iptables
lrwxrwxrwx 1 root root 26 Mar 28 19:05 /usr/sbin/iptables -> /etc/alternatives/iptables
root@pi02:/opt/cni/bin# iptables --version
iptables v1.8.7 (legacy)

Okay, I'll try to repro on my end.
One more thing to check - do you have access to the host node in your cluster?
And if you do, do either of the paths

/var/run/netns
/var/run/docker/netns

exist in the host node file system?

Yeah, I fully control the cluster (k3s on raspberry pis I own).
ubuntu@pi01:~$ sudo ls -lah /var/run/netns
total 0
drwxr-xr-x  2 root root  140 Apr 17 04:00 .
drwxr-xr-x 39 root root 1.1K Apr 17 12:19 ..
-r--r--r--  1 root root    0 Apr 17 04:00 cni-296f2c8d-82f4-d49f-180a-4afa9d25d950
-r--r--r--  1 root root    0 Apr 17 03:54 cni-3077d3cf-7e2c-7ba6-8506-e58eec0dc349
-r--r--r--  1 root root    0 Apr 17 03:54 cni-94f49917-6b90-b2f4-447b-b808bddae26f
-r--r--r--  1 root root    0 Apr 17 03:56 cni-9faec339-720b-cb6a-4107-37d1477527b9
-r--r--r--  1 root root    0 Apr 17 04:00 cni-a8ba5c59-ac6c-13f7-b311-eb9c139143f3
ubuntu@pi01:~$ sudo ls -lah /var/run/docker/netns
ls: cannot access '/var/run/docker/netns': No such file or directory

For repro purposes, here's more info on my setup, in case any of it helps:
K3s configured with this ansible role: https://github.com/PyratLabs/ansible-role-k3s
k3s server config (single control plane server pi01):
cluster-cidr: <IP_ADDRESS>/16
disable:
- servicelb
- traefik
- local-storage
disable-cloud-controller: true
disable-helm-controller: true
disable-network-policy: true
flannel-backend: none
node-external-ip: <IP_ADDRESS>
node-name: pi01.pico.home.arpa
node-taint:
- node-role.kubernetes.io/control-plane=:NoSchedule
service-cidr: <IP_ADDRESS>/16

k3s agent config (4x agent servers pi02-pi04):
---

node-external-ip: <IP_ADDRESS>
node-name: pi02.pico.home.arpa

tigera-operator helm chart values:
NOTES:

encapsulation disabled (no VXLAN, all routing)
SNAT disabled from hosts (pod CIDRs advertised to upstream routers, see below)
containerIPForwarding: Enabled per k3s recommendations in docs

installation:
  kubeletVolumePluginPath: None
  controlPlaneNodeSelector:
    node-role.kubernetes.io/control-plane: "true"
  controlPlaneReplicas: 1
  operator: Exists
  calicoNetwork:
    bgp: Enabled
    containerIPForwarding: Enabled
    ipPools:
      - cidr: <IP_ADDRESS>/16
        encapsulation: None
        natOutgoing: Disabled

apiServer:
  enabled: true

tolerations:
- key: node.kubernetes.io/not-ready
  effect: NoSchedule
  operator: Exists
- key: node-role.kubernetes.io/control-plane
  effect: NoSchedule
  operator: Exists

# NodeSelector for the tigera/operator pod.
nodeSelector:
  kubernetes.io/os: linux
  node-role.kubernetes.io/control-plane: "true"

Calico BGP config resources (installed by bedag/raw chart)
NOTES:

cluster AS 65002
nodeToNodeMeshEnabled is disabled in the BGPConfiguration, instead a node-to-node mesh is manually created with MetalLB routes excluded
cluster connects to 1 BGP router (turris) at AS 65000, does not export node-to-node routes
MetalLB peers w/ cluster on each node over <IP_ADDRESS> as AS 65003

resources:
- apiVersion: projectcalico.org/v3
  kind: BGPConfiguration
  metadata:
    name: default
  spec:
    logSeverityScreen: Info
    nodeToNodeMeshEnabled: false
    asNumber: 65002
- apiVersion: projectcalico.org/v3
  kind: IPPool
  metadata:
    name: metallb-ippool
  spec:
    cidr: <IP_ADDRESS>/16
    disabled: true
- apiVersion: projectcalico.org/v3
  kind: BGPFilter
  metadata:
    name: export-reject-all-learned-routes
  spec:
    exportV4:
      - action: Reject
        source: RemotePeers
- apiVersion: projectcalico.org/v3
  kind: BGPFilter
  metadata:
    name: export-reject-learned-node-routes
  spec:
    exportV4:
      - action: Reject
        matchOperator: In
        cidr: <IP_ADDRESS>/16
        source: RemotePeers
- apiVersion: projectcalico.org/v3
  kind: BGPFilter
  metadata:
    name: import-reject-lb-routes
  spec:
    exportV4:
      - action: Reject
        matchOperator: In
        cidr: <IP_ADDRESS>/16
        source: RemotePeers
- apiVersion: projectcalico.org/v3
  kind: BGPPeer
  metadata:
    name: node-to-node
  spec:
    nodeSelector: all()
    peerSelector: all()
    filters:
      - import-reject-lb-routes
- apiVersion: projectcalico.org/v3
  kind: BGPPeer
  metadata:
    name: turris-bgppeer-65000
  spec:
    nodeSelector: all()
    peerIP: <IP_ADDRESS>
    asNumber: 65000
    filters:
    - export-reject-learned-node-routes
- apiVersion: projectcalico.org/v3
  kind: BGPPeer
  metadata:
    name: metallb-bgppeer-65003
  spec:
    nodeSelector: all()
    peerIP: <IP_ADDRESS>
    asNumber: 65003
    filters:
    - export-reject-all-learned-routes

@bleggett any luck reproducing this?

@gclawes
No, but we have successfully tested ambient with flannel + k3d and calico + k3d.
If it works on k3d it should work for k3s.  Check the recently-updated K3D platform prerequisites.
There are 2 things that need to work:

The CNI bindir/confdir your Calico install is using must be the same one Istio is using, if it is non-default, you can override the paths for Istio. The linked doc has the locations for flannel + k3d, which uses a nonstandard path.

Calico + k3d uses the default /etc/cni/net.d, so yeah - you should not need to override this.

Your kernel needs proper iptables support with the required iptables extensions loaded. I would check the kernel prereqs page and make sure whatever kernel you have has all the correct iptables extensions loaded.

You can install Istio with --set global.logging.level="all:debug" which will put istio-cni into debug log mode - it will then echo the iptables rules it is trying (and failing) to apply, and you should be able to try applying them yourself via kubectl -n istio-system exec -it istio-cni-node-xjqj7 -- bash to understand why they are failing on your nodes.
The other things mentioned in this issue are probably not a factor - just these 2 things.
If you can repro the issue outside of your baremetal setup (with k3d or similar) that would help too.

Is there still an issue here with 1.23/1.22.4?

The original issue appears to be fixed, I can add pods to the mesh by going through the Getting Started guide up to this point: https://istio.io/latest/docs/ambient/getting-started/secure-and-visualize/#add-bookinfo-to-the-mesh
I don't see any issues adding the pods in the logs:
2024-08-21T19:44:34.463634Z     info    cni-agent       adding pod to the mesh  ns=default name=productpage-v1-d5789fdfb-tk78f
2024-08-21T19:44:34.511631Z     info    cni-agent       Running iptables-nft-restore with the following input:
* mangle
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_PRERT -m mark --mark 0x539/0xfff -j CONNMARK --set-xmark 0x111/0xfff
-A ISTIO_PRERT -s <IP_ADDRESS> -p tcp -m tcp -j ACCEPT
-A ISTIO_PRERT ! -d <IP_ADDRESS>/32 -p tcp -i lo -j ACCEPT
-A ISTIO_PRERT -p tcp -m tcp --dport 15008 -m mark ! --mark 0x539/0xfff -j TPROXY --on-port 15008 --tproxy-mark 0x111/0xfff
-A ISTIO_PRERT -p tcp -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ISTIO_PRERT ! -d <IP_ADDRESS>/32 -p tcp -m mark ! --mark 0x539/0xfff -j TPROXY --on-port 15006 --tproxy-mark 0x111/0xfff
-A ISTIO_OUTPUT -m connmark --mark 0x111/0xfff -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT
* nat
-N ISTIO_OUTPUT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -d <IP_ADDRESS> -p tcp -m tcp -j ACCEPT
-A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT
-A ISTIO_OUTPUT ! -d <IP_ADDRESS>/32 -o lo -j ACCEPT
-A ISTIO_OUTPUT ! -d <IP_ADDRESS>/32 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001
COMMIT  ns=default name=productpage-v1-d5789fdfb-tk78f

Before adding the pods to the mesh, I am able to navigate to the productpage page and I see the expected output from the guide: http://localhost:8080/productpage
However, after adding the pods to the mesh, opening productpage again returns this:
upstream connect error or disconnect/reset before headers. reset reason: connection termination

This is on 1.23:
$ istioctl version
client version: 1.23.0
control plane version: 1.23.0
data plane version: 1.23.0 (6 proxies)
$ kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.30.3+k3s1

The original issue appears to be fixed, I can add pods to the mesh by going through the Getting Started guide up to this point: https://istio.io/latest/docs/ambient/getting-started/secure-and-visualize/#add-bookinfo-to-the-mesh
I don't see any issues adding the pods in the logs:
2024-08-21T19:44:34.463634Z     info    cni-agent       adding pod to the mesh  ns=default name=productpage-v1-d5789fdfb-tk78f
2024-08-21T19:44:34.511631Z     info    cni-agent       Running iptables-nft-restore with the following input:
* mangle
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_PRERT -m mark --mark 0x539/0xfff -j CONNMARK --set-xmark 0x111/0xfff
-A ISTIO_PRERT -s <IP_ADDRESS> -p tcp -m tcp -j ACCEPT
-A ISTIO_PRERT ! -d <IP_ADDRESS>/32 -p tcp -i lo -j ACCEPT
-A ISTIO_PRERT -p tcp -m tcp --dport 15008 -m mark ! --mark 0x539/0xfff -j TPROXY --on-port 15008 --tproxy-mark 0x111/0xfff
-A ISTIO_PRERT -p tcp -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ISTIO_PRERT ! -d <IP_ADDRESS>/32 -p tcp -m mark ! --mark 0x539/0xfff -j TPROXY --on-port 15006 --tproxy-mark 0x111/0xfff
-A ISTIO_OUTPUT -m connmark --mark 0x111/0xfff -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT
* nat
-N ISTIO_OUTPUT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -d <IP_ADDRESS> -p tcp -m tcp -j ACCEPT
-A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT
-A ISTIO_OUTPUT ! -d <IP_ADDRESS>/32 -o lo -j ACCEPT
-A ISTIO_OUTPUT ! -d <IP_ADDRESS>/32 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001
COMMIT  ns=default name=productpage-v1-d5789fdfb-tk78f

Before adding the pods to the mesh, I am able to navigate to the productpage page and I see the expected output from the guide: http://localhost:8080/productpage
However, after adding the pods to the mesh, opening productpage again returns this:
upstream connect error or disconnect/reset before headers. reset reason: connection termination

This is on `1.23`:
$ istioctl version
client version: 1.23.0
control plane version: 1.23.0
data plane version: 1.23.0 (6 proxies)
$ kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.30.3+k3s1

I have the same problem

The original issue appears to be fixed, I can add pods to the mesh by going through the Getting Started guide up to this point: https://istio.io/latest/docs/ambient/getting-started/secure-and-visualize/#add-bookinfo-to-the-mesh
I don't see any issues adding the pods in the logs:
2024-08-21T19:44:34.463634Z     info    cni-agent       adding pod to the mesh  ns=default name=productpage-v1-d5789fdfb-tk78f
2024-08-21T19:44:34.511631Z     info    cni-agent       Running iptables-nft-restore with the following input:
* mangle
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_PRERT -m mark --mark 0x539/0xfff -j CONNMARK --set-xmark 0x111/0xfff
-A ISTIO_PRERT -s <IP_ADDRESS> -p tcp -m tcp -j ACCEPT
-A ISTIO_PRERT ! -d <IP_ADDRESS>/32 -p tcp -i lo -j ACCEPT
-A ISTIO_PRERT -p tcp -m tcp --dport 15008 -m mark ! --mark 0x539/0xfff -j TPROXY --on-port 15008 --tproxy-mark 0x111/0xfff
-A ISTIO_PRERT -p tcp -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ISTIO_PRERT ! -d <IP_ADDRESS>/32 -p tcp -m mark ! --mark 0x539/0xfff -j TPROXY --on-port 15006 --tproxy-mark 0x111/0xfff
-A ISTIO_OUTPUT -m connmark --mark 0x111/0xfff -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT
* nat
-N ISTIO_OUTPUT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -d <IP_ADDRESS> -p tcp -m tcp -j ACCEPT
-A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT
-A ISTIO_OUTPUT ! -d <IP_ADDRESS>/32 -o lo -j ACCEPT
-A ISTIO_OUTPUT ! -d <IP_ADDRESS>/32 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001
COMMIT  ns=default name=productpage-v1-d5789fdfb-tk78f

Before adding the pods to the mesh, I am able to navigate to the productpage page and I see the expected output from the guide: http://localhost:8080/productpage
However, after adding the pods to the mesh, opening productpage again returns this:
upstream connect error or disconnect/reset before headers. reset reason: connection termination

This is on `1.23`:
$ istioctl version
client version: 1.23.0
control plane version: 1.23.0
data plane version: 1.23.0 (6 proxies)
$ kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.30.3+k3s1

I tried setting up a kubespray cluster instead of my usual k3s one, and I do not experience the same issue, following the exact same Istio Ambient Getting Started procedure.  This is using the same calico configuration (my tigera-operator based, not kubespray's built-in config).

I tried setting up a kubespray cluster instead of my usual k3s one, and I do not experience the same issue, following the exact same Istio Ambient Getting Started procedure. This is using the same calico configuration (my tigera-operator based, not kubespray's built-in config).

Then it is very likely a k3s-specific path config override that is needed for your case, as mentioned in: https://istio.io/latest/docs/ambient/install/platform-prerequisites/#k3s
either the CNI bin dir, CNI conf dir, or CNI netns dir is unique for your k3s setup, somehow.
What we have documented there should work for a vanilla k3s install. If there's an easy way to reproduce your k3s setup, figuring out which of those 3 is the problem would be possible.

@gclawes Is this still an issue for you?
It looks like k3s recently changed how they handle CNI binary paths between versions so you should be able to use /var/lib/rancher/k3s/data/cni as the CNI bin dir override with recent K3S releases.

I'm not seeing this anymore on my setup on v1.24.1, though again my setup does not use the k3s CNI, so I do not believe the CNI path config for k3s was the source of this issue.
I disable the k3s built-in flannel with --flannel-backend=none --disable-network-policy and install calico via tigera-operator, which uses the standard CNI dirs:
ubuntu@pi01:~$ sudo ls -lah /etc/cni/net.d/
total 16K
drwx------ 2 root root 4.0K Dec  4 02:37 .
drwxr-xr-x 3 root root 4.0K Dec  4 02:20 ..
-rw-r--r-- 1 root root 1.4K Dec  4 02:37 10-calico.conflist
-rw------- 1 root root 2.1K Dec  4 02:34 calico-kubeconfig
ubuntu@pi01:~$ ls -lah /opt/cni/bin/
total 197M
drwxr-xr-x 2 root root 4.0K Dec  4 02:49 .
drwxr-xr-x 3 root root 4.0K Dec  4 02:29 ..
-rwxr-xr-x 1 root root 4.2M Dec  4 02:30 bandwidth
-rwxr-xr-x 1 root root  63M Dec  4 02:30 calico
-rwxr-xr-x 1 root root  63M Dec  4 02:30 calico-ipam
-rwxr-xr-x 1 root root 2.7M Dec  4 02:31 flannel
-rwxr-xr-x 1 root root 3.8M Dec  4 02:31 host-local
-rwxr-xr-x 1 root root  49M Dec  4 02:49 istio-cni
-rwxr-xr-x 1 root root 3.8M Dec  4 02:31 loopback
-rwxr-xr-x 1 root root 4.3M Dec  4 02:31 portmap
-rwxr-xr-x 1 root root 4.0M Dec  4 02:31 tuning

versions:
$ kubectl version
Client Version: v1.31.3
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s1

$ calicoctl version
Client Version:    3.29.1
Git commit:        ddfc3b1ea
Cluster Version:   v3.29.1
Cluster Type:      typha,kdd,k8s,operator,bgp

$ istioctl version
client version: 1.24.1
control plane version: 1.24.1
data plane version: 1.24.1 (2 proxies)

ok, if this or something related crops up again, let us know - thanks!


MLPerf: add flagfiles, remove eval games against target

Eval games against target will be played out-of-band at the end of the training run.

Addressed comments. I also changed gating_win_rate to 0.5 as we discussed yesterday.

Are the files for flags/19/ put as a placeholder for now ? flags/19/train.flags shows conv_width=64 while dual_net.py has that set to 256 for 19x19.


Update plutus to 1.13

Description

Checklist

[ ] Commit sequence broadly makes sense and commits have useful messages
[ ] New tests are added if needed and existing tests are updated
[ ] When applicable, versions are updated in .cabal and CHANGELOG.md files according to the
versioning process.
[ ] The version bounds in .cabal files for all affected packages are updated. If you change the bounds in a cabal file, that package itself must have a version increase. (See RELEASING.md)
[ ] All visible changes are prepended to the latest section of a CHANGELOG.md for the affected packages. New section is never added with the code changes. (See RELEASING.md)
[x] Code is formatted with fourmolu (use scripts/fourmolize.sh)
[x] Cabal files are formatted (use scripts/cabal-format.sh)
[x] hie.yaml has been updated (use scripts/gen-hie.sh)
[ ] Self-reviewed the diff

Thanks @lehins . I think we don't require plutus 1.13 in API, we should be good after https://github.com/input-output-hk/cardano-api/pull/270 so I won't be no longer working on this.

@carbolymer Cool. I'll have someone from ledger pick up and finish this PR at some later point.
Thank you.

Closing as duplicate of #3764


chore: afg-ext update

Signed-off-by: Rolson Quadras<EMAIL_ADDRESS>

Codecov Report

Merging #103 (3013927) into main (8014689) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #103   +/-   ##
=======================================
  Coverage   88.45%   88.45%
=======================================
  Files           7        7
  Lines         745      745
=======================================
  Hits          659      659
  Misses         47       47
  Partials       39       39

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8014689...3013927. Read the comment docs.


#192 core clean up usbc

USBC.cpp has 3 parts:

USB Device Driver
TinyCLR API
HAL layer.

USB Device Driver and TinyCLR API (locate on top and middle of file) should be same for all devices.

Waiting for more changes, going to move the generic USB Device Driver and TinyCLR API that should be the same for all devices into a generic device driver.


Changed track status to endangered

Did read the instructions carefully enough.   Myself and kotp are the only two presently maintaining this track.

Sorry, I was too eager to merge the last one..