OKD 4.13.0 IPI build errors

Discussed in https://github.com/okd-project/okd/discussions/1673

Originally posted by pzi123 July 30, 2023
Tried to install OKD 4.13.0 on vsphere. IPI failed unfortunately.  All build details are collected here https://github.com/pzi123/tmp/tree/main/4.13.0-error . The commands ran (RUNME) below. I would appreciate any feedback. As a  background on the vsphere I used I successfully built OKD cluster 4.8.57 in this environment with details on that installation here https://github.com/pzi123/tmp/tree/main/4.8.57-ok .
dig +noall +answer @<IP_ADDRESS> api.okd2.p31.home.pzi.net
dig +noall +answer @<IP_ADDRESS> api-int.okd2.p31.home.pzi.net
dig +noall +answer @<IP_ADDRESS> console-openshift-console.apps.okd2.p31.home.pzi.net

here=`pwd`

cd ~/.ssh
ssh-add -D
ssh-add id_rsa
ssh-add -l

cd $here
rm -rf inst
mkdir inst
cp tmp/install-config.yaml inst

wget https://github.com/okd-project/okd/releases/download/4.13.0-0.okd-2023-07-23-051208/openshift-install-linux-4.13.0-0.okd-2023-07-23-051208.tar.gz
wget https://github.com/okd-project/okd/releases/download/4.13.0-0.okd-2023-07-23-051208/openshift-client-linux-4.13.0-0.okd-2023-07-23-051208.tar.gz

tar xvf openshift-install-linux-4.13.0-0.okd-2023-07-23-051208.tar.gz
#./openshift-install create cluster install-config --dir=inst
./openshift-install create cluster --log-level=debug --dir=inst 2>&1 | tee log.txt

grep -i error log.txt > errors.txt

Hi,
We are not working on FCOS builds of OKD any more. Please see these documents...
https://okd.io/blog/2024/06/01/okd-future-statement
https://okd.io/blog/2024/07/30/okd-pre-release-testing
Please test with the OKD SCOS nightlies and file a new issue as needed.
Many thanks,
Jaime


Issue on: https://docs.microsoft.com/en-us/power-bi/collaborate-share/service-report-subscribe

The following consideration on the page mentioned in the title is not reliable:
"Currently, email subscriptions for reports and dashboards using live connection datasets aren't supported when subscribing users other than yourself, except for paginated reports. Read more about subscribing to paginated reports."
We were successful in subscribing report (using live connection to PBI Dataset) to more than one user.
In my opinion, the documentation needs to be modified.
Thanks,
Sahib

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 830da8e8-8526-0e6a-d83f-0c804226e006
Version Independent ID: 3e7b9d31-133d-1b2f-fded-d9218284155b
Content: Subscribe others to your reports and dashboards - Power BI
Content Source: powerbi-docs/collaborate-share/service-report-subscribe.md
Service: powerbi
Sub-service: pbi-collaborate-share
GitHub Login: @maggiesMSFT
Microsoft Alias: maggies

Thanks for your comment, @mukhijasahib.
I've assigned this issue to the doc author for review.

@KesemSharabi & team: just wanted to initiate a gentle follow up. let me know if any more details are needed.

Thanks, @mukhijasahib !
Adding @mihart , who's currently working on the subscriptions articles.

Hi @mukhijasahib,
I've contacted the PM for this feature and will update the article when I hear back.

Hi @mukhijasahib ,
Thank you for your suggestion.  We've made the update and set the article to republish.
All the best,
The DOCS team


nwnt req bump to 1.3.0

nwnt <1.3 not writing out empty lists means a version bump is absolutely necessary to prevent crashes in dialog. No nasher change though.

Turns out you were right when you pointed out the empty lists. Should've listened then. Oh well, confirmed to be a problem now (Causes either fail-to-start conversations or crashes mid-dialog)

It also dawns on me that because it's built in, probably have to do a nasher-version in order to make the releases not have the problem. Sorry about all this.


Add loading screen on labeling page

Expected Behavior
There should be loading screen on the labeling page so the user is informed of the actual state of the scans. (If they are loaded or not)
Actual Behavior
There is no loading screen. When scans are loading marker is empty and it has gray background. It is misleading user that something is wrong.
Steps to Reproduce the Problem

Log in,
Go to 'Labeling'
Wait for scan to load.

Duplicate of #45


[SD-1825] Respond to http4s MessageFailure with BadRequest

Adds middleware to handle any MessageFailures produced, turning them into ApiErrors.

[Current coverage][cc-pull] is 73.89%

Merging [#1234][cc-pull] into [master][cc-base-branch] will decrease coverage by 0.05%

@@             master      #1234   diff @@
==========================================
  Files           185        185
  Lines         10984      10992     +8
  Methods       10737      10747    +10
  Messages          0          0
  Branches        118        116     -2
==========================================
  Hits           8122       8122
- Misses         2862       2870     +8
  Partials          0          0

Powered by Codecov. Last updated by [99c45ab...a4cb03d][cc-compare]
[cc-base-branch]: https://codecov.io/gh/quasar-analytics/quasar/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/quasar-analytics/quasar/compare/99c45abe0381a8b7a1793f1ee53559baa263dbbd...a4cb03d70675ea146b88f737b598a7563c623ec4?src=pr
[cc-pull]: https://codecov.io/gh/quasar-analytics/quasar/pull/1234?src=pr

LGTM.


Simplify rpcDoRequest

Pull Request Test Coverage Report for Build 435647724

20 of 21 (95.24%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.08%) to 39.27%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

telegram/rpc.go
20
21
95.24%

Totals

Change from base Build 435516418:
0.08%

Covered Lines:
2304

Relevant Lines:
5867

💛  - Coveralls

Closing due to #62


add SDIFF and SDIFFSTORE.

relation #178

Thank ,many doubts have answers nom.

@TalZaccai Hi. I referred to the SortedSetDifference command and I need to reconsider.

@tisilent My apologies, I think I misspoke with some of the previous comments I've had, now that I'm looking at similar command implementations. Let me take another look at this PR...

@TalZaccai Conflict resolved.

@TalZaccai Next time, I will be careful...


i18n: update spanish translations

Updated translations:

banip
openvpn
simple-adblock
statistics
transmission
travelmate
uhttpd
unbound

Signed-off-by: Franco Castillo<EMAIL_ADDRESS>

Thanks


[Feature Request] Windows agent support

Now, portainer agent doesn't suport run in Windows server, is there any goal will be a Windows agent support ?
@deviantony

Yes, it is in our plans.
Related to #1186

That npipe thing will take a while as there is quite many dependencies but how about if we just create agent.exe which can run directly on Windows nodes and connect to tcp://localhost:2375 ?

Hi @olljanat, agree, although that'd be better to deploy it as container.
I can add a -H flag for the agent (with a default value of unix:///var/run/docker.sock) that you can tune to use any TCP address (-H tcp://localhost:2375 for example). Would you be keen to test it?

@deviantony Yes, -H parameter and binary on .exe format should enough as start point. I can do testing for it.

Hello all.
@deviantony I can test that to. Just deployed the agents on our swarm cluster, but only on linux nodes and we have windows server core 1709 and 1803 on our swarm cluster.

@deviantony when I can expect to see it?

It's actually a bit harder than just adding a -H parameter and I'm a bit busy on other topics at the moment, but this is in our priority list, we'll keep you posted.

Btw. I did some investigations how Rancher have solved this issue on their 1.6 version.
They use docker image to provide binaries/generate configs for Windows but actual agent service is installed directly to host. You can try that using these commands:
New-Item -Path 'C:\Program Files\rancher' -Type Directory
$obj=$(docker run --rm -v "C:\Program Files\rancher:C:\Program Files\rancher" rancher/agent-windows:v0.2.0 -RegisterUrl http://<IP_ADDRESS>:8080/v1/scripts/FAC4FECCC3B20AD98176:1514678400000:ax9B3cegEyRUTUrA74mB2CsiyK0)
$obj |& "C:\Program Files\rancher\startup.ps1"

You can find scripts which they have inside of docker image from here: https://github.com/rancher/rancher/tree/v1.6/agent-windows
and actual agent build scripts from here: https://github.com/rancher/agent/tree/v1.6/scripts/windows-ci
I think that similar solution would works with portainer Windows agent too.

@aiminickwong / @danielfrancora note that there is test version of Portainer with named pipe support on https://github.com/portainer/portainer/pull/2018
It would be nice if you can also test and comment there to how it works as that it is pre-requirement for Portainer Windows agent.

This will be available with the next version of the agent (will be available with the next Portainer release).


ensure fresh install works even if ifadv missing

Was just testing a new installation of talkr in an updated version of R (3.4.4). Building the package from github with devtools failed because it tries to build the vignette which assumes presence of ifadv which isn't there yet on the very first run.
@PabRod can you confirm/reproduce?
To reproduce:

fresh R version
fresh installation of Rstudio without any packages
run install.packages("devtools")
run devtools::install_github("elpaco-escience/talkr", build_vignettes = TRUE)

Produces this error:
── R CMD build ─────────────────────────────────────────────────────────────────────────────
✔  checking for file 'C:\Users\markd\AppData\Local\Temp\RtmpEl5gUg\remotes2ddc85a6197\elpaco-escience-talkr-73639d0/DESCRIPTION'
─  preparing 'talkr': (356ms)
✔  checking DESCRIPTION meta-information ...
─  installing the package to build vignettes
E  creating vignettes (5.7s)
   --- re-building 'workflows.Rmd' using rmarkdown

   Quitting from lines 35-38 [unnamed-chunk-2] (workflows.Rmd)
   Error: processing vignette 'workflows.Rmd' failed with diagnostics:
   there is no package called 'ifadv'
   --- failed re-building 'workflows.Rmd'

   SUMMARY: processing the following file failed:
     'workflows.Rmd'

   Error: Vignette re-building failed.
   Execution halted
Error: Failed to install 'talkr' from GitHub:
  ! System command 'Rcmd.exe' failed

Can also confirm installation does work after first installing ifadv.

TL;DR: Remotes are not allowed in CRAN.
In the CRAN guide, specifically in the Devtools dependencies subsection, we can read the following:

When you submit your package to CRAN, all of its dependencies must also be available on CRAN. For this reason, release() will warn you if you try to release a package with a Remotes field.

Quick fix
Replace ifadv::ifadv calls by an auxiliary function that accesses and loads the data into memory.
Possible fix
Consider publishing ifadv package on CRAN too. Would that make sense?

This would indeed not be a problem if both are on CRAN but note that this bug itself is unrelated to CRAN or its submission rules: installing talkr before installing ifadv fails


New resource: Add Alert resource type to cost-management

This checklist is used to make sure that common issues in a pull request are addressed. This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
PR information

[ ] The title of the PR is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
[ ] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
[ ] If applicable, the PR references the bug/issue that it fixes.
[ ] Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger

[ ] I have read the contribution guidelines.
[ ] My spec meets the review criteria:

[ ] The spec conforms to the Swagger 2.0 specification.
[ ] The spec follows the guidelines described in the Swagger checklist document.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.

Automation for azure-sdk-for-js

Automation for azure-sdk-for-python

Automation for azure-sdk-for-java

Automation for azure-sdk-for-node

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-go

Can one of the admins verify this patch?

@saschein could you fix the errors reported by model validator?

@saschein could you fix the errors reported by model validator?

@anuchandy Sorry, missed this - fixed. Thanks


Setting GUI testing to be executed using headless way

Setting GUI testing to be executed using headless way

need to add link here so we can find the code change that went in
https://gerrit.openbmc-project.xyz/#/c/11058/


Ctrl+Shift+Up no longer works to move item up

Outliner plugin v4.1.1
Obsidian v1.0.3
Have tested with all plugins disabled except this one and using default template.
ctrl+shift+up moves a standard bullet item, will not move a bullet item in an outline, either at top level or any subsidiary levels

@LeslieP-TX thank you for the feedback.
I can't reproduce the issue. could you please make a screen recording.

@LeslieP-TX thank you for the feedback.
I can't reproduce the issue. could you please make a screen recording.

Same issue here.
Reproduce the issue:

Create a list (3+ item)
Ctrl + Shift + Down is fine. It moves the item down. But Ctrl+Shift+Up, nothing happen.

Same with Ctrl+Down will unfold the list, but Ctrl+Up won't

@vslinko is there any update?

I've seen cases where it doesn't work properly if a character other than the tab character, such as a space, is put in front of the bullet(-). If even one of the documents has such a place, list up/down will not work for the entire document.
In this case, deleting all other characters besides the tab character before the bullet(-) works fine.

I changed the shortcut to another binding, e.g., Alt+Up, and it works. That's weird.

@khanhtoan529 that makes sense - this was the case for me.

It seems according to the thread there are some workarounds. Closing the issue. If the problem persists, please create a new issue.


Missing minus sign in equation 16.13

I believe there is a small typo in equation 16.13. There should be a minus sign in front of the log as in equation 6 of the original paper.

Thanks, fixed!


Class struct is unresolved unless it is in a folder named as the namespace the class is in

Describe the bug
When putting a class in a namespace, and putting the struct for it in a folder that is not in the structure of the namespace e.g. namespace is namespace::Class but the struct in the Data Type Manager is located in namespace/Subfolder/Class, the decompiler will not be unable to resolve the type.
To Reproduce

Create a Class that is not in the global namespace with a __thiscall function inside it and a corresponding struct
Move the struct outside of the namespace's folder
View the __thiscall function in the decompiler
The decompiler will say Low-level Error: Unable to resolve type: X

Expected behavior
The decompiler should find the struct used for the class in any category/folder. This is the case in Ghidra 10.1.5, but not in 10.2.
Screenshots

Environment:

OS: Arch Linux 5.10.68-1-vfio-lts
Java Version: 11.0.17
Ghidra Version: 10.2
Ghidra Origin: Arch Linux Community Repository

It seems that https://github.com/NationalSecurityAgency/ghidra/commit/0acb497263c209ac59b306ebdea8110547205058 is the cause of this issue. A way to disable this would be useful, as without it it is easier to organize classes in their original folder structure, and this "feature" breaks existing projects.

There is no association to the class. If you create a struct with the same name as the class, the first one with that name will be taken. If I remember correctly the search is done alphabetically. I've been crying about it for a few years now.

There is no association to the class. If you create a struct with the same name as the class, the first one with that name will be taken. If I remember correctly the search is done alphabetically. I've been crying about it for a few years now.

What you're describing is the wanted behavior that was the case, ...until 10.2

There is no association to the class. If you create a struct with the same name as the class, the first one with that name will be taken. If I remember correctly the search is done alphabetically. I've been crying about it for a few years now.

What you're describing is the wanted behavior that was the case, ...until 10.2

That should not be the wanted behavior. If you have two or more different classes with the same name the datatype chosen for the this parameter will be the same for each different class instead of the corresponding different struct for each class.

I did not know that classes could actually be associated directly to structs, AFAIK there is no way to find out which struct a class is linked to if any. But your explanation has solved my problem, thanks!

I did not know that classes could actually be associated directly to structs, AFAIK there is no way to find out which struct a class is linked to if any. But your explanation has solved my problem, thanks!

They are not. There is no association. I'm saying that it is something that should be and you should be able to move it but you currently can not. Renaming a class or the struct it is using (if it is the only one with the same name as the class) will cause the same problem.


Check if there is a worker that has been previously installed

In a recent practice, I found that if developers use a updated hook to guide users to a new version, they tend to fall into a trap.
Let me explain it with the following example steps:

First, the user visits our site for the first time. The service worker caches all the files for the current version A.
After a while, users visited our site for a second time. At this point, the browser directly displays the contents of version A to the user because of the Service worker interception.
At the same time, the browser requests sw.js from our server and is delighted to find that we had released version B, so the browser starts getting all the files for version B in the background with the new sw.js.
When the browser gets all the files for version B, our library register-service-worker emits the updated hook, so we pop up a prompt on the page asking the user if they want to use the new version B. If the user chooses "yes", then we will activate the worker of version B by calling skipWaiting, and then automatically refresh the page with window.location.reload. In most cases, up to this point, everything is going the way we expect it to.
Unfortunately, there are times when users choose not to update immediately, or they simply close our tabs. So this time we assume that the user chose to ignore our prompt.
Next, the user refreshes the page.
At this point, we found that our site no longer asked users if they want to use new version B, even if the worker for version B was in the waiting state.

So, what's the problem?
First, when version B is in the waiting state for the first time, simply refreshing the page does not activate B's worker. The reason for this is explained in this article:

Even if you only have one tab open to the demo, refreshing the page isn't enough to let the new version take over. This is due to how browser navigations work. When you navigate, the current page doesn't go away until the response headers have been received, and even then the current page may stay if the response has a Content-Disposition header. Because of this overlap, the current service worker is always controlling a client during a refresh.

Second, we shouldn't call skipWaiting before the user confirms, which is also explained in the above article:

Caution: skipWaiting() means that your new service worker is likely controlling pages that were loaded with an older version. This means some of your page's fetches will have been handled by your old service worker, but your new service worker will be handling subsequent fetches. If this might break things, don't use skipWaiting().

Thus, we find that in this case, when the new worker has entered the waiting (to be activated) state, we are supposed to go into the process of guiding the user to update. However, we can not achieve it simply by listening to a specified hook.
IMO, is that this case should trigger the updated event, and that's what updated is supposed to do.
In addition, this example shows what developers need to do when updated event does not cover the above case:
https://github.com/imyelo/pokequest-wiki/commit/023bb812585378c7a0981be33c33f404e1b92259
What do you think? @yyx990803

Will this fire the hook twice if:

There's a previous version waiting
And there is a new version found

More precisely, this special situation occurs only before the user closes all clients (pages) in the same scope. The new worker will be automatically activated when the page is reopened in a new tab.

@imyelo in that case I think it's better to ensure the hook is fired at most once for each session.

Yeah, I understand your concern. But the above situation is really difficult for the users to find out when they only read the examples in the readme.
Or maybe we can

drop the updated event of version C to keep the hook only triggered once?
or define a new special hook other than updated for this?

I also encountered similar requirements a week ago and found this solution.
And I still cannot find enough arguments for me to support this workaround
But there is one thing should be OK, when the synchronous value waiting exists, we shouldn't bind updatefound event.

@ulivz I just merged #7 - does that somewhat address the need?

@yyx990803
I just tested for that, when I omitted the first updatefound event and didn't skipWaiting the newest SW, when I refreshed the browser, the debugger still kept following status:

But onupdatefound wasn't be triggered again. so #7 didn't address our need.

Closed in favor of #9, thanks for the contribution!


.dmg file is corrupt for OS Big Sur

3.3.8a9.dmg keeps getting the "Electrum-XZC" is damaged and cant be opened.

I cannot reproduce it. Please check you system security settings which should be the reason for this.

I cannot reproduce it. Please check you system security settings which should be the reason for this.

Heres what I'm getting, just re-downloaded from github. MacOS BigSur 11.1

Heres what I'm getting, just re-downloaded from github. MacOS BigSur 11.1

Yes, we don't have signatures on the image. Please try

Preferred: running the xattr -cr /path/to/electrum-3.3.8a9.dmg in a terminal
Less secure: lower the whole system security level as described here: https://secure.clcbio.com/helpspot/index.php?pg=kb.page&id=323 or https://apple.stackexchange.com/questions/175634/all-downloaded-files-cannot-be-opened-as-they-are-damaged-and-should-be-moved-t
Don't forget to restore the setting after installing the app

Yes, we don't have signatures on the image. Please try

Preferred: running the xattr -cr /path/to/electrum-3.3.8a9.dmg in a terminal
Less secure: lower the whole system security level as described here: https://secure.clcbio.com/helpspot/index.php?pg=kb.page&id=323 or https://apple.stackexchange.com/questions/175634/all-downloaded-files-cannot-be-opened-as-they-are-damaged-and-should-be-moved-t
Don't forget to restore the setting after installing the app


add loud warning about prod

link to helm chart. our current operating docs kind of suck, would like to fix
this too, but alas. helm seems a better place to point anyway, the base issue
is just to inform people here

if you have a fever, I have a prescription...


No instructions about how to use config file

It's better to add how to use config file in the document, I spent about an hour searching for how to properly use the config file, and it's buried inside the comment section of a 3 years old thread.

https://github.com/typicode/json-server/issues/247#issuecomment-198086124

Also, there is no instruction about what are the options in the config file.

And why can't we indicate which file to watch inside the config file?

still haven't figured this out

why can't we indicate which file to watch inside the config file

which is crazy, that would be such a useful feature

It should be enough to update the README with:

Watching the db.json is only included in the CLI.
In order to watch when using json-server as a module you should use nodemon
e.g. in your package.json add   "scripts": {     "start": "nodemon server.js"   }

@typicode would you accept a pull request with just these 4 lines for the README?


rbd: update volume/snapshot-volume details on the image as metadata

Describe what this PR does
This helps Monitoring solutions without access to Kubernetes clusters to display the details of the PV/PVC/PVCNameSpace and snapshot-name/snapshot-namespace/snapshotcontent-name in their dashboard.
Is there anything that requires special attention
Scenarios covered:

Validate metadata is present after Create of a PV
Validate metadata is set after Provisioner pod is restarted
Validate metadata is updated after a PVC is deleted and PV is reattached to a new PVC
Validate metadata is updated after a PVC is deleted and PV is reattached to a new PVC on a different namespace
Validate metadata is set on snapshot create

Related issues
Fixes: #2874
Other details
Currently, this PR uses K8s CO specific keywords
// PV metadata keys
pvcNameKey      = "csi.storage.k8s.io/pvc/name"
pcNamespaceKey = "csi.storage.k8s.io/pvc/namespace"
pvNameKey       = "csi.storage.k8s.io/pv/name"

// snapshot metadata keys
volSnapNameKey        = "csi.storage.k8s.io/volumesnapshot/name"
volSnapNamespaceKey   = "csi.storage.k8s.io/volumesnapshot/namespace"
volSnapContentNameKey = "csi.storage.k8s.io/volumesnapshotcontent/name"

we might want to use generalized naming for keywords, let's discuss it on this PR.

Show available bot commands
These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)
/retest all: run this in case the CentOS CI failed to start/report any test
progress or results

/retest all

/retest ci/centos/mini-e2e/k8s-1.23

/retest ci/centos/mini-e2e-helm/k8s-1.22

/retest ci/centos/mini-e2e-helm/k8s-1.23

/retest ci/centos/mini-e2e/k8s-1.22

@Madhu-1 previously the reattach case was working fine and e2e was passing, after the rebase its failing any lead please?

@Madhu-1 previously the reattach case was working fine and e2e was passing, after the rebase its failing any lead please?

Mar 29 10:40:26.687: INFO: PersistentVolume pvc-a9a107d4-447f-41d7-81d3-4cdade20a21d found and phase=Bound (1.618557ms)
Mar 29 10:40:26.694: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-696bff5bc6-v4hd6 ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Mar 29 10:40:26.694: INFO: >>> kubeConfig: /root/.kube/config
Mar 29 10:40:26.695: INFO: ExecWithOptions: Clientset creation
Mar 29 10:40:26.695: INFO: ExecWithOptions: execute(POST https://<IP_ADDRESS>:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-696bff5bc6-v4hd6/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+ls+--format%3Djson+--pool%3Dreplicapool&container=rook-ceph-tools&container=rook-ceph-tools&stderr=true&stdout=true %!s(MISSING))
Mar 29 10:40:27.073: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-696bff5bc6-v4hd6 ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Mar 29 10:40:27.073: INFO: >>> kubeConfig: /root/.kube/config
Mar 29 10:40:27.074: INFO: ExecWithOptions: Clientset creation
Mar 29 10:40:27.074: INFO: ExecWithOptions: execute(POST https://<IP_ADDRESS>:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-696bff5bc6-v4hd6/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+ls+--format%3Djson+--pool%3Dreplicapool&container=rook-ceph-tools&container=rook-ceph-tools&stderr=true&stdout=true %!s(MISSING))
Mar 29 10:40:27.447: INFO: ExecWithOptions {Command:[/bin/sh -c rbd image-meta get --pool=replicapool --image=csi-vol-4254aac3-af44-11ec-a96f-5a9681a400cb csi.storage.k8s.io/pvc/name] Namespace:rook-ceph PodName:rook-ceph-tools-696bff5bc6-v4hd6 ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Mar 29 10:40:27.447: INFO: >>> kubeConfig: /root/.kube/config
Mar 29 10:40:27.448: INFO: ExecWithOptions: Clientset creation
Mar 29 10:40:27.448: INFO: ExecWithOptions: execute(POST https://<IP_ADDRESS>:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-696bff5bc6-v4hd6/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+image-meta+get+--pool%3Dreplicapool+--image%3Dcsi-vol-4254aac3-af44-11ec-a96f-5a9681a400cb+csi.storage.k8s.io%2Fpvc%2Fname&container=rook-ceph-tools&container=rook-ceph-tools&stderr=true&stdout=true %!s(MISSING))
Mar 29 10:40:27.868: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-696bff5bc6-v4hd6 ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Mar 29 10:40:27.868: INFO: >>> kubeConfig: /root/.kube/config
Mar 29 10:40:27.868: INFO: ExecWithOptions: Clientset creation
Mar 29 10:40:27.868: INFO: ExecWithOptions: execute(POST https://<IP_ADDRESS>:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-696bff5bc6-v4hd6/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+ls+--format%3Djson+--pool%3Dreplicapool&container=rook-ceph-tools&container=rook-ceph-tools&stderr=true&stdout=true %!s(MISSING))
Mar 29 10:40:28.259: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc-new  rbd-1318    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} to be in Bound state
Mar 29 10:40:28.259: INFO: waiting for PVC rbd-pvc-new (0 seconds elapsed)
Mar 29 10:40:30.261: INFO: waiting for PVC rbd-pvc-new (2 seconds elapsed)
Mar 29 10:40:30.266: INFO: Waiting for PV pvc-cf6ab5d2-9fd9-4cd6-ae16-e89409656174 to bind to PVC rbd-pvc-new

Looks like a new PV is getting created when you delete and recreate the PVC.

/retest ci/centos/mini-e2e-helm/k8s-1.23

Pass false or comment the line in values.yaml and deployment files ?
Let users enable it, if they need it.

Pass false or comment the line in values.yaml and deployment files ? Let users enable it, if they need it.

These are used by our e2e right? I want the e2e to test the same, probably ceph-csi/example/ is intended for users

@pkalever please review your commit messages for typos (documentation in subject), and be more verbose for the "add testcases" one.

On Fri, Apr 01, 2022 at 04:44:56AM -0700, Madhu Rajanna wrote:

@Madhu-1 commented on this pull request.

@@ -68,6 +68,11 @@ func init() {
flag.StringVar(&conf.NodeID, "nodeid", "", "node id")
flag.StringVar(&conf.PluginPath, "pluginpath", defaultPluginPath, "plugin path")
flag.StringVar(&conf.StagingPath, "stagingpath", defaultStagingPath, "staging path")

flag.BoolVar(

what Niels meant was to create the metadata if it's available in the request. let's not have a flag for this one as it will be useful to have the metadata on the ceph resources. is that correct @nixpanic? or set the metadata if it's available and the flag is set to true?

Indeed, I do not think there is a need for this flag. If the metadata is
passed to the CreateVolume call, just set it. Not all options need to be
configurable, that makes it just more complicated for users.

On Fri, Apr 01, 2022 at 04:44:56AM -0700, Madhu Rajanna wrote: @Madhu-1 commented on this pull request. > @@ -68,6 +68,11 @@ func init() { flag.StringVar(&conf.NodeID, "nodeid", "", "node id") flag.StringVar(&conf.PluginPath, "pluginpath", defaultPluginPath, "plugin path") flag.StringVar(&conf.StagingPath, "stagingpath", defaultStagingPath, "staging path") + flag.BoolVar( what Niels meant was to create the metadata if it's available in the request. let's not have a flag for this one as it will be useful to have the metadata on the ceph resources. is that correct @nixpanic? or set the metadata if it's available and the flag is set to true?
Indeed, I do not think there is a need for this flag. If the metadata is passed to the CreateVolume call, just set it. Not all options need to be configurable, that makes it just more complicated for users.

Which means we are depending on --extra-create-metadata=true option of the external provisioner, this option will dump other details too, what if the user is interested in other details? we might unnecessarily add metadata to images wasting a few jiffies of performance.

we might unnecessarily add metadata to images wasting a few jiffies of performance.

I do not think this will be noticeable. My preference is an as simple as possible solution, rather reducing extra configuration options, and not add more for little benefit.

/retest ci/centos/mini-e2e-helm/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.22

/retest ci/centos/mini-e2e-helm/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.22

@Mergifyio rebase

/retest ci/centos/mini-e2e/k8s-1.23

/retest ci/centos/mini-e2e/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.21

Failed with:
Apr  4 16:28:50.446: FAIL: backend images not matching kubernetes resource count,image count 23 kubernetes resource count 21

/retest ci/centos/mini-e2e-helm/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.21

@Mergifyio refresh

/retest ci/centos/mini-e2e-helm/k8s-1.21

@mergifyio refresh

/retest ci/centos/mini-e2e-helm/k8s-1.21

@Mergifyio refresh

@Mergifyio refresh

@pkalever the tests are not passing.. and its blocking the release.. can you take a look at it.?

/retes ci/centos/mini-e2e-helm/k8s-1.21

/retes ci/centos/mini-e2e-helm/k8s-1.21

Failure is due to known reason, @pkalever please check and confirm
Apr  5 07:26:20.106: INFO: ExecWithOptions: execute(POST https://<IP_ADDRESS>:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-6f56cdd85d-99zjs/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+ls+--format%3Djson+--pool%3Dreplicapool&container=rook-ceph-tools&container=rook-ceph-tools&stderr=true&stdout=true %!s(MISSING))
Apr  5 07:26:21.140: FAIL: backend images not matching kubernetes resource count,image count 23 kubernetes resource count 21
backend image Info:
 [csi-vol-28727ce8-b4a9-11ec-b5bc-1eb41208b084 csi-vol-3a6e6c62-b4a9-11ec-b5bc-1eb41208b084 csi-vol-3a6e6c62-b4a9-11ec-b5bc-1eb41208b084-temp csi-vol-3a899665-b4a9-11ec-b5bc-1eb41208b084 csi-vol-3a899665-b4a9-11ec-b5bc-1eb41208b084-temp csi-vol-3aa9b48a-b4a9-11ec-b5bc-1eb41208b084 csi-vol-3aa9b48a-b4a9-11ec-b5bc-1eb41208b084-temp csi-vol-3ac8b719-b4a9-11ec-b5bc-1eb41208b084 csi-vol-3ac8b719-b4a9-11ec-b5bc-1eb41208b084-temp csi-vol-3aeb2abc-b4a9-11ec-b5bc-1eb41208b084 csi-vol-3aeb2abc-b4a9-11ec-b5bc-1eb41208b084-temp csi-vol-3d24312b-b4a9-11ec-b5bc-1eb41208b084 csi-vol-3d24312b-b4a9-11ec-b5bc-1eb41208b084-temp csi-vol-3f502b26-b4a9-11ec-b5bc-1eb41208b084 csi-vol-3f502b26-b4a9-11ec-b5bc-1eb41208b084-temp csi-vol-3fe67f84-b4a9-11ec-b5bc-1eb41208b084 csi-vol-3fe67f84-b4a9-11ec-b5bc-1eb41208b084-temp csi-vol-409d1131-b4a9-11ec-b5bc-1eb41208b084 csi-vol-409d1131-b4a9-11ec-b5bc-1eb41208b084-temp csi-vol-40b7db93-b4a9-11ec-b5bc-1eb41208b084 csi-vol-40b7db93-b4a9-11ec-b5bc-1eb41208b084-temp csi-vol-40e2e952-b4a9-11ec-b5bc-1eb41208b084 csi-vol-40e2e952-b4a9-11ec-b5bc-1eb41208b084-temp]

/retest ci/centos/mini-e2e-helm/k8s-1.21

@Madhu-1  sorry I didn't get what is known reason? is this PR failing CI for a valid reason? meaning is this PR introducing any regression?

@Madhu-1 sorry I didn't get what is known reason? is this PR failing CI for a valid reason? meaning is this PR introducing any regression?

https://github.com/ceph/ceph-csi/issues/2327 covers the known reason

hmmm.. Failed again... @pkalever  @Madhu-1

@Madhu-1 manual merge ?

@Madhu-1 manual merge ?

If we manually merge it will keep happening on all the PRs and it's bad for CI?

hmmm.. Failed again... @pkalever @Madhu-1

This times its failed for something else
pr  5 08:45:18.682: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc-new  rbd-3274  d8d99c66-37a0-4142-85f9-6177bc0cff07  0 2022-04-05 08:45:12 +0100 BST <nil> <nil> map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:rbd.csi.ceph.com] [] [kubernetes.io/pvc-protection]  [{e2e.test Update v1 2022-04-05 08:45:12 +0100 BST FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2022-04-05 08:45:12 +0100 BST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}},"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} }]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:pvc-d8d99c66-37a0-4142-85f9-6177bc0cff07,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} to be in Bound state
Apr  5 08:45:18.682: INFO: waiting for PVC rbd-pvc-new (0 seconds elapsed)
Apr  5 08:45:18.688: INFO: Waiting for PV pvc-d8d99c66-37a0-4142-85f9-6177bc0cff07 to bind to PVC rbd-pvc-new
Apr  5 08:45:18.688: INFO: Waiting up to timeout=10m0s for PersistentVolumeClaims [rbd-pvc-new] to have phase Bound
Apr  5 08:45:18.691: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Pending instead of Bound.
Apr  5 08:45:20.695: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:22.699: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:24.704: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:26.708: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:28.711: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:30.714: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:32.719: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:34.724: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:36.727: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:38.731: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:40.735: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:42.740: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:44.744: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:46.749: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:48.753: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:50.757: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:52.761: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:54.766: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:56.770: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:45:58.774: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:00.778: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:02.782: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:04.786: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:06.790: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:08.795: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:10.799: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:12.804: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:14.809: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:16.812: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:18.817: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:20.825: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:22.829: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:24.834: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:26.838: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:28.842: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:30.846: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:32.850: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:34.854: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:36.858: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:38.862: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:40.866: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:42.870: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:44.875: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:46.879: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:48.885: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:50.889: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:52.894: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:54.898: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:56.902: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:46:58.907: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:00.911: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:02.916: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:04.921: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:06.926: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:08.930: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:10.934: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:12.939: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:14.946: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:16.950: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:18.953: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:20.957: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:22.963: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:24.967: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:26.972: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:28.977: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:30.981: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:32.985: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:34.990: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:36.995: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:39.001: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:41.006: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:43.010: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:45.015: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:47.021: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:49.025: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:51.030: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:53.034: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:55.039: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:57.044: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:47:59.049: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:01.055: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:03.059: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:05.064: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:07.069: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:09.074: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:11.078: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:13.083: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:15.088: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:17.093: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:19.097: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:21.102: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:23.106: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:25.111: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:27.117: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:29.122: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:31.127: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:33.131: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:35.136: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:37.141: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:39.145: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:41.150: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:43.154: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:45.158: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:47.168: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:49.172: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:51.176: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:53.180: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:55.184: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:57.193: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:48:59.198: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:01.207: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:03.211: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:05.215: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:07.220: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:09.224: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:11.228: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:13.232: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:15.239: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:17.243: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:19.247: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:21.252: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:23.256: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:25.261: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:27.267: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:29.271: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:31.276: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:33.279: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:35.283: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:37.287: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:39.291: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:41.295: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:43.298: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:45.302: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:47.306: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:49.311: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:51.314: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:53.319: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:55.323: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:57.334: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:49:59.338: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:01.341: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:03.345: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:05.348: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:07.355: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:09.360: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:11.366: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:13.370: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:15.375: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:17.380: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:19.386: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:21.391: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:23.396: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:25.401: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:27.405: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:29.411: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:31.416: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:33.421: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:35.427: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:37.431: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:39.437: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:41.443: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:43.448: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:45.454: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:47.458: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:49.464: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:51.468: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:53.474: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:55.480: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:57.485: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:50:59.490: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:01.496: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:03.501: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:05.506: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:07.512: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:09.517: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:11.523: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:13.528: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:15.535: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:17.541: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:19.546: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:21.552: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:23.557: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:25.562: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
Apr  5 08:51:27.567: INFO: PersistentVolumeClaim rbd-pvc-new found but phase is Lost instead of Bound.
    Apr  5 08:55:20.115: failed to create new PVC: failed to wait for the pv and pvc to bind: PVC "rbd-pvc-new" did not become Bound: PersistentVolumeClaims [rbd-pvc-new] not all in phase Bound within 10m0s

/retest ci/centos/mini-e2e-helm/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.21

Again failed with #2327:
Apr  5 11:04:06.829: FAIL: backend images not matching kubernetes resource count,image count 22 kubernetes resource count 21

@Mergifyio refresh

/retest ci/centos/mini-e2e-helm/k8s-1.21

@Mergifyio refresh

/restart ci/centos/mini-e2e-helm/k8s-1.22

/restart ci/centos/mini-e2e/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.22

/retest ci/centos/mini-e2e/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.23

/retest ci/centos/mini-e2e/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.23

@Mergify refresh

@mergify rebase


Bonfire: Convert HTML Entities - No real tests to run?

Challenge http://www.freecodecamp.com/challenges/bonfire-convert-html-entities has an issue. Please describe how to reproduce it, and include links to screen shots if possible.
Please correct me if I'm wrong. But I believe the exercise wants us to take characters like "&" and change them into their HTML entities: "&amp", right?
But the tests that are run make no sense (or at least the first one):

assert.strictEqual(convert('Dolce & Gabbana'), 'Dolce & Gabbana', 'should escape characters');should escape characters: expected 'Dolce & Gabbana' to equal 'Dolce & Gabbana'
^-- aren't both the same?

assert.strictEqual(convert('abc'), 'abc', 'should handle strings with nothing to escape');

This is what I see as well. I am not sure what is being asked of us. I wrote a function that passes one test, but not the other even though I have the appropriate escaped character.

I was able to pass this test by simply using
 str = str.replace(/\&/g, "&");

Since the included assertion only tests the & case and the "nothing to escape case", this was way too easy, IMO. It doesn't test for the other cases included in the instructions, such as <, >,", and '.
I think the problem that AkiraLaine is seeing is that when the assertion tests are rendered to the page, it should say:
assert.strictEqual(convert('Dolce & Gabbana'), 'Dolce & Gabbana', 'should escape characters');should escape characters: expected 'Dolce & Gabbana' to equal 'Dolce &amp Gabbana'

When the assertion tests are written into the HTML page in the browser, it's converting those escaped characters back into their final values, so the two cases look identical. Does that make sense?

@kurzninja Oh, I understand now!  Thanks a lot for your help. @arecvlohe I hope you understand too.


Specify how to validate the next data script

We need a proper specification of this, to be included in the EUTXO specification.
This was discussed in the Plutus Call for the 21st of May.
Recording: https://drive.google.com/file/d/1urWw7V2CDXDy934z01zwN69Iwgk3MUot/view?usp=sharing
Slides by Jann: https://docs.google.com/presentation/d/1RCRDajMcBaqPFNjaw2mx7qHaeglHgbrin1x4wjv5qgw/edit#slide=id.g57fbb6ddb5_0_0
GitHub document: https://github.com/input-output-hk/plutus/blob/master/docs/extended-utxo/ValidatingTheNextDataScript.md
GitHub issue #426: https://github.com/input-output-hk/plutus/issues/426
GitHub issue #488: https://github.com/input-output-hk/plutus/issues/488

The exact mechanics of what happens hasn't been pinned down yet. Here's what I'd been thinking, so it's down on paper:

Add a builtin type sealed : * -> *, a builtin function unseal : forall a . sealed a -> a, and a builtin function seal : forall a . a -> sealed a.

The idea is that we need not include both seal and unseal in the set of available builtins when typechecking something.

The redeemer is now passed a value of sealed a for every output in the transaction, in order. So if there are three, where the second has type b, then the signature of the redeemer is () -> sealed b -> () -> redeemerType.

The author of the redeemer knows what outputs there are, so it's their responsibility to add enough arguments.

PROBLEM: what about coin selection?

To ensure that neither the redeemer nor the data scripts create any sealed values, we typecheck them without seal in the builtin environment (the redeemer must at least have sealed in the environment, since it needs to include that in its argument signatures).

The final application, where the redeemer is passed its arguments, gets typechecked with seal. This means that the validating node can pass arguments like seal 1 and it will typecheck.

The advantages of this scheme are:

We don't need to do anything terribly clever to prevent people from creating sealed values, we just exploit the fact that we can typecheck them in different environments to ensure that some builtins aren't used.
We don't necessarily need to code "sealed" as "serialized". We can just have type Serialized a = Sealed (a, ByteString).

cc @mchakravarty @effectfully @j-mueller

Currently we construct the redeemer values at (Haskell) runtime using lift. In your proposal the redeemer becomes an arbitrary function, so we can't use lift. And we should try to avoid requiring the use of the plugin for the redeemer, because we can't rely on having GHC around when we build the redeemer.
I think we would have to do something like this. Similar to the applyScript function that we already have, we add combinators like const, uncurry, flip that we can use to construct a function of the correct type. Then in the contract code, when we know what the transaction* looks like, we use these combinators to construct the redeemer that is expected by the validator.
* modulo coin selection, see below

sealed (a, bytestring) The bytestring is a hash, correct?

PROBLEM: what about coin selection?

I think when we talk about "the transaction" as far as Plutus is concerned the term should always refer to the transaction before coin selection, ie. a transaction that is unsigned and potentially unbalanced. Since for every unbalanced transaction t, if t spends at least one UTXO, there can at most be one proper transaction t' (on the blockchain) that subsumes t, this should be safe.

In your proposal the redeemer becomes an arbitrary function, so we can't use lift.

I think this is reasonably okay: conceptually, I think most redeemers will end up looking like \dataScripts -> (dataScriptData, redeemerData). So you will probably end up writing a function that does just the data script handling using the plugin, which can take the redeemer as its first argument. Then you apply them to get the final redeemer value.
e.g.
redeemerFunc :: RedeemerData -> () -> Interesting -> () -> (RedeemerData, Interesting)
redeemerFunc r _ i _ = (r, i)

redeemer :: () -> Interesting -> () -> (RedeemerData, Interesting)
redeemer = compile redeemerFunc `apply` redeemerLifted

I agree this is pretty awkward, though. But then this whole idiom is pretty awkward :man_shrugging:

The bytestring is a hash, correct?

Yeah. The idea being that a "serialized" value is just a value and its hash, where we trust that it was provided by the validating node. We could put it in a datatype too, though.

I think when we talk about "the transaction" as far as Plutus is concerned the term should always refer to the transaction before coin selection, ie. a transaction that is unsigned and potentially unbalanced.

But at the point of validation we have no way of knowing which inputs/outputs are fee outputs, unless we persuade the chain team to add that as a concept.
Here's a simpler solution, that occurred to me: only pass the data scripts for outputs that have them as arguments. Provided the fee inputs/outputs don't have data scripts, they the won't show up and it's fine. So ignore all those silly unit arguments I had earlier.

I'm not sure that the details of this stuff belong in the EUTXO specification since that's currently agnostic about what goes on at the script level.  But it's got to go somewhere, and I don't think there's anywhere else that's suitable.  I think that in the main document I'll explain the problem and say that some means has to be provided for validators to access the content of data scripts, and put the Plutus-specific details in an appendix.

@michaelpj wrote,

The idea is that we need not include both seal and unseal in the set of available builtins when typechecking something.

What is the problem with allowing unseal? I think, only seal needs special treatment.
Re coin selection, coin selection will only ever add pubkey inputs and outputs. They have neither redeemer scripts nor data scripts. So, I don't think that causes any issues.
Re redeemer type, somebody (I forgot who) suggested to maybe just pass a n-tuple of the data scripts to the redeemer. I think, that is a good idea. In PLC, it'll desugar to its curried form, of course, but I think, it is the simpler API on the source level. This hopefully simplifies the situation with lift as well.

@kwxm Sorry, but I disagree. This mechanism is an essential part of EUTxO and we need to describe it.
We actually don't think, we need to describe the problem much. The document is supposed to be a specification of EUTxO — i.e., we want concisely and precisely describe how it works. We can leave motivation and design space exploration to research papers and tutorials.

@michaelpj

unless we persuade the chain team to add that as a concept.

I think that would be the cleanest solution but I'm not sure how practical it is. I don't have any objections to the proposal, thanks for the clarifications.

@kwxm Sorry, but I disagree. This mechanism is an essential part of EUTxO and we need to describe it.

I take your point, but this is a particularly irritating problem. Until we get to this issue the EUTXO model is essentially agnostic about what goes on at the script level, but now it looks as if we have to start talking about specific Plutus types in order to explain how EUTXO should work, which (a) reduces the generality of the model, and (b) might be a bit unpleasant from the point of veiw of formal specification, since the interface between the transaction level and the script level becomes messy. Given the amount of time that's been spent on this though, maybe we just have to do it that way.

Until we get to this issue the EUTXO model is essentially agnostic about what goes on at the script level, but now it looks as if we have to start talking about specific Plutus types in order to explain how EUTXO should work

I don't know that that's true. At the moment we have two levels of explanation:

Ledger layer. Script layer is opaque, just provides mystery functions. At this point it's still straightforward: you don't even need to change the signature of the validation function, since you're already passing it the pending transaction which has the data scripts in it.
Script layer. We're already specifying quite a lot of details about how you glue the various scripts together in the validation (for example, we probably need to mention the adaptor that turns booleans into errors if we haven't already). So I think it's okay for this to be complicated.

I know I keep banging on about this layering thing but I really do think it's key to making this document make sense.

Re coin selection, coin selection will only ever add pubkey inputs and outputs. They have neither redeemer scripts nor data scripts. So, I don't think that causes any issues.

Provided you ignore outputs without data scripts, which I agree we should do. I was just being silly at first.

What is the problem with allowing unseal? I think, only seal needs special treatment.

Yes indeed, and of course at least the validator will need to use unseal.

Re redeemer type, somebody (I forgot who) suggested to maybe just pass a n-tuple of the data scripts to the redeemer. I think, that is a good idea. In PLC, it'll desugar to its curried form, of course, but I think, it is the simpler API on the source level. This hopefully simplifies the situation with lift as well.

So the idea here is that it makes it easier to e.g. ignore all the data scripts because you always just ignore one argument? It's not quite that simple, because you'll still need to specify the types of all the data scripts so you can give the type of the tuple argument that you're ignoring...

I know I keep banging on about this layering thing but I really do think it's key to making this document make sense.

I've tried to improve that, so maybe you could have a look at the current version. There are still a few  places where things aren't fully described, like precisely how validation takes place and how a PendingTx object gets converted to a script.  I was considering relegating the detailed descriptions of these to an appendix, or parehthetical subsections.

Here's a reason not to pass a tuple to the redeemer: it requires the validating node to know all the types of the data scripts.
If the redeemer has type dt1 -> ... -> dtn -> rt, then the validating node can just do foldr apply dataScripts redeemer to pass them all in.
If the redeemer has type (dt1, ..., dtn) -> rt, then the validating node needs to construct an encoded tuple of the data scripts, given only their serialized forms. This, in general, requires knowing the types of the data scripts, so we'd be requiring more type inference and term construction work from the validating node, whereas otherwise it's just application.

I don't think I understand that. How can you foldr apply redeemer datascripts (I assume the datascripts have to come last, right, it's the lists...), where datascripts is a (I assume from what you are saying) list of serialised scripts. (And if they are not serialised, how can they be in a list?) Sorry, I don't get it at all.

We've deserialized all the scripts into Programs. We have an apply function that applies one program to another. Crucially, this doesn't require you to know anything about them: you just make an Apply node and stuff the two programs in either side.
On the other hand, making a tuple requires you to construct a tuple, which requires you to know the types of the components (recall: a scott-encoded tuple looks like /\r . \(k: a1 ->... -> an -> r). k v1 ... vn, so you need the types in order to give the signature for k).

Add a builtin type sealed : * -> *, a builtin function unseal : forall a . sealed a -> a, and a builtin function seal : forall a . a -> sealed a.

Did this with the "old" builtins machinery: #1226. But we decided not to go with this particular implementation, because it's way too hacky.

We've deserialized all the scripts into Programs. We have an apply function that applies one program to another. Crucially, this doesn't require you to know anything about them: you just make an Apply node and stuff the two programs in either side.

Yes, that is what I thought, but you earlier wrote, "the validating node needs to construct an encoded tuple of the data scripts, given only their serialized forms." We deserialise anyway, so I am not sure why we only have their serialised form in the tuple construction.
So, I think, what you are really saying is that if we only use applications, then we don't need to know the types of the datascript to construct the overall expression to validate, whereas if we need an abstraction (to build a tuple), we need those types. That makes sense. Given that the datascript are deserialised, getting their type isn't particularly hard, but it is an extra step.
An alternative would be to include the types of data scripts (or any serialised program, for that matter) in the serialised form.

We deserialise anyway, so I am not sure why we only have their serialised form in the tuple construction.

Sorry yes, that was me being unhelpful. The fact that they're serialized isn't relevant.

Given that the datascript are deserialised, getting their type isn't particularly hard, but it is an extra step.

It means running type inference, which is expensive. Perhaps okay, though.

An alternative would be to include the types of data scripts (or any serialised program, for that matter) in the serialised form.

We can't do this: we'd be trusting that we weren't being lied to about the type, so we'd have to check it anyway.

We can't do this: we'd be trusting that we weren't being lied to about the type, so we'd have to check it anyway.

We are checking it anyway when we type check the overall expression that we evaluate for validation. If we don't, I could provide a redeemer whose expected argument doesn't match the type of the datascript that it is applied to (among other things).

That's true.
On the other hand, here's a reason to pass a tuple. When we're constructing a redeemer, we want to write something like ignoring @??? actualRedeemer. What goes in the ???? You don't want to have to pass n types, otherwise we'd have to have n different versions of ignoring. If we're passing a tuple, we can just pass the tuple type, i.e. ignoring @(n1, n2, n3....) actualRedeemer.
Could we take the tuple argument but make an uncurried function under the hood? I think this would be tricky, and we'd need TH to do it properly.
So in sum, I'm not sure what to do.

Decision in the Plutus call from 9 July was to go with the curried version as this, after all, provides more flexibility on the Plutus Tx level.

After all that discussion it turns out we still need the types of the data scripts. Why? Because the validating node still needs to call seal on each of the data scripts. But seal is polymorphic, so first of all it needs to be instantiated at the argument type, so we need to know the type of the data script we're wrapping.

There's now a draft of a specification of the current solution to this problem in the EUTXO specification: see https://github.com/input-output-hk/plutus/pull/1426 .
With luck, https://github.com/input-output-hk/plutus/issues/1436 will provide a much simpler solution.

Closed due to being superseded by #1436.


ntr:  pointed nucleus during mitotic cell cycle

term: pointed nucleus during mitotic cell cycle
def
A physical cellular phenotype observed in the vegetative growth phase of the life cycle in which the  nucleus is " pointed".
(Good luck modelling that one!)
Comment: might also be described as "pointed nuclear chromatin"
(maybe this should be the primary phenotype with has_output ?)
child of
FYPO:0000062 abnormal nuclear morphology during vegetative growth
(add synonym "deformed nucleus" to this term?

cut1 C-term OX in  PMID:12390246

pointed nucleus during mitotic interphase FYPO:0005347

edit file: 653f4b9a27d3fcabf1d06845eeb5caab64d1d108
release: 21dbe123051bc992f88e2e2f9cc898264ea4e28d


Battle Box Music Timing Not Matching Event's Music Timing

Hello!
I am wondering about the Battle Box music that is played in the mod itself, as it starts in the wrong time compared to the event.
Here's what the main event music time goes:

Pre-game - 20 seconds - music starts at 1:25
Round countdown - 5 seconds - music at 1:45
Actual game - 50 seconds - music at 1:50
Overtime - 10 seconds - overtime music starts (should already be in mcc island)

What I suggest is the following for the mod itself in MCC Island: (timing is based on what is right now on the server)

Pre-game - 10 seconds (from the round title) - music starts at 1:40
Actual game - 50 seonds - music at 1:50
Overtime - 10 seconds - already applied on server

TL;DR music should start at 1:50 to match the event's music timing.
Thank you so much for reading.
Due

Hi!
The battle box music was originally started at round start rather than pre-game because Island starts the countdown music.
However i have wanted to change it, just not the highest priority things on my list at the moment.
I will do this when i can :D

Added in 1.1.0


Component level tokenization - Loading

User Story

As a [user role below]:
Carbon adopter

I need to:
theme various tokens for the Button component

so that I can:
apply an overall theme in my application

Additional information

This is component-level tokenization work identified in the epic https://github.com/carbon-design-system/ibm-dotcom-library/issues/2503
Latest code can be found at _loading.scss change in: https://github.com/carbon-design-system/carbon/compare/master...asudoh:alternate-type-space

Acceptance criteria

[ ] Component-level tokens applied in Carbon

Carbon rejected the pull request as they do not have the resources to take on the work.  Closing the epic (#2503) and all of the linked issues.
Copy of Jeff's email summarizing the next steps:
We have worked extensively with Akira Sudoh (former Carbon team member) and have recently created a comprehensive code contribution to Carbon to introduce API tokens that would enable the expressive theme. After review from the Carbon team, they have unfortunately rejected our code proposal, stating that it would be too much for their team to support at this time. They have taken considerable resource reductions and their team is much smaller than it was previously which most likely contributed to this decision on their end.
With this news, our team is essentially back to supporting a proxy on our end. However, we have had very valuable learnings when working with Akira, and plan to execute on this new contingency plan:

Identify key Carbon components we would like to support in the expressive theme ◦We can begin with a set of essential components we would like to support, then expand over time

We will make full use of our automated visual regression tool that we recently stood up. This will be extended to our standalone storybook instance with the Carbon components that have the expressive theme enabled. The work that needs to be done here: ◦Activate visual regression testing for Carbon expressive components
-Create additional e2e tests to cover all variants

Take all of Akira's work, and apply the tokenization to the IBM.com Library instead ◦Rather than a hardcoded way of applying the proxy, we are thinking of having the API tokens as an offering from our team instead. This can be ultimately used by other IBM.com Library adopters if the need arises, plus adds more flexibility and scalability on our end

Updates to documentation/design kits for our adopters ◦We will need to own and create all supporting documentation for the expressive theme, so there is some upfront and continuing work around this.

We have almost all of the carbon components with a hardcoded proxy already in place so it serves as a very good starting point. We will be applying the action items above in order to future proof it as a manageable solution going forward.


helm debug ls not showing any additional info

helm --debug ls
Error: could not find tiller

I expect some debug output because tiller is running in the cluster..
helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.11", GitCommit:"dc4f6dda6a08aae2108d7a7fdc2a44fa23900f4c", GitTreeState:"clean", BuildDate:"2018-11-10T20:22:02Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd6
```4"}

I'm going to close this ticket due to inactivity, but please re-open if this still needs to be addressed. Thanks!


Add an acceptance test for the getblocktemplate method

Motivation
The vector and snapshot tests for the getblocktemplate RPC method won't be able to use the non-finalized state to test the contextual validity checks for https://github.com/ZcashFoundation/zebra/pull/5630.
Designs

Use from_script_hash(Mainnet, [0x7e; 20]) as the miner_address in the config
Sync Zebra to the network tip
Call the getblocktemplate RPC method
Assert that the call was successful

Re-opening this ticket, because the test PR was reverted due to intermittent CI failures.

Hey team! Please add your planning poker estimate with Zenhub @arya2 @conradoplg @dconnolly @oxarbitrage @teor2345

So I think this just needs a quick fix to remove an assertion, or fix a bug.
Did you want an estimate for the whole ticket, or just that fix?

Hey @arya2 I think the errors in your PR were from code I wrote, so I'm going to try to fix them.


Move the DAGs folder to a shared Persistent Volume

Description: Instead of "baking" the DAGs folder into each SPS Docker image, move it to a shared location on a Persistent Volume, where updates can be immediately propagated to all Airflow components.
See https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#mounting-dags-from-an-externally-populated-pvc for instructions.
Acceptance Criteria:
o Demonstrated execution of an existing DAG that is written to the shared PV.
o All nightly tests keep working when executed versus the updated SPS with shared DAGs folder.

Notes for implementation:
Using an S3 Mountpoint-backed persistent volume:
https://github.com/apache/airflow/discussions/24049#discussioncomment-8222261

I attempted to use S3 MountPoint as a source for the PVC, but ran into an issue which seems to be related to SELinux and the golden AMI. More info here: https://github.com/awslabs/mountpoint-s3-csi-driver/issues/169

In the meantime, Drew's changes seem to work.
I was able to deploy the system using this branch, and I was able to copy a modified DAG to one of the scheduler pod's /opt/airflow/dags folder and see the changes reflected in the UI. Note that copying the modified DAG to one of the webserver pods does not seem to work - the UI never showed the updated DAG.
Closing the ticket anyway.


Add other part of build

Currently, our build function only runs the appropriate scripts, but doesn't actually save whatever artifacts / random stuff that would have been created from the scripts in any layer.
We should consider / add the appropriate layer to the return so that any scripts that create new files aren't lost in the build process

After talking with Arjun, these cases aren't in the scope of the buildpack so the prospect of adding a layer is to be ignored


fix Merge.input_shape()

The input_shape for a Merge node should be a list comprising the output_shape of each of the nodes it merges. The existing code does not reflect this, and fails if any of the merged nodes perform computations that are "shape changing".

and fails

What fails?

After looking more closely at the code, it seems a Merge layer does not simply operate on the outputs of the layers it merges. Instead, it wraps the merged layers all the way from their inputs to their outputs. Why? The simplest interpretation of Merge's functionality is as an operation applied to the outputs of the merged layers. The "end-to-end" interpretation becomes problematic in non-trivial graph topologies.
I'm currently working with graphs that reuse the output of one or more layers across multiple 'join' merges. This is fraught with problems in Keras' current form. Layers whose inputs are provided by a 'join' merge need access to the output shapes of the merged nodes, in order to compute their own output shape. This is what my change fixed, for my strictly implicit usage of Merge. But, the change that I submitted will probably break code that constructs Merge layers explicitly, rather than using them implicitly via Graph.add_node() with multiple inputs.
The correct fix is to change Merge.output_shape for 'join' merges so that it returns the list of output shapes of the merged layers. It currently returns None, which breaks shape inference in my graphs. This change will only alter Merge's behaviour for 'join' style merges. I made this change in my branch, and it seems to work. The tests pass, but they didn't catch problems with my initial change, which probably would have broken existing code...
In general, any reuse of the output of a layer is problematic in Keras' current form. In particular, a new symbolic computation representing the output will be constructed for each time it's used as input to another layer. The property L.input which is intended to prevent redundant computation of outputs does not prevent L.get_output() from being called multiple times when multiple layers receive the output of layer L as input. This behaviour doesn't always seem to be caught and corrected by Theano's graph optimizations.
Here's example code for a layer that's easiest to implement using a 'join' merge:
https://gist.github.com/Philip-Bachman/ac3d2ea2b9f4a6a547fc
Here's a pair of scripts demonstrating problems with output re-computation:

https://gist.github.com/Philip-Bachman/59aca83c0a265b39f74f
https://gist.github.com/Philip-Bachman/e357f4b0dce0ce5788ff
-- these scripts compare output reuse via multiple calls to layer.get_output(), and output
reuse via multiple references to a single Theano symbolic variable.

Closing outdated PRs following the release of Keras 1.0.


[FEATURE] Show hidden groups only for admins - hide for all others

This is related to https://github.com/rokwire/illinois-app/issues/2423

Done in version 1.2.3


test don't support Masonry?

I mean that it don't support Masonry?

It does. Masonry it's basically extra layer for NSLayoutConstraints.
What kind of problem you have. Write something more, or upload some sample.

Can you upload a sample project as @Grubas7 suggested? This will helps us track this down

thx

Close this issue?


Custom exceptions

Since we are doing away with the Response object, we should define some custom exceptions for errors.

Custom exceptions have been defined in utils.py


Library schematic conflicts with itself when used with applyToSubtree

🐞 Bug report
Command (mark with an x)
- [ ] new
- [ ] build
- [ ] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc
- [X] schematics

Is this a regression?
Not that I'm aware of.
Description
I tried to create a schematic that would create an Anglar application and then create a library within that application, but the library schematic wouldn't run in the folder created by ng-new. That's explained in issue #16080, which is distinct from the issue I'm reporting here. It serves as background about why I needed to use applyToSubtree.
When I execute the library schematic within applyToSubtree, it seems to find the right folder and generates a list of files, but it then conflicts on the library's module file. To be clear, the library schematic creates a new file in a new folder and then fails with a merge conflict. I suspect that's because it calls schematics in turn to create a module, component and service.
Whichever way, there is no apparent way to get around this, because the merge happens within the library schematic itself and I'm not given the chance to supply a MergeStrategy.
Just because I know someone will ask: No, the file does not exist outside of the schematic and it was not created by the ng-new call.
🔬 Minimal Reproduction
You can find a minimal reproduction in this repo
Here's an abstract. I use this factory:
export function createProject(_options: any): Rule {
  return (_tree: Tree, _context: SchematicContext) => {
    return chain([
      externalSchematic('@schematics/angular', 'ng-new', optionsApp),
      applyToSubtree('NewAngularAppWithSubtree', [
        externalSchematic('@schematics/angular', 'library', optionsLib)
      ])
    ]);
  };
}

NewAngularAppWithSubtree is the path created by the ng-new call. A merge error then occurs on this file: /NewAngularAppWithSubtree/projects/new-lib/src/lib/new-lib.module.ts. Note that this file was created by the library schematic, not by ng-new.
🔥 Exception or Error

An error occured:
Error: A merge conflicted on path "/NewAngularAppWithSubtree/projects/new-lib/src/lib/new-lib.module.ts".
    at C:\Users\cobusk\AppData\Roaming\npm\node_modules\@angular-devkit\schematics-cli\node_modules\@angular-devkit\schematics\src\tree\host-tree.js:132:35
    at Array.forEach ()
    at HostTree.merge (C:\Users\cobusk\AppData\Roaming\npm\node_modules\@angular-devkit\schematics-cli\node_modules\@angular-devkit\schematics\src\tree\host-tree.js:121:23)
    at ScopedTree.merge (C:\dev\scratchpad\scaffolding-tests\AngularProjectMergeIssues\node_modules\@angular-devkit\schematics\src\tree\scoped.js:72:20)
    at MapSubscriber.project (C:\dev\scratchpad\scaffolding-tests\node_modules\@angular-devkit\schematics\src\rules\schematic.js:42:19)
    at MapSubscriber._next (C:\dev\scratchpad\scaffolding-tests\node_modules\rxjs\internal\operators\map.js:49:35)
    at MapSubscriber.Subscriber.next (C:\dev\scratchpad\scaffolding-tests\node_modules\rxjs\internal\Subscriber.js:66:18)
    at TapSubscriber._next (C:\dev\scratchpad\scaffolding-tests\node_modules\rxjs\internal\operators\tap.js:65:26)
    at TapSubscriber.Subscriber.next (C:\dev\scratchpad\scaffolding-tests\node_modules\rxjs\internal\Subscriber.js:66:18)
    at TakeLastSubscriber._complete (C:\dev\scratchpad\scaffolding-tests\node_modules\rxjs\internal\operators\takeLast.js:71:29)
🌍 Your Environment

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/

Angular CLI: 8.3.1
Node: 12.12.0
OS: win32 x64
Angular: undefined
...

Package                      Version
------------------------------------------------------
@angular-devkit/architect    0.803.1 (cli-only)
@angular-devkit/core         8.3.17
@angular-devkit/schematics   8.3.17
@schematics/angular          8.3.1 (cli-only)
@schematics/update           0.803.1 (cli-only)
rxjs                         6.4.0
typescript                   3.5.3

I have the same issue when creating a custom project analog to this tutorial and then trying to create a lazy module with route.
The first issue is that when the module schematic calls the component schematic, the root path is wrong. This can be solved by adding an execution context {scope: [projectName]} and removing the [projectName] from the path.1 But then the CLI reports:

Error: A merge conflicted on path schematics route [pathToLazy.module.ts]

1 The module, pipe and service schematics all work fine, only directive and component are writing a root path (which is missing the project name) to the tree.

Same issue here.
@pfeileon you wrote:

When I don't add a scope and instead hack getWorkspace(...) to add the project name to the path for the component schematic, only, there is no error and it works.

Would you mind to share your "hacked" getWorkspace() solution that works for you? I didn't manage to solve the issue locally by trying out lot's of things with pathes and handing over options etc.

By "hack" I mean that I executed my schematics with the value hardcoded into getWorkspace() for investigation purposes. I don't have a solution other than copying the files unfortunately.


docs examples

Maybe you can make public the docs examples, that would be of help to many.

Hi @juicycleff !
It's true, the doc are not public yet. Let me quote our CONTRIBUTING.

You can visit the docs website, however its source is not in the repo yet. Our docs are still coupled to Remote's internal Design System and integration tests. The effort to decouple it at the moment is too high.

The good news is that we are planning to build a Playground using just React for anyone to try this out more easily, and perhaps help contribute to the library more easily.

Closing in favor of https://github.com/remoteoss/json-schema-form/issues/9


StatisticsNotCollectedError when running optimize stable diffusion

When I run train_text_to_image_qat.py with fp16 in Google Colab, I get the StatisticsNotCollectedError error:
python train_text_to_image_qat.py \
    --ema_device="cpu" \
    --use_kd \
    --model_id="svjack/Stable-Diffusion-Pokemon-en" \
    --center_crop \
    --random_flip \
    --dataloader_num_workers=1 \
    --dataset_name="lambdalabs/pokemon-blip-captions" \
    --max_train_steps=8000 \
    --opt_init_steps=300 \
    --tome_ratio=0.5 \
    --quantization_mode="aggressive" \
    --mixed_precision="fp16" \
    --output_dir=sd-quantized-pokemon

Error message:
Traceback (most recent call last):
  File "/content/train_text_to_image_qat.py", line 1148, in <module>
    main()
  File "/content/train_text_to_image_qat.py", line 972, in main
    compression_controller, unet = create_compressed_model(unet, nncf_config)
  File "/usr/local/lib/python3.10/dist-packages/nncf/telemetry/decorator.py", line 71, in wrapped
    retval = fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/nncf/torch/model_creation.py", line 117, in create_compressed_model
    compressed_model = builder.apply_to(nncf_network)
  File "/usr/local/lib/python3.10/dist-packages/nncf/torch/compression_method_api.py", line 123, in apply_to
    transformation_layout = self.get_transformation_layout(model)
  File "/usr/local/lib/python3.10/dist-packages/nncf/torch/compression_method_api.py", line 142, in get_transformation_layout
    layout = self._get_transformation_layout(model)
  File "/usr/local/lib/python3.10/dist-packages/nncf/torch/quantization/algo.py", line 626, in _get_transformation_layout
    self._pt_quantizer_setup = self._get_quantizer_setup(target_model)
  File "/usr/local/lib/python3.10/dist-packages/nncf/torch/quantization/algo.py", line 721, in _get_quantizer_setup
    stats_for_range_init = self._get_statistics_for_final_range_init(
  File "/usr/local/lib/python3.10/dist-packages/nncf/torch/quantization/algo.py", line 694, in _get_statistics_for_final_range_init
    return self.get_statistics_for_quantizer_setup(target_model, quantizer_setup, range_init_params)
  File "/usr/local/lib/python3.10/dist-packages/nncf/torch/quantization/algo.py", line 688, in get_statistics_for_quantizer_setup
    retval[ip] = {rs: collector.get_statistics() for rs, collector in rs_vs_collector.items()}
  File "/usr/local/lib/python3.10/dist-packages/nncf/torch/quantization/algo.py", line 688, in <dictcomp>
    retval[ip] = {rs: collector.get_statistics() for rs, collector in rs_vs_collector.items()}
  File "/usr/local/lib/python3.10/dist-packages/nncf/common/tensor_statistics/collectors.py", line 66, in get_statistics
    raise StatisticsNotCollectedError()
nncf.common.tensor_statistics.collectors.StatisticsNotCollectedError

@AlexKoff88 do you have bandwidth to take a look ?

Fixed by NNCF: https://github.com/openvinotoolkit/nncf/issues/1997


Candidate Generation Entity Linking

I was trying to use TURL for a different Dataset to see how it performs, Is it possible to upload the code you used for Entity Linking for the candidate generation.

I will upload the preprocess code for entity linking later. In short, we use the MediaWiki lookup API for candidate generation. You can construct the URL accordingly and get the results.

okay thank you :)

Has this code been uploaded somewhere?


DPDK: Use generic CPU instructions

This commits removes AVX512 support and enables generic CPU options for
the DPDK build. The IPDK CI builds on Azure VMs which support AVX512.
However, since Virtualbox doesn't support this (see [1] and [2]), we
should skip this for p4-dpdk-target when building DPDK.
Fixes: https://github.com/ipdk-io/ipdk/issues/118
[1] https://forums.virtualbox.org/viewtopic.php?t=86047
[2] https://forums.virtualbox.org/viewtopic.php?f=7&t=87131
Signed-off-by: Kyle Mestery<EMAIL_ADDRESS>

Hmmmm, I signed the CLA and the bot doesn't appear to pick that up, even after force pushing again.

@stolsma This should fix IPDK issue 118, FYI.

Ugh. I'm sorry about the CLA. Could you nudge the ONF staff?

Ugh. I'm sorry about the CLA. Could you nudge the ONF staff?

Happy to, can you help point me at who to contact and how to do that? Thanks!

@mestery Nice! Good job!!!!