Title
Machine Learning for Stroke Lesion Segmentation
Short description and the goals for the OHBM BrainHack
We'd love for people interested in machine learning for image segmentation to try out our stroke lesion segmentation challenge! It's built with the RAMP platform and meant to be a collaborative and educational way to get into new research areas. It has a starter kit for people to get familiar with some basic steps for creating a lesion segmentation algorithm, and learn about packages such as PyBIDS along the way. We just released it and hope people can find it useful to learn and work together on a complicated neuroimaging problem!
Link to the Project
https://ramp.studio/problems/stroke_lesions
Image for the OHBM brainhack website
https://drive.google.com/file/d/1TDsEzr1R2T8O49ryNMqwPVFHjoXDOhaC/view?usp=sharing
Project lead
Sook-Lei Liew, Github: leiliew, discord: sliew
Main Hub
Americas
Other Hub covered by the leaders

[ ] Glasgow
[ ] Asia / Pacific
[ ] Europe / Middle East / Africa
[ ] Americas

Skills
Familiarity with python/juptyer notebook
Recommended tutorials for new contributors

[ ] Set up a brainhack friendly computing environment
[ ] A introduction to Bash
[ ] Python: Writing a script
[X] Python: Data analysis with Python
[ ] Python: Visualisation
[X] Machine learning basics
[X] BIDS
[X] Machine learning for neuroimaging
[ ] VCS: Using Git and Github
[ ] VCS: Data management with Datalad
[ ] Virtualization
[ ] High Performance Computing: Oracle cloud resource

Good first issues
Test out the starter kit, see if you can run it and see if you can extend it to create new lesion segmentation solutions - we welcome any feedback :)
Twitter summary
A beginner-friendly intro to machine learning for stroke lesion segmentation with a tutorial/starter kit!
Short name for the Discord chat channel (~15 chars)
stroke-lesion-segmentation
Please read and follow the OHBM Code of Conduct

[X] I agree to follow the OHBM Code of Conduct during the hackathon

Thank you for submitting the project! We have 35 projects right now, woohoo! But that means the projects pitches will have to be short. We will give you tomorrow 2 minutes to pitch your project, you can have one slide or no slides!
If you decide to use a slide, please include the link to the slide here.
And don't worry, you will still have more time to talk about your project during the BrainHack :-)

Great! I won't be there since I'm virtual, but I pre-recorded a short intro to the project you can see here (it's less than 2 min!) https://drive.google.com/file/d/1LRSOeT9AicgYNWzlwD0NJoO7TW7vUF2G/view?usp=sharing. I've also included the slide with links here in case it's helpful! I'll be on discord during PST hours if there's any questions :) Cheers!
ML_for_Stroke_Lesion_Segmentation.pdf

P.S. Brainhack is all about fun, collaborative learning & education, hence we wanted people to get to work together and try out our RAMP challenge! (And it might be a great one to do after the ML for neuroimaging tutorial?). BUT, if you are a more competitive type of person (and no judging if you are), you can also try out our MICCAI ISLES Challenge (http://www.isles-challenge.org/) or our Grand Challenge (https://atlas.grand-challenge.org/) with the same stroke data (http://fcon_1000.projects.nitrc.org/indi/retro/atlas.html)! Either way, happy hacking!


Cant installed couchdb: Error response from daemon: OCI runtime create failed:

When I tried to install it appears this error. I do have the local.ini exactly as in the documentation
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/local.ini" to rootfs at "/opt/couchdb/etc/local.ini" caused: mount through procfd: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
ERRO[0004] error waiting for container: context canceled ```

Can you show the docker code you ran?
From what I see it doesn't seem to find the file local.ini
Instead to make sure you can use the full path like this:
home/ubuntu/path/to/local.ini:/opt/couchdb/etc/local.ini

I've used the full path and now it's working. Thanks!


Fix Vulnerabilities + Fix Test Timeout

Description
Fixing Vulnerabilities
Running npm audit returns an embarrassingly high number of vulnerabilities:
found 38332 vulnerabilities (38309 low, 16 moderate, 7 high) in 1202 scanned packages
Updating Jest to 26 fixes almost all of them 😂
Of the 3 remaining vulnerabilities, all are upstream issues with browser-sync as a dependency of 11ty, being tracked here.
Jest
Normally, running npm test will all pass. But sporadically, mergePixels.test.js will fail with:
 : Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.Error:
As described in this Stack Overflow answer, I'm specifying a higher timeout for the mergePixels test.
Related issues
https://github.com/11ty/eleventy/issues/1164
Checklist

[x] I ran npm test locally and it passed without errors.

[x] I acknowledge that all my contributions will be made under the project's license.

Thank you @shawntax! Appreciate the fixes :) It was on my list to fix the dependencies but thanks for taking care of it.


Add hasRemoteParent property to SpanImpl and SpanData.

Codecov Report

Merging #283 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##             master     #283      +/-   ##
============================================
+ Coverage     93.57%   93.57%   +<.01%
  Complexity      430      430
============================================
  Files            76       76
  Lines          1650     1651       +1
  Branches         92       92
============================================
+ Hits           1544     1545       +1
  Misses           81       81
  Partials         25       25

Impacted Files
Coverage Δ
Complexity Δ

...ava/com/google/instrumentation/trace/SpanData.java
100% <ø> (ø)
2 <0> (ø)
:arrow_down:

...ava/com/google/instrumentation/trace/SpanImpl.java
76.92% <100%> (+0.45%)
10 <0> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b70ad7...93b2a6b. Read the comment docs.

What is the use case for this? For dependency link filtering?
On 8 May 2017 6:13 am, "Bogdan Drutu"<EMAIL_ADDRESS>wrote:

You can view, comment on, or merge this pull request online at:
https://github.com/google/instrumentation-java/pull/283
Commit Summary

Add hasRemoteParent property to SpanImpl and SpanData.

File Changes

M core/src/main/java/com/google/instrumentation/trace/SpanData.java
https://github.com/google/instrumentation-java/pull/283/files#diff-0
(10)
M core/src/test/java/com/google/instrumentation/trace/
SpanDataTest.java
https://github.com/google/instrumentation-java/pull/283/files#diff-1
(56)
M core_impl/src/main/java/com/google/instrumentation/trace/
SpanImpl.java
https://github.com/google/instrumentation-java/pull/283/files#diff-2
(9)
M core_impl/src/test/java/com/google/instrumentation/trace/
SpanImplTest.java
https://github.com/google/instrumentation-java/pull/283/files#diff-3
(11)

Patch Links:

https://github.com/google/instrumentation-java/pull/283.patch
https://github.com/google/instrumentation-java/pull/283.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/google/instrumentation-java/pull/283, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AAD6153SWdlfqelW5NZ3tRoG1oC_UE--ks5r3kHsgaJpZM4NTUiq
.


Question: How do we approach outbox pattern with marten?

Pattern in reference: https://microservices.io/patterns/data/transactional-outbox.html
Is the outbox pattern a pattern that is feasible or make sense using with marten? I was going to keep it simple for now and not use a connector (i.e. https://debezium.io/documentation/reference/connectors/postgresql.html) and be able to poll for newer events at least to have something working.
I initially went down the path of using the event store for the outbox pattern, but when it came to manually fetching the stream, I was hoping that I can get events after a given timestamp. What I notice is that the fetch stream gets beginning of the stream to the given timestamp.
So then, I was thinking what that would look like with the document store, but it seems like different events would not store in the same table.
Thanks in advance! Feel free to challenge and guide my direction.

https://developers.google.com/protocol-buffers/docs/proto3#oneof
Grpc service to get events


Bump configure-aws-credentials version

Hello, Peter
A new major version of the configure-aws-credentials was just released and let's switch to it from an intermediate one.
Thanks!

Thanks a lot!


Use python-jose instead of python-jose-cryptodome

python-jose is now also using pycryptodome. See https://github.com/mpdavis/python-jose/pull/83 and https://github.com/mpdavis/python-jose/blob/master/setup.py#L58.

Changed to python-jose>=2.0.0, which is the first version using pycryptodome: https://github.com/mpdavis/python-jose/releases/tag/2.0.0.


Minor bug: Color schema not recognized if != (^[A-Z].*)

This is in the dubious maybe "its a feature not a bug" field.
But json schemas for color schemes fail if the first letter is lowercase.
If its intentional, close this and move on, though it seems an odd restriction, and further may cause issues with localization down the road.
an example is excessive here but still:
  "schemes": [
    {
      "name": "Scratch2",
      "background": "#15202c",
      "foreground": "#808080",
<!--truncated -->

works, but the following does not:
  "schemes": [
    {
      "name": "scratch",
      "background": "#2d374b",
      "foreground": "#808080",
<!--trunc.  -->

I'm pretty confident that's not the case. I'm using ayu as my default color scheme and have for like a year now.
{
     "profiles":
     {
          "defaults":
          {
               "colorScheme" : "ayu",
               "experimental.retroTerminalEffect": false
          },
     },
    // Add custom color schemes to this array.
    // To learn more about color schemes, visit https://aka.ms/terminal-color-schemes
    "schemes": [
        {
          "name": "ayu",
          "black": "#000000",
          "red": "#ff3333",
          "green": "#b8cc52",
          "yellow": "#e7c547",
          "blue": "#36a3d9",
          "purple": "#f07178",
          "cyan": "#95e6cb",
          "white": "#ffffff",
          "brightBlack": "#323232",
          "brightRed": "#ff6565",
          "brightGreen": "#eafe84",
          "brightYellow": "#fff779",
          "brightBlue": "#68d5ff",
          "brightPurple": "#ffa3aa",
          "brightCyan": "#c7fffd",
          "brightWhite": "#ffffff",
          "background": "#0f1419",
          "foreground": "#e6e1cf"
        }
   ]
}

A longer example might actually be helpful here 😄


Multiplexing registration overview

When we acquired multiplexing data, we will have multiple cycles that are acquired independently. Each cycle contains multiple channels. Due to inaccuracies in the stage handling and potential movement of the biological sample between rounds, we need to register the rounds to each other.
There are different approaches for such registration. At a very high level, we have: Image-based registration vs. segmentation-based registration.
In image-based registration, we have a channel (typically our nuclear channel, DAPI) that is acquired in each cycle. We calculate the optimal transformation parameters to register all cycles to the first cycle. Then we transform the images and can do our downstream processing as if all the images were acquired at the same time.
In segmentation-based registration, we do not modify the images, but instead process the cycles separately. We run a segmentation for each cycle (=> need to have both nuclear & membrane marker in each cycle) and make measurements per cycle. We then find matching objects based on a point-cloud registration of the segmented objects. We can then combine measurements for different objects based on the matching. Theoretically, one could also apply such point-cloud matching back to the images, but we haven't done thorough tests for that yet.

What is required for Fractal implementation?
Segmentation-based registration doesn't need any special registration tasks, but will require a downstream processing task that calculates the matching of objects. That could be done via a napari workflow. Thus, for registration, this just requires that the metadata of which channel belongs to which cycle is saved, processing is then handled during data analysis.
For image-based registration, we will need to do some processing and adapting the ROIs. I made an overview here:

There are 2 ways we can do image-based registration and they depend on what processing route we choose in stitching (see here: https://github.com/fractal-analytics-platform/fractal-tasks-core/issues/11).
Registration (1): Per well. If we are able to stitch the wells, then we can perform the image-based registration on the whole well at once and find a transformation (see here for discussion of registration methods: https://github.com/fractal-analytics-platform/fractal-tasks-core/issues/36) that works on the whole well to register the cycles. The easiest case for this is processing is just finding 2D rigid shifts when aligning MIPs (that will probably be the first registration to be implemented). But one can also go with 3D affine transformations.
What do we need to handle? We will need to shrink some of the ROIs at the edge of the wells. All ROIs should contain image data for all channels, while the region outside of the ROIs may not have information for all channels.
Registration (2): Per FOV. When we process per FOV (we are worried about FOV boundaries or when we acquire in search-first mode), we also want to do registration for each FOV (or maybe down the road registration per arbitrary ROI, as FOVs are just one type of ROIs for us). Typically, those would be 3D registration, like a 3D affine registration.
What do we need to handle? We will need to shrink each ROIs to the lowest common denominator of shared area. In this case, it may also make sense to set all area outside of the ROIs to 0, because otherwise there could be conflicts (e.g. if two FOVs are shifted towards each other)

Priorities for implementation

Segmentation-based registration: Support having multiple channels without doing any image-based registration. Users can then add functionality to napari workflows to calculate matching
Registration per well for the 2D case: MIP registration, e.g. used for registering organoids during multiplexing
Registration per FOV, image-based. This is how we processed the cardiomyocyte data so far. Also how we would process search-first data like zebrafish search-first. => https://github.com/fractal-analytics-platform/fractal-tasks-core/issues/40
(Out of scope for the time being) Ways to apply segmentation-based registration back on the images

@gusqgm @nrepina @MaksHess What are your impressions of those registration workflows?

Revisiting this overview now that we're starting to tackle registration. Looking at the progress with transformations in OME-NGFF (see https://github.com/ome/ngff/issues/94) and at examples for multiplexed imaging there (e.g. see this user story): We should really keep an eye on those and I'd see such transformations as the optimal place where we'd store complex transformations.
This spec is not accepted though and also its implementation has been moving somewhat slowly. Thus, I'd have this plan for how we proceed, with the eventual goal of using the OME-NGFF transformations when they are available:
Level 1: We work with ROIs (our custom tables) and modify the ROIs per cycle. Limited to only work for translation in the ROI version, expansion to complex transformations with level 3. This allows us to have "raw" ROIs and aligned ROIs. Loading data from aligned ROIs should return aligned images (if all that needed to be transformed was a translation).
Requirement for this to work well: Every cycle loads data based on its own ROI table
Level 2: Burn in transformation for given things (i.e. burn in a complex 3D b-spline alignment that’s too expensive to calculate on-the-fly, see comment by @MaksHess above)
Level 3: We add OME-NGFF transformation, any ROI loading also applies transformations (see https://github.com/ome/ngff/issues/94). Then, we wouldn’t have to change the ROIs anymore. Likely, this will mean adopting some parts of the SpatialData spec, as they are progressing well with actually using transformations in OME-Zarrs, see https://github.com/ome/ome-zarr-py/issues/229

I think we need two independent workflows: calculate transform & apply transform

Thinking about this again, still seems like the way to go. There will be multiple ways to calculate a transform (based on an image, a label image, based on beads etc.) and then there are the 3 levels above on what apply transform could mean (i.e. where is that transform actually stored => only in Level 2 as part of the image, in the others as some specific metadata).

Another issue that fits well with this discussion is registering channels within a cycle to correct for chromatic shifts.

I think chromatic transformations may also eventually become part of the OME-NGFF spec, see user story by Marvin here: https://github.com/ome/ngff/issues/84#issuecomment-1644245278


Issue with k8s.io/docs/reference/kubectl/overview/

Hi All,
In the subject of this ticket, the link is mentioned, an example mentioned is incorrect.
We are creating a shell script "kubectl-hello" and placing it in the /use/local/bin then how come the output is "hello world" with "kubectl hello" command shouldn't it be "kubectl-hello" ?
The Examples: Creating and using plugins
create a simple plugin in any language and name the resulting executable file so that it begins with the prefix "kubectl-"
cat ./kubectl-hello
#!/bin/sh
this plugin prints the words "hello world"
echo "hello world"
With a plugin written, let's make it executable:
chmod a+x ./kubectl-hello
and move it to a location in our PATH
sudo mv ./kubectl-hello /usr/local/bin
sudo chown root:root /usr/local/bin
You have now created and "installed" a kubectl plugin.
You can begin using this plugin by invoking it from kubectl as if it were a regular command
kubectl hello
The plugin mentioned should be kubectl-hello instead of kubectl hello"
Similar issue in the document in other examples:
kubectl whoami -> kubectl-whoami

/kind support
@praveensh0 the text of the page looks correct to me. Did you try running kubectl hello?

/sig cli

@sftim When I ran it, it's working but I am wondering how it's working. How /kubectl-hello is translated into /kubectl hello?


[FEATURE] how can I quickly test a huggingface transformer model

Is your feature request related to a problem? Please describe.
I'm frustrated when I try to apply this project on a huggingface transformer model, i.e. a bert model.
Describe the solution you'd like
I can not find a clear doc to direct me to port a huggingface model into the colossal-ai. Apparently, all of the examples are related to visual models, but most of the large model applications are on NLP scenarios.
Describe alternatives you've considered
Provide an example to tell me how to simply move my training process to colossal-ai.

Hi, feifeibear,
Thank you for your comments.
We are working NLP scenarios, such as BERT, GPT2, GPT3, and these examples and detailed documents will be updated very soon.
Sincerely,
Bin


Update dependencies

This updates:

The Android Gradle plugin (3.1.0)
Gradle itself (4.7)
Kotlin (1.2.41)

Also, the dependency on the deprecated kotlin-stdlib-jre7 has been
exchanged with kotlin-stdlib-jdk7 and the explicit setting of the
build tools version has been removed, since that gets managed by the
Android Gradle plugin automatically now.

Any reason you went with 3.1.0 over 3.1.2 on the Gradle plugin!

No, updated!


CIP-0079? | Enforceable royalties

Overview
This CIP introduces a technical design to enforce royalties on-chain. This CIP is still developing and we hope to receive much constructive feedback.
Technical Requirements
The initial draft of the CIP was developed with the following technical requirements (ordered for reference, not by priority):

Interaction with a contract occurs (asset is unlocked from a script UTxO) must trigger royalties payments. Importantly, this must operate independently and outside the control of any specific marketplace. Solutions requiring tacit social agreement (e.g. "every marketplace should use X script") are disallowed - any nonconforming party can circumvent such an agreement.

It must not redefine the ownership of native assets: Cardano has already defined ownership of native assets - creating an auxiliary definition would require at least

extensive changes in wallets and dApps
operating outside the technical guarantees provided by Cardano’s proven, academic models (insecure).

It should be backwards-compatible: Current NFT collections should be able to upgrade their collections to adopt this standard and enforce royalties.

Benefits
Adopting this CIP would guarantee that creators would receive royalties on any transaction involving their created assets - not only those creators for whom particular marketplaces process their royalties.
This also allows creators control over their royalties, instead of delegating that control to marketplaces.
Draft’s Shortcomings
The initial draft of the CIP successfully meets these three technical requirements. However, we have encountered the following shortcomings (ordered arbitrarily):

Creators must maintain a whitelists which burdens newer marketplaces and the contract update process.
Current assets require a migration in order to receive the aforementioned benefits of this CIP
This draft proposal requires actors to watch the chain, mint Proof Tokens, and applying penalties. This implies and requires significant adoption by a distributed set of parties with competing economic interests.

We encourage and appreciate any feedback.

One of the drawbacks of this CIP (as stated) is that creators must maintain a whitelist of scripts that they "allow" their NFTs to reside in. With the Cardano ecosystem growing so rapidly it is impossible to keep track of all the script addresses that are acceptable. I believe this would stifle development of new contracts where they struggle to get whitelisted for collections.
Another problem I see (if I am understanding correctly) is that this system seems to penalise the user and not the marketplace that is selling tokens with 0% royalties. If I knew nothing about cardano NFTs and I stumbled across marketplace with an NFT I liked, I could buy it without realising the marketplace is 0% royalties and then have my NFT rendered worthless by someone minting the proof token because it was at a script address that wasn't whitelisted.
I think starting this discussion is the right way to go as there needs to be an enforceable royalties method, however this seems like only a small part of the solution and we have a long way to go from here.

Hey Martin,
This is definitely a problem worth solving.
One question I had was if there would ever be a scenario where a user purchased an NFT from a "malicious" marketplace and sold it to another user on a compliant marketplace such as jpg.store. Would the new owner of this asset be in a position where they could be left with a "useless" asset despite never acting maliciously?

One of the drawbacks of this CIP (as stated) is that creators must maintain a whitelist of scripts that they "allow" their NFTs to reside in. With the Cardano ecosystem growing so rapidly it is impossible to keep track of all the script addresses that are acceptable. I believe this would stifle development of new contracts where they struggle to get whitelisted for collections.
Another problem I see (if I am understanding correctly) is that this system seems to penalise the user and not the marketplace that is selling tokens with 0% royalties. If I knew nothing about cardano NFTs and I stumbled across marketplace with an NFT I liked, I could buy it without realising the marketplace is 0% royalties and then have my NFT rendered worthless by someone minting the proof token because it was at a script address that wasn't whitelisted.
I think starting this discussion is the right way to go as there needs to be an enforceable royalties method, however this seems like only a small part of the solution and we have a long way to go from here.

Thank you for the feedback.
Please read my comment below regarding your second point.
Regarding the first one, ideally the marketplace should be punished. However, we did not find a good way of accomplishing this.

Definitely appreciate the approach this is taking and the need to solve for skirting royalties.  Tackling the problem is important.  Some additions I think this needs:

Tokenomics or other incentives to the "watchers" and a discussion of specifically who they should be
A pseudocode (or real code, thinking Helios for lightweight nature of referencing) implementation of  the different endpoints
A mechanism for "restitution"... I don't think NFTs should be permanently rugged there should be a way to restore the trust of the user in case it was an accidental royalty-skimp (e.g., unwitting buyers pay 5% plus some fee to unrug the NFT)

What happens if you trade NFT for NFT, or a bundle of NFTs for other NFTs? Or go with completely crazy selling mechanisms that cannot be captured inside the royalty config. It would limit the possibilities of your asset.

Thank you for the feedback.
Please read my comment below regarding your second point.
Regarding the first one, ideally the marketplace should be punished. However, we did not find a good way of accomplishing this.

I still think this approach will stifle innovation on Cardano through the whitelisting system. It should ultimately be up to users to decide what contracts they use and there should not be any consequences for doing so. What I can see happening with this whitelist system that could help contract developers is that a central repository could be created to keep track of all the various scripts that are "trustworthy" however this in itself would require moderation and would favour open source contracts over commercial closed-source ones (whilst we should strive for open source there are some cases where closed source is necessary).
I also don't think we should push for a system that punishes users just because we can't think of a better alternative.

I don't see why such a CIP would be necessary. If payment of royalties needs be enforced, the one and only place for this should be the validator of the marketplace smart contract.
No need for any change on protocol level which would destroy one of the advantages of native assets on Cardano and make the ecosystem more centralized.
Speaking from a creator point of view, I don't want to keep any whitelists or blacklists, for both practical and ideological reasons.

I don't see why such a CIP would be necessary. If payment of royalties needs be enforced, the one and only place for this should be the validator of the marketplace smart contract. No need for any change on protocol level which would destroy one of the advantages of native assets on Cardano and make the ecosystem more centralized.
Speaking from a creator point of view, I don't want to keep any whitelists or blacklists, for both practical and ideological reasons.

I see how creators can have their reasons not to opt into the system. Royalties are opt-in, they are a feature for creators, established in a contractual way when the mint occurs. This CIP aims to provide a way to enforce such a pact in an on-chain manner.

Although the implementation can be improved (e.g some alternatives to not maintain a whitelist), I believe that there is consensus in the community and evidence from other ecosystems that royalties are a necessary feature.

Although the implementation can be improved (e.g some alternatives to not maintain a whitelist), I believe that there is consensus in the community and evidence from other ecosystems that royalties are a necessary feature.

I don't think anyone is arguing the need for royalties but this CIP seems to have too many flaws to be implementable. Surely the best course of action is to have marketplaces enforce royalties through their smart contracts rather than through continuous monitoring of the blockchain? It shouldn't be too hard to add the royalty address and percentage into the datum of the smart contract listings and therefore enforce that the correct royalties are paid. This would also solve the issue of external marketplaces that use the same contract as it would force them to pay royalties also without the need for the transaction to be signed by the original marketplace.
Whilst this doesn't stop a 0% royalty marketplace from existing, it would at least stop bad actors bypassing the royalties on trustworthy marketplace contracts until a true on-chain solution is possible.

Although the implementation can be improved (e.g some alternatives to not maintain a whitelist), I believe that there is consensus in the community and evidence from other ecosystems that royalties are a necessary feature.

There is probably no doubt that royalties are a useful feature, I'm just concerned that the protocol is not the right place to implement them. When speaking of evidence from other ecosystems, it should also has to be mentioned that similar features (in this case implemented in the token SCs) have also been abused a lot e.g. as "sales tax" for fungible tokens as integral part of many ponzinomics.
It is also not always clear if a certain transaction is a purchase, enforcing royalties on non-purchase transactions can kill new innovative business ideas.

Whilst this doesn't stop a 0% royalty marketplace from existing, it would at least stop bad actors bypassing the royalties on trustworthy marketplace contracts until a true on-chain solution is possible.

This is exactly what we are looking for. What could be the "true on-chain solution"?
Spending policies? Freezing policies? If so, then we would like this CIP to be the place to discuss it.

So the watcher will reference the UTxO and immediately render the NFT useless, before anyone is able to buy.

Since the actual user token is not burned, I don't think it will stop anyone from buying/selling these assets. Maybe because metadata are cached for a few hours so people can't even see that it's invalid. In general I think explaining someone this concept of reference NFT and user NFT is way too difficult. As long as they hold something in their wallet they think they own it.

This is exactly what we are looking for. What could be the "true on-chain solution"?
Spending policies? Freezing policies? If so, then we would like this CIP to be the place to discuss it.

To be honest I'm not sure there is a one solution fits all approach to enforcing royalties on Cardano. I think it is up to the marketplaces to enforce the royalties but in a decentralised way, i.e by enforcing it in the smart contract validator. If a marketplace chooses not to do this then they should be quickly abandoned by the community.

If a marketplace chooses not to do this then they should be quickly abandoned by the community.

That's how I see it too. The only way to enforce it is through social pressure and have the rules embedded in each marketplace contracts.

It shouldn't be too hard to add the royalty address and percentage into the datum of the smart contract listings and therefore enforce that the correct royalties are paid.

This is already what we're doing on JPG Store. It's not enough to protect creators royalties. There will be competition that pops up which doesn't honor the existing CIPs.

Whilst this doesn't stop a 0% royalty marketplace from existing, it would at least stop bad actors bypassing the royalties on trustworthy marketplace contracts until a true on-chain solution is possible.

This is exactly what we are looking for. What could be the "true on-chain solution"?
Spending policies? Freezing policies? If so, then we would like this CIP to be the place to discuss it.

This is the purpose of this CIP - a truly decentralized approach to enforcing royalties. I think we should steer this discussion back towards that goal. This isn't the place to ask why we can't just "rely on marketplaces to do the right thing", if you're curious about that I'd encourage you to read the threads below and discuss in the NFT guild Discord: discord.gg/9ZQXs8Xu2q
https://twitter.com/PuffYachty/status/1584277265311240192?s=20&t=7HYUHIAhkvyqtue0lUUk_A
https://twitter.com/kemos4be/status/1579314353895264256?s=46&t=ej4nMqcR5kLq1HV_dqSzpQ
https://twitter.com/MagicEden/status/1581101016686399491?s=20&t=YT9Q_aBwyLSxymjoj8HFCg

We've already seen that social pressure isn't enough on other blockchains. What makes you think it'd be any different on Cardano?

Because social pressure on Cardano is (or at least can be) a wildly different entity. Along those lines, and to invert your question, what makes you assume the experiences and trajectory of Magic Eden and Open Sea will be the same for Cardano? This entire problem derives from a choice by one marketplace on Solana to change their approach to royalties, and then go off on a crusade to come back with an on-chain solution for their ecosystem.
This history of NFTs on Cardano has been much different, with CIP-27 offering an on-chain solution to royalties before the implementation of smart contracts. This can be applied in single-asset sales, as well as bundle sales with different policy IDs and royalty percentages for each individual asset. This is, and has been, the configuration on some marketplaces in Cardano since last year. This CIP is not the be-all-end-all of royalties on Cardano, but could be, nevertheless, a meaningful starting place for future development. There's lots of fun things we still need to take care of. For example, solutions are needed for sales conditions where a royalty is stipulated, but payments are done via native assets. But problems like those are what make Cardano, Cardano. Ultimately, I feel we should not assume the past performance of other ecosystems is a guarantee of future results in our own.

We've already seen that social pressure isn't enough on other blockchains. What makes you think it'd be any different on Cardano?

Perhaps it isn't but this isn't the way to go about enforcing the royalties. There are major flaws in this idea that would be an incredible hinderance to innovation on the chain. How do you decide who is a good and a bad actor? How do you then keep track of every smart contract address as more and more marketplaces and dapps become available and similarly how will you store those addresses when the list gets too big to store in transaction datums? How do you ensure the marketplace is punished and not the user? How do you define the rules of when royalties should be paid?
There are so many questions that need to be sorted IMO before a CIP like this can even be considered. Lets take a step back and approach this in the way Cardano is renowned for - research backed and right first time. There is no need to rush into this when the problem hasn't even affected us yet.

This is already what we're doing on JPG Store. It's not enough to protect creators royalties. There will be competition that pops up which doesn't honor the existing CIPs.

If you are already doing this then why is the jpg signing key required?

I'm not sure what kind of research is needed here, but I have a dog in this fight as a creator: I want to make sure no one circumvents my royalties.
I view this proposal as a starting point and people should propose alternatives.  At the very least there are three use cases that need to be covered:

Wallet-to-wallet transfers (no royalty required)
Direct-trade escrows (e.g., trading tent) - open to whether or not royalties are required since the currency is a barter/native asset
Marketplaces - royalties must be enforced and bad actors must be punished or good actors must be rewarded

Open questions:

Do we need to protect new buyers who will not know about these standards and might be susceptible to ending up on a zero-royalties site accidentally?
Should there be a mechanism for restitution once a zero-royalty marketplace has come in the picture?
Should this be enforced at the token level or the protocol level (e.g., through "watcher nodes")?

Guys, what I do know is that allowing marketplaces to enforce this through the smart contracts is not enough and the day will come when some bad actor circumvents this.  When that happens, I and other creators will stop developing for Cardano.  Reattaching this photo on what a detriment this would have

Let's not argue over "if" this will happen.  If there are more appropriate forums for this or other considerations that need to be taken into account let's do that.  I'm not willing to wait until someone starts ripping me off to try to put a bandage on it.

I'm not sure what kind of research is needed here, but I have a dog in this fight as a creator: I want to make sure no one circumvents my royalties.

I think we all agree that we don't want royalties circumvented, the issue is implementing this correctly first time.

I view this proposal as a starting point and people should propose alternatives. At the very least there are three use cases that need to be covered:

Wallet-to-wallet transfers (no royalty required)
Direct-trade escrows (e.g., trading tent) - open to whether or not royalties are required since the currency is a barter/native asset

I think royalties should not apply here if it is a straight trade of native assets since there is no "value" being exchanged.

Open questions:

Do we need to protect new buyers who will not know about these standards and might be susceptible to ending up on a zero-royalties site accidentally?

IMO absolutely. If mainstream adoption is ever going to happen, new users cannot be caught out by a nuance they don't even know about. This is why I am so strongly against punishing the user by making the NFT worthless.

Should there be a mechanism for restitution once a zero-royalty marketplace has come in the picture?
Should this be enforced at the token level or the protocol level (e.g., through "watcher nodes")?

Both good questions that require a full discussion.

Guys, what I do know is that allowing marketplaces to enforce this through the smart contracts is not enough and the day will come when some bad actor circumvents this. When that happens, I and other creators will stop developing for Cardano. Reattaching this photo on what a detriment this would have

You are right it is not enough for the long term but in the short term it is. We are at such an early stage that user sentiment towards a marketplace is everything. We haven't reached the point of having a Cardano equivalent of OpenSea and so it is very easy for users to switch marketplaces to the most reputable. There will almost always be a way around royalties no matter what, we just need to find they way that means 99% of people abide them. As for stopping developing on Cardano, piracy is rampant in the entire tech sector despite it supposedly having strong consequences but that doesn't stop creators from creating because 99% of people follow the 'rules'.

Let's not argue over "if" this will happen. If there are more appropriate forums for this or other considerations that need to be taken into account let's do that. I'm not willing to wait until someone starts ripping me off to try to put a bandage on it.

Likewise I'm sure you don't want to be in a situation where people are put off buying your NFTs because of the possibility of it being made worthless all because they sent it to a contract that wasn't whitelisted. This is why the correct action needs to be taken rather than rushing into the first solution that comes to mind.

I'm not sure what kind of research is needed here, but I have a dog in this fight as a creator: I want to make sure no one circumvents my royalties.

I think we all agree that we don't want royalties circumvented, the issue is implementing this correctly first time.

I view this proposal as a starting point and people should propose alternatives. At the very least there are three use cases that need to be covered:

Wallet-to-wallet transfers (no royalty required)
Direct-trade escrows (e.g., trading tent) - open to whether or not royalties are required since the currency is a barter/native asset

I think royalties should not apply here if it is a straight trade of native assets since there is no "value" being exchanged.

Open questions:

Do we need to protect new buyers who will not know about these standards and might be susceptible to ending up on a zero-royalties site accidentally?

IMO absolutely. If mainstream adoption is ever going to happen, new users cannot be caught out by a nuance they don't even know about. This is why I am so strongly against punishing the user by making the NFT worthless.

Should there be a mechanism for restitution once a zero-royalty marketplace has come in the picture?
Should this be enforced at the token level or the protocol level (e.g., through "watcher nodes")?

Both good questions that require a full discussion.

Guys, what I do know is that allowing marketplaces to enforce this through the smart contracts is not enough and the day will come when some bad actor circumvents this. When that happens, I and other creators will stop developing for Cardano. Reattaching this photo on what a detriment this would have

You are right it is not enough for the long term but in the short term it is. We are at such an early stage that user sentiment towards a marketplace is everything. We haven't reached the point of having a Cardano equivalent of OpenSea and so it is very easy for users to switch marketplaces to the most reputable. There will almost always be a way around royalties no matter what, we just need to find they way that means 99% of people abide them. As for stopping developing on Cardano, piracy is rampant in the entire tech sector despite it supposedly having strong consequences but that doesn't stop creators from creating because 99% of people follow the 'rules'.

To be clear when I say "developing" I mean the entire community.  I am not sure how tech applies here, but if artists weren't paid by Spotify they would pull their music and go to court to enforce licensing.  This is a decentralized ecosystem, but we have to recognize that without royalties communities and artists will disappear quickly.

Let's not argue over "if" this will happen. If there are more appropriate forums for this or other considerations that need to be taken into account let's do that. I'm not willing to wait until someone starts ripping me off to try to put a bandage on it.

Likewise I'm sure you don't want to be in a situation where people are put off buying your NFTs because of the possibility of it being made worthless all because they sent it to a contract that wasn't whitelisted. This is why the correct action needs to be taken rather than rushing into the first solution that comes to mind.

Agree I don't want to rush it, I just don't want to waste time questioning the premise either.

@MartinSchere it's great to see that this is generating so much interest, but that also creates more a problem with assigning a CIP number that's not guaranteed yet (since people often discuss on social networking & create unmaintainable references).
Can you remove the number 79 from the document title (and header, when you create one), and rename the folder of your README from CIP-0079 to something non-numerical like CIP-enforceable royalties? 🙏

@MartinSchere I like the idea of enforceable royalties, but not the punishment part.
I think that this standard need to define something that includes:

Mints with a multi-sig approach instead of smart contracts
NFT Marketplaces with a multi-sig approach instead of smart contracts
Projects with their own Marketplaces

How do you think you are going to track addresses? marketplace addresses could change continuously and make it impossible to track.
Why not define a holders-centered standard, where users that buy/sell NFTs in Good Actors Marketplaces receive an extra token/future royalties/whatever.
Royalties of future sales look like a nice idea to explore.
Change from an enforcement/penalty to an enforcement/reward approach.

I think that we should not abandon the discussion.
This CIP could be rewritten with a completely different approach (e.g "freezing policies"?), and we are happy to see creative suggestions.
What is not acceptable is to do nothing about the issue, waiting for a zero-royalty marketplace to enter the space and destroy the ecosystem, as evidenced in other spaces.

I think royalties should not apply here if it is a straight trade of native assets since there is no "value" being exchanged.

Yes, NFT >< NFT trades should not be accounted for.
Whenever ₳ is included, sure. That's how we've approached it in Tent.
What about if I buy an NFT with $yummi? $yummi royalties to the Project?

Yes, NFT >< NFT trades should not be accounted for.

Lock ADA in a script utxo and mint an NFT X that can unlock that UTxO -> Trade NFT X for any other NFT -> Unlock ADA with NFT X -> Paid 0 royalties.

Yes, NFT >< NFT trades should not be accounted for.

Lock ADA in a script utxo and mint an NFT X that can unlock that UTxO -> Trade NFT X for any other NFT -> Unlock ADA with NFT X -> Paid 0 royalties.

I think this is the major issue. There will always be some way around the royalties no matter what.

What if you had a seed in the contract, and we autenticate with a generated number using the same seed with this contract to change the datum with the owner of the token, and in this process we pass the ADA in accordance to the royalties? The problem is how to know the NFT price at the transference moment, for this one could use an trusted oracle

What if you had a seed in the contract, and we autenticate with a generated number using the same seed with this contract to change the datum with the new owner of the token, and in this process we pass the ADA in accordance to the royalties? The problem is how to know the NFT price at the tx moment, for this one could use a trusted oracle
We could have a listing of trusted NFTs (the ones who are paying royalties), and the market can regulate itself this way
The idea of the seed is to have this "counter", a "number generator" that sync's with NFT transactions. One must pay all tx royalties to "unlist" some NFT.
I believe this also aligns with #357

If I understand it correctly, this seems like an overly complicated way of simply enforcing royalty payments in a marketplace contract and still relies on trusting the marketplaces to do so.

@Tenancio yes, actually it really does not matter if it's a counter, just an oracle, many oracles, in the end is up to the marketplace to go with the contract.

I have a potential proposal.
Rather than maintaining a whitelist of allowed smart contracts, we have an on chain governance contract that maintains a blacklist of untrustworthy sites via storing tokens of a specific policy/asset id in a locked smart contract address. Wallet developers should standardise a check on these tokens before connecting to a site and not allow a user to connect their wallet if the site is blacklisted.
Advantages:

User is not punished by using 0% royalty marketplaces without knowing.
Almost completely decentralised (requires wallets to conform to the standard)
Doesn't require creators to maintain a whitelist of allowed contract addresses
Increases cost of bypassing the restriction since buying new domains costs money.

Disadvantages:

Requires all wallets to implement the standard (easily doable though)
Will not affect Cardano-CLI or other Plutus scripts from interacting with contracts from blacklisted sites
Governance contract would need to be developed, audited and deployed which will take time.

I'm not a wallet developer so I can't say how easy this is to implement but I can't imagine it would be a huge job (feel free to correct me)
There are of course ways around this but it seems like it would certainly discourage most users from using the blacklisted sites.

@Tenancio yes, actually it really does not matter if it's a counter, just an oracle, many oracles, in the end is up to the marketplace to go with the contract.
I like the idea of having a listing so buyers can check if an NFT is paying royalties before buying, but I don't like the idea of forcing anyone to pay royalties, this sounds like an invasion of the NFT owner's property rights.

I'm still not sure I understand the need for an oracle. But I do kind of agree with the fact we can't FORCE anyone to pay the royalties. They are opt-in after all. But I do think we should make it more difficult to bypass paying royalties.

I am late in the convo but maybe I am lost,

Why can't each smartcontract marketplace regulate the royalties in the validator itself?
In case a marketplace is doing 0 royalties listing blockchain would show in the transactions

instead of watchers I would prefer a script that mint "rewards" for good actors, but what are the benefits for this? Maybe all marketplaces could create collections purchasable with  these tokens?
Instead of punish the bad actors, incentivize the good ones
I think CIP68 helps a lot giving power to creators to define or update the royalties address of their collection
But what in case of rugpulls? should they still be in control of their royalties? Decentralization works in both ways

Why can't each smartcontract marketplace regulate the royalties in the validator itself?

It can and imo should.

In case a marketplace is doing 0 royalties listing blockchain would show in the transactions

Indeed it would yes but that doesn't stop them doing it.

instead of watchers I would prefer a script that mint "rewards" for good actors, but what are the benefits for this? Maybe all marketplaces could create collections purchasable with these tokens?

How would you define a good actor in a decentralised way? And like you say, what would the benefit be?

But what in case of rugpulls? should they still be in control of their royalties? Decentralization works in both ways

Interesting point, I think in the context of the on-chain governance blacklist I proposed above you could have another type of list for "blacklisted" NFT policies. Tokens can still be sold normally but marketplaces can check if the project is blacklisted and thereby allow ignoring the royalties.

Rather than maintaining a whitelist of allowed smart contracts, we have an on chain governance contract that maintains a blacklist of untrustworthy sites via storing tokens of a specific policy/asset id in a locked smart contract address. Wallet developers should standardise a check on these tokens before connecting to a site and not allow a user to connect their wallet if the site is blacklisted.

This is an interesting idea, but has its own set of pros/cons. The main con I see is that it's another form of centralization and requires wallets to conform to the standard. Even if we get all wallets on the same page, there's also nothing stopping someone from creating a new wallet which doesn't conform.

Rather than maintaining a whitelist of allowed smart contracts, we have an on chain governance contract that maintains a blacklist of untrustworthy sites via storing tokens of a specific policy/asset id in a locked smart contract address. Wallet developers should standardise a check on these tokens before connecting to a site and not allow a user to connect their wallet if the site is blacklisted.

This is an interesting idea, but has its own set of pros/cons. The main con I see is that it's another form of centralization and requires wallets to conform to the standard. Even if we get all wallets on the same page, there's also nothing stopping someone from creating a new wallet which doesn't conform.

Indeed, I could use cardano-cli and that's it, bypass the wallet

For reference, this CIP follows in the footsteps of the same approach that OpenSea is taking: https://github.com/ProjectOpenSea/operator-filter-registry
Rather than a whitelist approach, they went with a blacklist. This might be easier to maintain overtime if we imagine that more marketplaces will support royalties than those that don't. This eliminates a barrier to entry for new entrants.
This was clearly effective, as it led to the largest 0% royalty marketplace on ETH switching course and supporting royalties today.

This is an interesting idea, but has its own set of pros/cons. The main con I see is that it's another form of centralization and requires wallets to conform to the standard. Even if we get all wallets on the same page, there's also nothing stopping someone from creating a new wallet which doesn't conform.

I agree, the idea isn't perfect but it could be along the right lines. I think centralisation here isn't so much of an issue seeing as it would need be an open-source, audited contract that anyone could interact with to start a backlist vote etc. meaning it should be far less centralised than I fear this proposal would end up being. But you are right, it is still possible to bypass it as I eluded to which could be conceived as an issue.

Indeed, I could use cardano-cli and that's it, bypass the wallet

Yep exactly, this is the main downfall of the approach.

Rather than a whitelist approach, they went with a blacklist. This might be easier to maintain overtime if we imagine that more marketplaces will support royalties than those that don't. This eliminates a barrier to entry for new entrants.

This is exactly what I was trying to propose, a whitelist solution would be far harder to maintain than a blacklist. The main issue with porting this over to Cardano is that native assets are not bound by smart contracts so they can be sent a blacklisted address either way if there is no intervention from a front end or wallet and I am still against punishing NFT owners instead of the marketplaces other than cases where it is absolutely certain the owner is deliberately attempting to avoid royalties to make a few extra Ada.

From descentralization standpoint, having a governance system is really better than having whiltelisting in creators hands? Who is gonna vote for blacklisting, based on what criteria, a governance system in hands of a few players could be potentially worse than giving decision power to many policy creators. Having the majority of the voting tokens basically means that one player could subvert the governance objectives with anti-competitive behavior

The reason I said that was because with a creator-defined whitelist approach many creators will want a go-to repository with a majority of the trusted marketplaces and contracts. Who controls that and what stops them from altering it as they wish? With an on-chain governance at least everyone will be able to have a say and so it wouldn't be in the hands of a few players. Obviously the voting mechanism would need to be carefully designed to mitigate the risk of a single entity owning a majority. Something like quadratic voting could work in this case.
Perhaps the two could be combined, have a global on-chain blacklist but allow policy creators to override this in their royalty tokens or add extra addresses to the blacklist. This would allow more freedom to creators to decide what their tokens can/can't be used for.

Oh maybe better: Allow the creator to override the governance entity of their tokens, something like "my minting policy is following whitelisting/backlisting of this governance entity".

I like this a lot

I like this a lot. Still need to figure out what blacklisting/whitelisting means though

I was thinking in leaving the choice open, (blacklist or whitelist), and let the governance entity to define what each word means and addition/removal criteria. The DAO would suggest the application/specification for wallets implementations, this way the creator will use it's interpretation of what is fair and choose it accordingly.
Of course the most logical path is to use blacklist to remove bad actors and whitelist as endorsement.

I like this a lot. Still need to figure out what blacklisting/whitelisting means though

I was thinking in leaving the choice open, (blacklist or whitelist), and let the governance entity define what each word means and addition / removal criteria. The entity would suggest the application / specification for wallet implementations, this way the creator will use his interpretation of what is fair and choose it accordingly.
Of course the most logical path is to use blacklist to remove bad actors and whitelist as endorsement.

My meaning was, what will a whitelist/blacklist do? Punishing users should still be off the table but how will the whitelist/blacklist be enforced?

I have a lot of problems with what is being discussed here. this smells just like the fight we have had in the USA for the last decade with manufacturers over right to repair. If someone owns something it is their right to do with it what they please, without conditions being forced upon it.
Same thing here. if someone owns the NFT it is their choice to list it where they want or sell it how they want. If they choose to sell it through a platform that supports royalties, then Awesome! and if not, that is their choice. Crypto, and specifically Cardano is all about a free market ecosystem. If we don't want a marketplace that doesn't support royalties to come in and overtake the other marketplaces then the current Marketplaces need to provide a product that is so far superior to the others that the users choose it.
JPG store literally gained their market share by supporting Royalties. and in doing so, the NFT creators pushed their communities to it and away from CNFT.io. This CIP is wrong because, JPG store trying to do this would have been like cnft.io trying to create a CIP to ban royalties back in Aug. of 21. It isnt how the free market works.
Now, there is a different story for a new marketplace aggregating other marketplaces listings and then not paying royalties. But, I believe we already have a way to fix that. Can't the Marketplace use the Royalty system laid out in CIP-27 to build the SC in a way that forces the royalties to be paid automatically on each transaction when the buyer interacts with the SC? basically as part of the SC it finds the 777 tag on the policy ID and then uses the address and percentage in that to automatically send the royalties? that way even if an aggregator uses the listings from jpg.store, the SC automatically forces it to pay royalties?
If that can be done, it makes way more sense, because then if the owner posts on JPG.store, they know the royalties will get paid no matter what, and it still doesnt take away their free choice whether or not to post it on a royalty marketplace.

Also, We could reach out to the wallets to implement a way to say to the buyer "this NFT creator requests a X% royalty, but the transaction is excluding that. Do you wish to include the royalty?" then the person can choose to pay it..

how will the whitelist/blacklist be enforced?

It could be enforced in the protocol, but I believe it should not be enforced, only recommended.

Now, there is a different story for a new marketplace aggregating other marketplaces listings and then not paying royalties. But, I believe we already have a way to fix that. Can't the Marketplace use the Royalty system laid out in CIP-27 to build the SC in a way that forces the royalties to be paid automatically on each transaction when the buyer interacts with the SC? basically as part of the SC it finds the 777 tag on the policy ID and then uses the address and percentage in that to automatically send the royalties? that way even if an aggregator uses the listings from jpg.store, the SC automatically forces it to pay royalties?

It's not possible to read metadata like the 777 tag directly, but there are for sure ways to implement it into a smart contract by the marketplace doing one transaction setting the datum of a uniquely defined UTXO for each collection.
Not sure if this is actually the case, did not look at their contract and seen contradicting statements, but: if it's possible to build a valid buy transaction for existing listings, circumventing royalties, this would simply be a bug/shortcoming of their code.

It sounds like there's a lot of interest in defining the problem, and not much agreement about what the solution should look like. Perhaps it would be good for the participants here to collaborate on writing a CPS? Then we can try and get people on the same page on what the different requirements that people care about are, what the use cases are, etc.

It sounds like there's a lot of interest in defining the problem, and not much agreement about what the solution should look like. Perhaps it would be good for the participants here to collaborate on writing a CPS? Then we can try and get people on the same page on what the different requirements that people care about are, what the use cases are, etc.

I agree. Many of us feel that there is no problem necessarily however I am happy to take part in a discussion should the other part of the community feel a solution to enforceable royalties is required and I think a more specific scope should be defined on what we are trying to solve. i.e should all royalties be enforced or should there be optional royalties? What counts as a transaction where royalties should be applied? and more.
I was particularly intrigued to read #336 and the solution proposed (and rejected) in there and I think this could prove to be a valuable starting discussion.

As per today's editor discussion, we'll postpone assigning a number to this proposal because it seems like it would deserve being framed as a CPS first (given the numerous comments and areas of debates regarding the definition of the requirements themselves).
Waiting for @MartinSchere to comment on this.

also @MartinSchere - thanks for removing the inappropriate number 79 (which now belongs to what will be a very popular CIP) but you have left a redundant copy of your README there in your branch: can you delete the first of these please?

'CIP-0079/README.md'
'CIP-enforceable-royalties/README.md'

Hello!
Apologies for the delayed response.
I agree, we should discuss this as a CPS.
@rphair I removed the README from the CIP-0079 folder.

Should we close this in the meantime?

@MartinSchere I don't think it needs to be closed, since leaving it open would make it easier for discussion points to insert into the new CPS so they can be followed by existing reviewers.  It's only been a Draft so far so you can just mark it as Ready for review when it's re-posted in your branch as a CPS with the CIP document removed.


Magic Burst

Separate Magic Burst spells vs. normal spells. Should also be easy, just need to test.

Done!


fix: filter out empty env vars

fixes: #1101

@edewit can we close?

@ia3andy no we still need to filter out the empty keys, I've rebased and fixed it to use the string[][] version of the component


Added chainerx support on initializers

I blowed up the previous PR #7531 .
Adds support for chainerx arrays on chainer initializers

Jenkins, test this please

Jenkins CI test (for commit 16c103c077fac0dcbb433540454860813eaa939c, target branch master) succeeded!


NOAUTH Authentication required

Hi,
I have updated my gorse version to latest v0.2.5, along with the latest config.toml . Now we are getting below error after the application runs for few time:-
**API Request : curl -k http://localhost:8084/api/recommend/1084
{**
  "level": "error",
  "ts":<PHONE_NUMBER>.2515652,
  "caller": "master/tasks.go:237",
  "msg": "failed to cache neighbors of items",
  "error": "NOAUTH Authentication required.",
  "errorVerbose": "NOAUTH Authentication required.\n/go/gorse/storage/cache/redis.go:45: \n/go/gorse/master/tasks.go:232: \n/go/gorse/base/parallel.go:78: ",
  "stacktrace": "github.com/zhenghaoz/gorse/master.(*Master).runFindItemNeighborsTask\n\t/go/gorse/master/tasks.go:237\ngithub.com/zhenghaoz/gorse/master.(*Master).runRankingRelatedTasks\n\t/go/gorse/master/tasks.go:455\ngithub.com/zhenghaoz/gorse/master.(*Master).RunPrivilegedTasksLoop\n\t/go/gorse/master/master.go:285"
}

After restart the master container the issue resolves.

It is because the configuration haven't been applied.


Upgrade base images, install kubectl, disable autoDelete

This pull request contains three patches, all relatively minor:

Upgrade base images to the current 1604-LTS, which is the current LTS Ubuntu release.
Install kubectl package in Spinnaker instance. It is useful for debugging kubernetes authentication issues.
Disable autoDelete option for persistent disks. It causes the instance to reset to the default state on each restart.

Hold on, I am not sure about autoDelete after all.


what happens if we exceed isize::MAX on 32-bit platforms?

The ptr::offset function, which underlies safe slice indexing, says in its docs:

If any of the following conditions are violated, the result is Undefined Behavior...The computed offset, in bytes, cannot overflow an isize...memory acquired directly from allocators or memory mapped files may be too large to handle with this function.

Since the map function is unsafe, it's arguably fine for it to expose possible UB in this way. But I think most people reading the docs won't have any idea that this is a requirement. ~Maybe it would be better for Deref to panic rather than to return a slice that's "unsoundly large"?~ (Edit: Probably just return an error if we try to mmap something larger than isize::MAX?)

@oconnor663 can you be more specific with your concern? Do you have a specific potentially buggy line of code in mind, or perhaps a reproducible example?

I'm worried about what happens in the case where someone on a 32-bit system memory maps a 3 GB file (that is, longer than std::isize::MAX but shorter than std::usize::MAX), and then tries to read one of the bytes near the end. Here's what I believe will happen:

memmap checks that the file isn't too long for usize, and it'll assemble a &[u8] slice that's 3 GB long.
Next, the caller indexes into that slice to read a byte, somewhere near the end. Maybe just to print it or something.
That indexing operation translates into usize's SliceIndex::get_unchecked() implementation, which in turn calls std::ptr::add, with a usize argument that's larger than std::isize::MAX.

According to the ptr::add / ptr::offset docs I linked above, that's all it takes to trigger undefined behavior. And those docs specifically note that memory mapping is one way to get your hands on a slice large enough to hit this.
That said, I have no idea what actually happens in this case. It could be that, even though the usize gets cast into an isize and becomes negative, the result of all the pointer arithmetic still ends up being what the caller thought they were asking for? Maybe everything appears to work? Even if that's the case, I don't think we want to rely on it remaining the case.
My instinct is that MmapOptions::get_len, which is currently making sure the length doesn't overflow usize, should instead be making sure the length doesn't overflow isize. But maybe someone who knows more about why these limitations were originally put in place could give better advice.

Early indications are that test fails with ENOMEM on 32bit platforms, which seems like it might be the correct behavior.

Hmm, maybe we should test whether a 2^31 - 2 byte mmap (just under isize::MAX) succeeds? If not, it could be that we're blowing past some per-process limit on the system.

Some more discussion here: https://www.reddit.com/r/rust/comments/9ghwuv/hey_rustaceans_got_an_easy_question_ask_here/e6fx4h2/
Relevant article: https://trust-in-soft.com/objects-larger-than-ptrdiff_max-bytes/
I get the feeling that these are problems we don't want to touch with a 10 foot pole :)

Update: As of https://github.com/rust-lang/rust/pull/53784, slice::from_raw_parts will panic in debug mode, if the length in bytes exceeds isize::MAX.

Note that stdlib debug asserts don’t ever run in official compiler builds. Only for like the rust-lang test suite.

@oconnor663 The only places that use ptr::offset in the code use it either with an alignment, which never exceeds the length of a page (4069), or the offset provided to MmapOptions::offset(usize).  I agree that if this offset usize overflows isize::MAX it will cause issues, but I'm not seeing any reason why the length exceeding isize::MAX would be problematic, since slice::from_raw_parts is used for creating the slice.

Woops, sorry I completely missed the link between ptr::offset and from_raw_parts (https://github.com/rust-lang/rust/pull/53784 makes it clear).


Blank Screen on Boot (screen is working)

My comma two boots up showing the comma logo, but then goes to a blank screen and never fully starts. I can hold down the power button and I will see the reboot/power down options, but that's all I can do.
Device/Version information:

Device: comma two
Dongle ID: a2b1d650214053d7
Route: N/A
Timestamp: N/A
Version: Unable to get because of bug
Car make/model: Toyota Prius Prime 2020

I see your latest routes are from master. There's many ways this can happen when you're checking out other branches/forks, including not initializing the submodules. I suggest factory resetting your device and sticking with the release branch.


Inconsistent BITCOUNT: BITCOUNT against test vector #4

Steps to reproduce
Run the commands mentioned in the test on Line 68 in the file https://github.com/AshwinKul28/dice-tests/blob/main/tcltests/unit/bitpos.tcl#L68.
Expected output
The expected output when the above set of commands when run on Redis
Expected [r bitcount str 0 -1 bit]

Observed output
The observed output when the above set of commands when run on DiceDB
26 (context: type eval line 5 cmd {assert {[r bitcount str 0 -1 bit]

The steps to run the test cases are mentioned in the README of the dice-tests repository.
Expectations for resolution
This issue will be considered resolved when the following things are done:

Changes in the dice repository code to meet the expected behavior.
Successful run of the tcl test behavior.

You can find the tests under the tests directory of the dice repository and the steps to run are in the README file. Refer to the following links to set up DiceDB and Redis 7.2.5 locally

Setup DiceDB locally
Setup Redis 7.2.5 locally
Setup DiceDB CLI

Hi @lucifercr07 , can you assign this task to me?

@Nijin-P-S assigned, thanks for contributing.

Hi @lucifercr07 ,
I wanted to reach out regarding the issues I was assigned. Despite my efforts, I’m finding it challenging to make progress on them. Additionally, I have a busy period at my full-time job coming up and won’t be able to dedicate the time needed for the next couple of weeks.
I think it would be best for me to unassign these issues so someone else can take them on. Thank you for understanding, and I hope to contribute again in the future!

@lucifercr07 I see there's no assignee on this issue, can I take this up?

I'm getting this error:
Cleanup: may take some time... OK
Starting test server at port 21079
[ready]: 35098
Testing unit/auth
[1/6 done]: unit/auth (0 seconds)
Testing unit/bitpos
[exception]: Executing test client: ERR unknown command 'FLUSHALL', with args beginning with:.
ERR unknown command 'FLUSHALL', with args beginning with:
    while executing
"[srv $level "client"] {*}$args"
    (procedure "r" line 7)
    invoked from within
"r flushall"
    (procedure "run_external_server_test" line 24)
    invoked from within
"run_external_server_test $code $overrides"
    (procedure "start_server" line 61)

while running the test suite.
@lucifercr07

I'm getting the command not known error directly running it in dicedb instance as well. is it not yet integrated @lucifercr07 ?

@lucifercr07 @AshwinKul28 I confirmed that all BITCOUNT test cases pass on the master branch, both with and without multithreading enabled. However, the issue occurs when running the test cases on the dicedb Docker image.
I’d appreciate some help in reproducing this.
[ok]: BITCOUNT returns 0 against non existing key (1 ms)
[ok]: BITCOUNT returns 0 with out of range indexes (0 ms)
[ok]: BITCOUNT returns 0 with negative indexes where start > end (0 ms)
[ok]: BITCOUNT against test vector #1 (0 ms)
[ok]: BITCOUNT against test vector #2 (0 ms)
[ok]: BITCOUNT against test vector #3 (0 ms)
[ok]: BITCOUNT against test vector #4 (0 ms)
[ok]: BITCOUNT against test vector #5 (1 ms)
[ok]: BITCOUNT fuzzing without start/end (132 ms)
[ok]: BITCOUNT with start, end (1 ms)
[ok]: BITCOUNT syntax error #1 (1 ms)
[ok]: BITCOUNT syntax error #2 (0 ms)
[ok]: BITCOUNT regression test for github issue #582 (0 ms)
[ok]: BITCOUNT misaligned prefix (0 ms)
[ok]: BITCOUNT misaligned prefix + full words + remainder (0 ms)
[ok]: BITOP NOT (empty string) (1 ms)

This issue is not reproducible after building and running the master branch in Docker. @lucifercr07 @AshwinKul28 , should we go ahead and close it?

This issue is not reproducible after building and running the master branch in Docker. @lucifercr07 @AshwinKul28 , should we go ahead and close it?
cc @JyotinderSingh

Sounds good


Alternative frontend

Hello jo-m,
i have started an alternative frontend with server-side rendering, a demo can be seen at https://trains.shakik.de/s/
It is pretty bare-bones for now, and i don't intend for it to reach feature-parity with your Vue frontend.
Notable features:

automatic reloading of the front page whenever a new train has been recorded
horizontally scrollable thumbnails
loads fairly quickly
(best of trains list, but that requires an additional table with view statistics, and it's broken with trains_v2)

Currently i have put up the code at https://gitlab.aachen.ccc.de/clonejo/onlytrains-frontend-rs/.
I am wondering if you would want to take the code into the main trainbot repo, but i assume you don't want to take maintainership for a bunch of Rust code :)
It is probably better if i just host a separate repo, then it is clear the frontend is not always kept up to date with trainbot and i can just push/review changes myself.
Best,
clonejo

Nice!
I think to keep things uncomplicated, it is easier if you keep hosting the repo under your control. If you want, you can make a MR to link to the alternative frontend from the main README.
best
jo-m


feat(ui): option to disable workflow editor (#11391)

Fixes https://github.com/argoproj/argo-workflows/issues/11391

As discussed by agilgur5 in #11391 this doesn't prevent submission of arbitrary workflows and there is already a mechanism that does.

templateReferencing is too basic, doesn't allow certain superusers to not use template ie via cli


draw fake skybox to fix missing skybox on portal

This draws a huge box on the portal when the skybox isn't rendered

drawing a box with a negative size would probably be more efficient than drawing 8 quads

or maybe just draw 1 quad thats always in view


Items disappearing in Grids

Issue description: I played with RS and had plenty of Items in my Quantum Storage Disk. I see that they are Items in the Disk, but i cant accses them directly from the Grid. Used all Grids to check!
What happens: Items not showing up on Grid
What you expected to happen: Items showing up on Grid
Steps to reproduce:

Play RS
Have a Quantum Storage with 500k+ Items
Continue to play RS
...

Version (make sure you are on the latest version before reporting):

Minecraft: Minecraft Version: 1.12.2
Forge: forge-<IP_ADDRESS>06
Refined Storage:r efinedstorage-1.6.14.jar

Does this issue occur on a server? [no]
If a (crash)log is relevant for this issue, link it here:
[pastebin/gist/etc link here]

Restarting and/or deleting playerdata usually fixes this.


Cast button not showing up for Custom Receiver or Styled Media Receiver Applications

Hi there,
I have cloned this repo and trying to cast the media from my browser, but Cast button is not visible (checked in elements & found "display: none" for it). Although it's working fine for DEFAULT_MEDIA_RECEIVER.
Steps I followed:

Registered my device on google cast console [status - Ready For Testing].
Registered Application and configured the "Application ID" into sender App (Tested for both Custom & Styled Media Receiver)
Deployed both Sender & Receiver applications and put the URL into cast console URLs.

Can anyone help me to know if something is missing or wrong in the above steps.
Note: Working fine for DEFAULT_MEDIA_RECEIVER & I'm connected to the same network.
Thanks

Is your registered receiver app in a "published" state in the developer console?

Hi, Sorry for late reply but I found the issue. I was using the wrong Serial Number to register my application. Actually there are multiple serial numbers like -

In CC box
Settings -> About -> Status -> S/N
Setting -> Cast ->  S/N

Here is the link to find the correct S/N - https://developers.google.com/cast/docs/registration#find_device_serial_number


cri/server: use consistent alias for pkg/ioutil

Consistently use cioutil to prevent it being confused for Golang's ioutil.

failures with go modules on this PR as well; I'm wondering if we're hitting github rate-limits or so?
go mod download<EMAIL_ADDRESS>invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/50116b685aa496e1a693e3c110151d91606e5be8b29e7400b3b606b9eed5ad41: exit status 128:
	fatal: unable to access 'https://github.com/julienschmidt/httprouter/': Failed to connect to github.com port 443: Connection timed out
make: *** [integration] Error 1
Makefile:201: recipe for target 'integration' failed
Error: Process completed with exit code 2.

Yeah, the name is a bit poorly chosen. We were already using this alias throughout our code, and I thought that renaming the package would potentially be more impactful than just aligning this file.

(rebased to get a fresh run of CI)


feat: add prometheus/promlens

prometheus/promlens: PromLens – The query builder, analyzer, and explainer for PromQL
$ aqua g -i prometheus/promlens

How to confirm if this package works well
Reviewers aren't necessarily familiar with this package, so please describe how to confirm if this package works well. Please confirm if this package works well yourself as much as possible.
Command and output
$  promlens
ts=2022-12-15T09:17:20.998Z caller=main.go:202 level=info msg="No link sharing backends are enabled - disabling link sharing functionality."
ts=2022-12-15T09:17:20.999Z caller=main.go:216 level=info msg="No Grafana backend enabled - disabling Grafana datasource integration."
ts=2022-12-15T09:17:20.999Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:8080
ts=2022-12-15T09:17:20.999Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:8080

You can access promlens from your browser at http://localhost:8080.

Thank you for your contribution!

Released. https://github.com/aquaproj/aqua-registry/releases/tag/v3.105.0


Some thoughts on architecture

Context: I received an email asking what I think about dumper.io. Haven't used it, didn't read the code (yet).
After a quick look, it seems the database backup procedures are supposed to run from the same process that serves the HTTP requests.
I think this a bad choice to carry on such operations, mainly because it breaks separation of concerns for processes. This is important ruby web server handles just web requests, and doesn't try doing anything else, especially tasks with different lifecycle models. Not a big deal for small projects, but after a certail system size and load this becomes important.
There are a lot of potential issues with threading (if you use standard ruby implementation, and even with jruby). Basicly, it depends on the workload types and the load that each server has to handle, but given the interpreter thread would block on system calls and database drivers ruby are typically implemented using blocking API - doing database activity in the background may cause unwanted behavior and a decrease in service quality during the backup.
UPD: ok, scratch this part, I've checked out the readme and the execution model is alright (won't be causing the issues I'm describing).
The other issue is the database backup process should really be managed separately from any other job types: having a separate small process would be a better solution in my opinion. It has a benefit of having the ability to spawn in wherever needed - given I typically deploy separate nodes for web servers, background jobs workers and other stuff. Having a separate web server nodes makes the system load on the nodes more predictable and easier to manage. For example, memory consumption is very different on web and background job worker nodes in a typical deployment. Having said that, it seems to be a much better idea to have a separate node for such kind of process. We're talikng about database backups - it doesn't need to scale together with web servers (hopefully).
That's my thoughts.
Some more thoughs on the general approach to database backups
Postgres is my most used database, so I'll be talking about it. Probably this can be applied to other databases.
In general, for postgres backups today I'd use some solution that stores WALs and periodically does the full database snapshots. This is better than just a dump, because with such approach the backup sizes are much smaller, and there's no load to the database server (except for the file copying) or table locking - so the backup can be done online. A small bonus is that one could restore the database to a specific point in time, not just to the predefined snapshot, as long as that point in time is covered my WALs you have.
An example of this technique can be seen in the snapshotting implemnetation of AWS RDS for Postgres.
This, of course, is not directly relevant to the dumper because it's a different approach to the same problem. But I just wanted to share my thoughs on this topic too. I'd still find usecases for something similar to dumper, but probably for managing the development environment.

Hey @MOZGIII, thanks for your thoughts.
Dumper agent runs as a thread, just like (everyone's favorite) New Relic does, and its sole purpose is mostly to sleep and wakes up every 1-10 minutes to check if there's a job, and when there's a job, we fork(2) a child process that lives independent from the web app processes, and then it fork-exec a command line tool such as pg_dump. We even use posix_spawn(3) to ensure O(1) fast forking from a large Rails process.
And if you use forking web servers such as Unicorn, Passenger or Puma, the agent only runs inside the singleton, master process because the only current thread is inherited to a child process on Linux and worker processes won't have the agent thread. Therefore, by design, only one agent runs per server. The overhead of all this is negligible, and in fact, no customers reported increased server resource usage so far.
Obviously, Dumper is not intended to be used by Facebook or Google, but for small- to mid-sized projects that includes up to 10-20 servers, the architecture makes sense I believe.
As to postgres back up using WAL (in case of MySQL, binlog), I agree that incremental backups are desirable to minimize the load to the database. But also keep in mind that when restoring, you need to start roll-forward from the last dump, which can be very far away, and so many files can be involved that is prone to trigger the secondary error. For that reason, most DB admins take a logical dump every once in a while anyway. But thank you for pointing that out, WAL/binlog support is actually on the upcoming feature list.
Thanks for your feedback! 👍

Don't know about backups, but I'd really look forward to some semi-hosted (like dumper) online migration implementations.

What do you mean by migration?

DB schema and data updates (stuff you usually run along with the application code updates).

For example, this is an opensource online migration tool for MySQL (why do people even use it now?) from GitHub: https://github.com/github/gh-ost

Ah, ok. Online migration tools are critical if you have a large table because basically it makes the table read-only, which is often unacceptable. I like Github's implementation because it's trigger-less. In any case, it's an advanced topic for relatively large scale projects. Maybe replica as a service can be a possible solution?

Maybe, I don't really know. I've implemeted this paradigm couple times. In general you'd need a smart database balancer (like pg_bouncer, but hooked into some state manager), and the logic to apply migrations. Databases usually go blue-green, so users would probably provide their own replicas.
The thing is, there are a lot of different usecases, however there's still some common logic around the online migration process that can be automated. Like switching the databases on balancer and rolling app updates. Probably this would work best if integrated with kubernetes or sth like that.

And the reason I want a semi-hosted service (or a tool) for that is because it's one of those things that you don't want to invest to as a typical online business owner (unless you really have resource to spare), but as an engineer it would benefit you a lot.

Yeah, I hear you. It's a PITA for sure, and kubernetes / blue-green mitigated some, but so far as I can see, there's no single best practice. That makes it really hard to generalize as a product. Anyway, thanks for the input - it's a good discussion. 😄


fix: Allow version range instead of fixed version

Description

Motivation and Context

Breaking Changes

How Has This Been Tested?

[ ] I have updated at least one of the examples/* to demonstrate and validate my change(s)
[ ] I have tested and validated these changes using one or more of the provided examples/* projects

[ ] I have executed pre-commit run -a on my pull request

This PR is included in version 1.2.1 :tada:


IPv4 ServiceEntry is not tracked properly on EKS when using ISTIO_ENABLE_IPV4_OUTBOUND_LISTENER_FOR_IPV6_CLUSTERS

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description
On an IPv6 EKS cluster, with the IPv4 link-local egress, I'm using the new flag ISTIO_ENABLE_IPV4_OUTBOUND_LISTENER_FOR_IPV6_CLUSTERS. It appears to properly allow IPv4 egress traffic from my pods, which it did not without the flag.
When viewing the traffic in the sample/addons configs for prometheus and kiali, all IPv4 traffic is tagged as PassthroughCluster as the destination.
When comparing this behavior to my IPv4-only EKS clusters, the IPv4 traffic that matches a ServiceEntry definition is properly tracked as targeting that ServiceEntry object, and not PassthroughCluster. This allows me to differentiate which traffic I have proper ServiceEntry definitions for, versus unlabeled traffic.
To restate the above:

Cluster Type
Egress Traffic Type
Label

IPv4
IPv4
ServiceEntry ✅

IPv6
IPv6
ServiceEntry ✅

IPv6
IPv4
PassthroughCluster :x:

Example ServiceEntry:
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: smtp-traffic
spec:
  exportTo:
  - .
  hosts:
  - email-smtp.us-east-1.amazonaws.com
  location: MESH_EXTERNAL
  ports:
  - name: smtp
    number: 587
    protocol: TCP
  resolution: DNS

Version
Tested with istio 1.20.1.

Additional Information
Related: https://github.com/istio/istio/issues/46719 https://github.com/istio/istio/pull/46800

@mmerickel thanks for reporting this.
I will take a look at it when i have some bandwidth in the coming weeks.

Can you also attach a screenshot so I can trace the issue through the metrics pipelines

FWIW it feels likely this is related to https://github.com/istio/istio/issues/49476

We've hit the same problem, eks is v6, and we're trying to turn on whitelist-only egress, and it's problematic to make it work for ipv4 addresses.
Only domains that you enlist explicitly work fine when you force dns proxy https://istio.io/latest/docs/ops/configuration/traffic-management/dns-proxy/ - they then get resolved to ipv6 egress address and are routed just fine.
But forcing dns proxy does not work for wildcard domains, tho they can't be added to ServiceEntry with "resolution: DNS" attribute.
So, we can serve
ipv4.com
in our egress, but
*.ipv4.com
just does not hit it in any way.

This is still a problem on istio 1.20.4. IPv4 traffic is not properly tagged if it matches a ServiceEntry.

When looking at istioctl pc l myapp-web-5f5b6c695f-h5l68 the only things listening on ipv4 interfaces are the following two entries:
::,<IP_ADDRESS>           15001 ALL                                                                                PassthroughCluster
::,<IP_ADDRESS>           15001 Addr: *:15001                                                                                Non-HTTP/Non-TCP

I think this explains why everything is marked as PassthroughCluster.
It'd be great if istio could do something to make EKS-over-IPv6 a little more of a first class citizen.

When ISTIO_ENABLE_IPV4_OUTBOUND_LISTENER_FOR_IPV6_CLUSTERS is true all the service entries should be listening on both ipv6 and ipv4 interfaces.

Not stale… I hope a team member can keep this open. Thanks.

@day0ops can you keep this open?

Can this please be reopened? It is still an issue in the latest release 1.21.2.

I feel like the issue here is pretty clear based on my analysis above - that when this cluster setting is defined the istio-proxy should be listening on both ipv4 and ipv6 addresses instead of just ipv6 so that it can be used with ipv4 external services. Is IPv6 on EKS just not a concern for istio? How are other people not running into this issue on EKS? Are folks not tracking egress traffic to external services?

I suspect this is very related to https://github.com/istio/istio/issues/48829 in that the service entries simply are not tracking ipv4 traffic when used in this mode.

This is still an active issue.

not stale

Should be fixed by https://github.com/istio/istio/pull/52733. Thanks!

Thank you so much for taking this on @howardjohn. Can’t wait to add back some service entries.


Unable to add pytest argument "--reruns=1"

I am trying to add additional argument for test path and reruns, but it fails when I add the argument for rerun

Can you provide me a failing example workflow?

Should probably be fixed with https://github.com/pavelzw/pytest-action/commit/510c5e90c360a185039bea56ce8b3e7e51a16507
If this is still an issue, feel free to reopen.


Tasks are dispatched with invalid ranges

See the attached code for a minimal repro. I am new to this API so if I am making an obvious mistake, please do let me know. This bug appears to occur reliably when TASK_COUNT >= 50 and when TASK_COUNT is 5000 it will always happen.
Context / use case: I am developing a terrain streaming system. On world load I spawn about 5000~ tasks which procedurally generate each terrain cell's heightmap, then another 5000~ with various dependencies to generate the meshes.
#include "TaskScheduler.h"

int main(int, char**)
{
    static enki::TaskScheduler s_scheduler;
    s_scheduler.Initialize();

    static constexpr size_t TASK_RANGE = 65*65;
    static constexpr size_t TASK_COUNT = 50;

    struct Repro : public enki::ITaskSet
    {
        Repro() : enki::ITaskSet(TASK_RANGE) {};
        virtual void ExecuteRange( enki::TaskSetPartition range, uint32_t) override
        {
            if (range.start > TASK_RANGE || range.end > TASK_RANGE)
            {
                __debugbreak(); // bug?!
            }
        }
    };

    std::vector<std::unique_ptr<Repro>> jobs;

    for (size_t i = 0; i < TASK_COUNT; ++i)
    {
        jobs.emplace_back(std::make_unique<Repro>());
    }

    for (std::unique_ptr<Repro>& job : jobs)
    {
        s_scheduler.AddTaskSetToPipe(job.get());
    }

    s_scheduler.WaitforAll();
}

From a preliminary search of the library code, the problem appears to occur only when the task buffer is full and it must be executed immediately. The code does not properly handle ranges that are not divisible by the range step, so what should be the final step overruns into an infinite loop.
https://github.com/dougbinks/enkiTS/blob/master/src/TaskScheduler.cpp#L656
I have removed this block - it doesn't make sense to me, if we were simply executing the same job immediately because there was no room, why would we have to tweak the range that we've already calculated? This fixed the problem for me. I did verify that every element of the range was being generated with the following test:
struct Repro : public enki::ITaskSet
{
    std::vector<int> data;
    Repro() : enki::ITaskSet(TASK_RANGE) { data.resize(TASK_RANGE); };
    virtual void ExecuteRange( enki::TaskSetPartition range, uint32_t) override
    {
        if (range.start > TASK_RANGE || range.end > TASK_RANGE)
        {
            __debugbreak(); // bug?!
        }

        for (size_t i = range.start; i < range.end; ++i)
        {
            data[i] = 1;
        }
    }
};

... later

for (std::unique_ptr<Repro>& job : jobs)
{
    int count = 0;
    for (size_t i = 0; i < TASK_RANGE; ++i)
    {
        count += job->data[i];
    }
    ASSERT(count == TASK_RANGE);
}

Many thanks for this report and especially for a simple reproduction - it seems my own tests for this didn't trigger the exact flow required to get the issue.
I've added your reproduction as part of the test suite, fixed the issue and added an extra assert.

Thanks, that seems to have fixed it for me!

Excellent, I'll close this issue and the PR then, but please do re-open them if there are any further problems with this.
You should watch #62 if you are interested in an example of how to add lots of tasks.


ALDoc repeats remarks until they change

1. Describe the bug
If xml comments include remarks, the same remark is repeated for each method without a remark.
2. To Reproduce
Use the following code to have the first two procedures contain the remarks from the first:
    /// <summary>
    /// An important procedure
    /// </summary>
    /// <param name="tableId">The ID of the table</param>
    /// <param name="tableFilterText">a table filter as text</param>
    /// <param name="doLookup">Select whether you like to do a lookup or validation.</param>
    /// <returns>A table filter as text</returns>
    /// <remarks>
    ///   The filter is composed from the table name and field CAPTIONS!
    /// </remarks>
    procedure HandleTableFilter(tableId: Integer; tableFilterText: Text; doLookup: Boolean): Text
    begin
    end;

    /// <summary>
    /// Apply a table filter to a record reference.
    /// </summary>
    /// <param name="tableFilterText">The table filter as text.</param>
    /// <param name="toRecRef">The record reference to apply the filter to.</param>
    procedure ApplyTableFilter(tableFilterText: Text; var toRecRef: RecordRef)
    begin
    end;

    /// <summary>
    /// A third procedure
    /// </summary>
    /// <param name="tableFilterText">The table filter as text.</param>
    /// <param name="toRecRef">The record reference to apply the filter to.</param>
    /// <remarks>
    ///	This one is shown.
    /// </remarks>
    procedure ThirdProcedure(tableFilterText: Text; var toRecRef: RecordRef)
    begin
    end;

3. Expected behavior
The remarks (and possibly other values) must be cleared after processing each procedure.
4. Actual behavior
The first two procedures contain the remarks from the first:
5. Versions:
Name: AL Language extension for Microsoft Dynamics 365 Business Central
Id: ms-dynamics-smb.al
Description: Development tools for Dynamics 365 Business Central
Version: 12.0.847725
Publisher: Microsoft
Version: 1.81.0 (system setup)
Commit: 6445d93c81ebe42c4cbd7a60712e0b17d9463e97
Date: 2023-08-02T12:37:13.485Z
Electron: 22.3.18
ElectronBuildId: 22689846
Chromium: 108.0.5359.215
Node.js: 16.17.1
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621

I'm sorry, the issue seems to be in docfx. The generated reference yaml itself does not repeat. Closing this issue.

It is really hard to get to the point if you don't know the tools well. I now assume again it is AlDoc. If you change the template file BusinessCentralApplicationObject.html.primary.js slightly, it works. it can be an issue in docfx probably, but eventually this change is easier:
From:
function updateMemberDetails(members) {
	if (members) {
		members.forEach(element => {
			element.commentId = element.commentId || null;
			element.examples = element.example || null;
			element.remarks = element.remarks || null;
			element.summary = element.summary || null;
			element.description = element.description || null;
			updateSyntax(element.syntax);
			updateObsoletion(element);
		});
	}
}

To:
function updateMemberDetails(members) {
	if (members) {
		members.forEach(element => {
			element.commentId = element.commentId || null;
			element.examples = element.example || null;
			element.remarks = element.remarks || "";
			element.summary = element.summary || null;
			element.description = element.description || null;
			updateSyntax(element.syntax);
			updateObsoletion(element);
		});
	}
}

Hi,
We have internally discussed this issue and have decided not to accept this issue to be fixed at the moment to prioritize other issues. Thank you for raising this concern.
Thanks


add sqlquery receiver from contrib

@pmcollins updated to remove testcontainers version bump and I'll address that in a subsequent PR.

Thanks @rmfitzpatrick for your help on this PR 🙇


Validation NoneAttestation fails

I am trying to validate NoneAttestation (user refused to hand over the key attestation) and it fails asi the verify_x509_chain function gets empty chain.
The NoneAttestation doesn't have a CA trust chain and it probably shouldn't be subjected to these checks.

If I understand you correctly, you've configured the Fido2Server instance to use attestation=DIRECT, to request attestation from the user. The user is then refusing the request for attestation, which (on some browsers) returns a registration with NONE attestation. Am I right so far?
The default is to not validate attestation at all, and should have no problem handling this case. If you provide your own verify_attestation function to the server, then it's up to you to handle this case. The abstract AttestationVerifier will throw an UnsupportedType exception, which you can catch outside of the call to register_complete if you wish. The exception contains a reference to the auth_data object.
You could also define your own logic for validating the NONE attestation, but I can't really say I see a scenario where it makes sense to verify some attestation, but then also allow NONE attestation.

Closing issue due to inactivity.