Comparison to others?

I read this article and it left me perplex... https://medium.com/javascript-scene/identity-crisis-how-modern-applications-generate-unique-ids-39562736f557
Would be great to put this generator in context..
How does it compare to nanoid, ulid, uuidv7 or cuid?
what's the randomness size?  is it sortable?
something like this.. https://blog.daveallie.com/ulid-primary-keys
Can you help add a vertical to this project? https://github.com/adileo/awesome-identifiers

Cuid2 is good when you need something secure, extremely collision resistant, and your system is distributed or decentralized (e.g. you want to be able to create records with ids on the client side), or you are building software that may need to scale horizontally).

Nanoid has weaker anti-collision and security guarantees because you have to trust the host’s web crypto API implementation of the “cryptographically secure” random number generator. Until 2018, Chromium’s wasn’t, and it’s not alone.
ulid and uuidv7 (and the now deprecated Cuid v1) leak creation times, which may be a security hazard. They do it to make a faster db primary key. Monotonically sortable ids are faster to look up. Cuid2 intentionally trades a little lookup performance for security. Over 11 years’ experience using Cuid taught us we were optimizing for a little performance gain at the expense of a lot of security.

Total entropy is 36^(n-1)*26 where n defaults to 24 but can be as much as 32. You can roughly estimate how many ids you can generate before reaching 50% chance of collision with: sqrt(36^(n-1)*26).
V4 UUID has 5.316912E+36 max entropy, but actually far less because most random generators used to make them are not random enough, leading to billions of real-life id collisions. Cuid2 has up to 4.57458E+49, roughly 13 orders of magnitude larger and defaults to 1.62155E+37 (still many times larger than uuid) and you’ll need to generate ~4.0268498e+18 to reach 50% chance of collision.
Cuid2 also has much better guarantees of using the full entropy range evenly, reducing chances of collision much better than any normal identifier, even when those identifiers use web crypto APIs to source their random entropy. We test this very thoroughly (in parallel on different CPU cores to simulate distributed use) and produce histograms and randograms with each release, to prove it.
It is not sortable, for security reasons. No sortable id is safe from leaking user data.

Hi Eric,
Thanks for the prompt reply,
first, let me share my status,

Could you help me fill the CUID2 column?

Now, let me try to get context about your comments:
I must point that as engineers, we work on facts, not opinions or open statements. Please
"Nanoid has weaker anti-collision and security guarantees than Cuid2 because you have to trust the host’s web crypto API implementation of the “cryptographically secure” random number generator. Until 2018, Chromium’s wasn’t, and it’s not alone."

this is a valid statement, however the creation of ids typically happens serverside and not on the browser (which would be an even bigger security risk).  How does cuid2 solve for this issue?

ulid and uuidv7 (and the now deprecated Cuid v1) leak creation times, which may be a security hazard. They do it to make a faster db primary key. Monotonically sortable ids are faster to look up. Cuid2 intentionally trades a little lookup performance for security. Over 11 year’s experience using Cuid taught us we were optimizing for a little performance gain at the expense of a lot of security.

If I may, can you please be more specific... is CUID2 k-sortable? if not, you are solving for a different use case, let's be fair and point it out in the table above.

"Cuid2 also has much better guarantees of using the full entropy range evenly, reducing chances of collision much better than any normal identifier, even when those identifiers use web crypto APIs to source their random entropy. We test this very thoroughly (in parallel on different CPU cores to simulate distributed use) and produce histograms and randograms with each release, to prove it."

Have you done a distribution analysis of the other GUID?  We know UUID <v5 such in that matter, what about v7, Ulid, nanoid and co? once again… a scientific approach

If you could help me fill the table above, I think it will bring clear perspective of benefits and shortcoming of each GUID.
There is rarely one way to skin a cat in Architecture. It all depends on requirements.

"Total entropy of Cuid2 is 36^(n-1)*26 where n defaults to 24 but can be as much as 32. You can roughly estimate how many ids you can generate before reaching 50% chance of collision with: sqrt(36^(n-1)*26).
V4 UUID has 5.316912E+36 max entropy, but actually uses far less in real life (closer to 5 digits of a Cuid2) because most random generators used to make them are not random enough, leading to billions of real-life id collisions. Cuid2 has up to 4.57458E+49, roughly 13 orders of magnitude larger and defaults to 1.62155E+37 (still many times larger than uuid) and you’ll need to generate ~4.0268498e+18 to reach 50% chance of collision."

Not an expert, let's make it simple : How many bit of randomness do you use?  What's the size of an id generated by cuid2?

Cuid2 intentionally trades a little lookup performance for security. Over 11 year’s experience using Cuid taught us we were optimizing for a little performance gain at the expense of a lot of security.

Mind providing more details. What use case are you referencing? Problem statement, issue and how did you fix it?  More generally, why do you see an id generation date as a security leak issue?

also, feel free to point any issues in the table above. just trying to capture the proper dimensions and data.

I don’t have time to address everything right now, but I can start by dispelling a couple misconceptions.

I must point that as engineers, we work on facts, not opinions or open statements.

Yes. When I say that Nanoid, and all other PSRNG-based identifiers that offload their security to the browser or OS are less secure and collision-resistant than Cuid2, that is a fact, not an opinion, because as I mentioned before, browsers and operating systems don’t have a great track record of generating truly random entropy - even in their crypto APIs which are supposed to be CSPRNGs. Also, those entropy sources use pools that can run out of entropy if you hit them with too many requests too fast, which is why Nanoid keeps a buffer. It may be possible to outrun Nanoid’s buffer or run a denial of service attack on the OS random entropy.
Poor random number generators caused the id collisions that inspired the original Cuid in the first place.
Why not use window.crypto to get random values? You have to trust 2 things:

The entropy source. Entropy is notoriously hard to do right. I don’t trust any single source. Hence, the entropy smoothie in Cuid2.
The hash function. Generally, a CSPRNG (“Cryptographically Secure” Pseudorandom Number Generator) needs to hash its entropy, both for security, and for random distribution reasons in order to prevent collisions. But RNGs are generally assumed to be safe and can have bugs for years before they are properly reported and patched. See https://bugs.chromium.org/p/chromium/issues/detail?id=552749 for example.

this is a valid statement, however the creation of ids typically happens serverside and not on the browser (which would be an even bigger security risk). How does cuid2 solve for this issue?

It isn’t true that “the creation of ids typically happens on the server side”. Every major app I have been involved in since 2012 generates a ton of ids client-side, thanks to the distributed capabilities provided by Cuid (and we plan to continue with Cuid2). Since I mentor and advise teams, that’s hundreds of apps. This makes our apps feel more responsive to users because db round-trips are reduced, and allows users to easily work offline.
Generating ids with Cuid2 is not a security risk in the browser because the chances of accidentally guessing a valid existing id are astronomically low, and you can’t update a record you don’t have access to even if you know a valid id (assuming standard server authentication/authorization). Since the data in an id is not used for anything other than unique identification and the character set i’d limited, it does not pose an insertion attack threat.
Unlike Nanoid, Cuid2 does not need to trust the random numbers it uses because it mixes them with other, guaranteed sources of entropy, and relies on the NIST-standard Sha3 hashing algorithm for random distribution, instead: the most secure, tiny cryptographically secure hashing algorithm we could find.

If I may, can you please be more specific... is CUID2 k-sortable? if not, you are solving for a different use case, let's be fair and point it out in the table above.

No, Cuid2 is not k-sortable, but that’s feature, not a bug.
All k-sortable ids are also insecure because they leak timestamp, creation order, or both. Cuid v1 is k-sortable. I realize now, that was a mistake. With modern systems, it is possible to generate k-sortable indexes for any field of any type where you need to optimize lookup performance. Given that, and the ease of adding createdAt db fields, there is no valid reason to leak this information to clients via ids.

Not an expert, let's make it simple : How many bit of randomness do you use? What's the size of an id generated by cuid2?

By default, 124 bits. Length is a parameter in Cuid2, so you can adjust for your particular needs.

Mind providing more details. What use case are you referencing? Problem statement, issue and how did you fix it? More generally, why do you see an id generation date as a security leak issue?

Sure. One obvious example: In all of Europe, personally identifiable information is protected, including things like user age. If a user has an id with a timestamp, and that id is publicly exposed in any way (either visible in the UX or not) your leaking PII, and could be on the hook in  a lawsuit.
Worse though, there are now many Web3 protocols where user money is at risk, and moronically increasing or timestamp ids dramatically reduce the unguessable entropy available in an id, by making a huge chunk of the id guessable. This could make things like secret links (e.g. things like Zoom classroom or private company meeting links) brute-forceable.
In Web3, secret links for airdrop participants, tokens for session authentication, secret nonces for signature validations (to prevent replay attacks, etc) could all be put at risk by monotonic ids.

Hi, well this is great education, thank you.
I am starting to see the details of your implementation and the differences with others
I started filling/completing the xls, it's still a WIP. I know your time is valuable, if I may, I will just ask a few question for the items I am not sure about.
As far as K-sorted or not, IMHO security goes beyond an ID generator: it's an architecture and should be left for the designers to understand +/-.
Below my current draft.

Suggestion: Add a field in the security section called “Cryptographically Secure”.
Cuid2’s output comes from an audited implementation of a NIST-standardized cryptographically secure hashing algorithm (Sha3). It also uses multiple, independent sources of entropy, uses the most input entropy,  and fully utilizes every bit of its available input and output entropy. As far as I’m aware, Cuid2 makes the strongest security guarantees out of all the options you have listed.
Anything using the Web crypto API should be at least partially cryptographically secure, though that is not guaranteed, so that might need an asterisk and a link to one or more of the bugs against web crypto API randomness. Nanoid falls in this category. xid gets partial credit for using a CSPRNG, but only for a small part of the id. It also uses a known-insecure hash (MD5) to hash the hostname, meaning it gets zero points for securely concealing entropy sources.
Anything not using well-known cryptographically secure random number or hashing algorithm would fail this test. Examples: include all uuid versions, cuid, etc, though some uuid v4 implementations use a CSPRNG, putting them in the same category as Nanoid.

As far as K-sorted or not, IMHO security goes beyond an ID generator: it's an architecture and should be left for the designers to understand +/-.

If your ids leak information, your ids are insecure by definition. See the Principle of Least Privilege.

"If your ids leak information, your ids are insecure by definition. See the Principle of Least Privilege."
This sounds like a wiiiide open statement, Eric...
In reality, it narrows to data classification.
In that regard, I haven't heard of any lawsuit against a date attached to an id generation and if it even happens… Then, IMHO the ID itself would fall into the same category.
Anyways, thank you for all these details - I will update the xls tomorrow for your feedback
Great discussion, learned many things.

Here are all the answers:
Cuid2 features:

* Best for: Cryptographically secure, completely opaque, great for distributed systems, flexible length.
* Example: ff2249p15jdlryxw2lhv54g8
* Number range: -
* Bits: Variable. Default: 124, Max: 171
* GB: Binary: 16 Strings: 20
* Generation speed: Fast - Synchronous, offline (For reference, a very slow id would be a db auto-increment, which requires both asynchronous I/O and distributed coordination to prevent collisions, a slow id would be one that is distributed-safe but requires centralized coordination and async round-trips, e.g. Snowflake. All distributed, strongly unique IDs without central coordination would be “Fast”. Something like Nanoid might be “Very fast” because it also uses a very fast algorithm).
* Require coordination: No
* Lifespan: N/A

Bit Distribution

* Time: 64 bits (ms resolution, hashed into 119 bits)
* Sequence/Counter: 64 bits (hashed into 119 bits)
* Machine/Process ID: Host fingerprint: Unlimited, ~320 bits (hashed into 119 bits)
* Randomness: 64+6 bits + ~124 bit salt (hashed into 119 bits)

DB

* Suggested DB Type: varchar(24)
* DB Performance impact: About the same as Nanoid, which should have a high impact (because it is a long, random, non-k-sortable id), Cuid would have medium impact because it’s monotonically sortable but also longer than auto-increment, and a varchar instead of a BigInt. Example of low impact would be a monotonically sortable BigInt.
* Entropy Sources:
  - Initial letter
  - Current system time
  - Pseudorandom values
  - Host fingerprint (Default, configurable: hashed global keys)
  - Session counter
* Collision risks: Cryptographically secure hash. 50% chance at ~sqrt(36^n-1 * 26) (Default: ~4.7383813e+18 generated ids)
* K-sortable: No
* Distributed: yes
* Cursor-based support is basically the same as K-sortable, so no.
* Needs to be configured: No, but it is an option.

Security

* Is the implementation only based on randomness?: No
* Predictability of the ID: Very low (for reference, Nanoid would be low unless there was a serious bug in the CSPRNG, V4 UUID would be high (deterministic next-id calculations are possible in old implementations of JavaScript and PHP, at least) unless it used CSPRNG, Auto-increment would be very high)
* Leaks count: No
* Leaks machine/process info: No
* Leaks the date: No

Shortcomings

* Slightly larger than Nanoid (still <5k, gzipped)

In that regard, I haven't heard of any lawsuit against a date attached to an id generation and if it even happens… Then, IMHO the ID itself would fall into the same category.

Yes, it happens. e.g. duplicate SIM assignments, etc. The id would not qualify by itself because a secure id by itself doesn't leak PII.

Morning Eric,
All good points - not disagreeing.
I couldn't find the detail of the lawsuit, but from the title, it's about a complete Security Architecture, not a (non PII) date leakage.
Putting your statement in context, assume your ulid user id is 01arz3ndektsv4rrffq69g5fav and that related records were creates within a 1ms range, I still have to go through a 2^80 combinations for each ms/record type.  Definitely less than 2^119, but we are not solving for the same problem. Again, it's only an id.. and so many other elements are to be considered to assess proper protection.
Anyway, I am not trying to make a recommendation, rather to provide context - so that engineers can make their own choices based on their NFR
Thanks for all the details and adjustments,
Please find the latest draft in attachment - will run it by other authors to get their feedback/corrections.
It was a pleasure.

Thanks for building the table. Please ping back when you publish.

Putting your statement in context, assume your ulid user id is 01arz3ndektsv4rrffq69g5fav and that related records were creates within a 1ms range, I still have to go through a 2^80 combinations for each ms/record type

If the creation time itself is a user privacy violation (e.g. exact timestamp of a medical event, bank transaction, etc) it’s already game over. You leaked sensitive data already.
ulid is not guaranteed cryptographically secure, which means there may be a deterministic algorithm to guess the next random number, which reduces time to crack the next valid id to an 0(1) operation. This isn’t theoretical. Attacks like this actually happened.

More on 2: Imagine you just saw a ulid on a blockchain representing a purchase of a $350k NFT (say, tokenized real-estate). This transaction represents the life’s savings of the user. Now, the next step after purchase is to open a DM encrypted with the purchaser’s public key. This DM contains a token that will transfer governance permissions to the buyer. The token is a ulid, and happens to have been generated as the very next id in the sequence after the transaction record for the purchase. A hacker sees the purchase id on the blockchain, uses the O(1) next id attack to get the token, transfers governance to their own wallet, and then immediately swaps the token with an open order book offer and pockets the money. They buyer is out $300k.
This is a fictional scenario but the mechanics are real. We saw real attacks like this one happen to v4 uuid users, and a user reported a next random number vulnerability in Cuid v1. Luckily, the other entropy sources in Cuid combined were enough to prevent any real losses that we’re aware of, but that was too close for comfort.
Security in layers! In order to be genuinely safe, you need multiple sources of entropy to reduce chances that an attacker can manipulate all of them, and then you need to guard that entropy like Fort Knox to prevent leaking any the attack tools to the attackers in the first place.

Again, it's only an id.

The only way to be sure it’s only an id is to hash the entropy with a salted, cryptographically secure hashing algorithm.

Just did a quick benchmark,
uuidv1 from id128          8,585,492 ops/sec
uuidv4 from id128         20,844,616 ops/sec
uuid v4                    8,876,529 ops/sec
uuid v7                      464,017 ops/sec
nanoid                     5,382,926 ops/sec
cuid                         357,886 ops/sec
cuid2                         51,715 ops/sec
ulid (monotonic)          12,003,564 ops/sec

PS: you are missing an index.d.ts
declare module "@paralleldrive/cuid2" {
  export const createId: Function;
}

A good balance between fast and slow is a feature of secure hashing algorithms. We are very intentionally NOT the fastest. Password hashing algorithms loop the hash thousands of times on purpose just to make them slower. Likewise, slowing down hashing to make it expensive to attack is exactly what provides security for the Bitcoin network.
If it were really cheap to run Cuid2, you could just rent a huge cluster of GPUs for cheap and run distributed collision and timing attacks to defeat the security.
Cuid2 is fast enough to be unnoticeable to users, and much faster than a server round-trip, but too slow to run brute force, timing, or statistical analysis attacks to guess existing ids or recover entropy from a hash.
If you want to generate lots of ids during render animation frames (weird but ok I guess?) use ulid. If you want to build secure software that still performs really well on the RAIL performance model, use Cuid2.

Right, somehow I was expecting this comment... ;-)
We all know that the more salt iterations, the better hash
"balance" unfortunately puts cuid2 on the far edge.
This finding tend to confirm my assumptions - there is no magic.

it's performance vs security
an id itself is nothing in view of a proper security architecture.

In the end it's all about finding the proper tradeoffs.

Knowing the facts is a good start. Knowing which facts are important is even better.
I’ve been building software all my life, and I’ve never seen a situation where the generation of globally unique ids was my bottleneck. Usually, in situations where speed is the most important thing, I don’t actually need global uniqueness, and a simple incrementing counter (count++) is good enough and would dramatically out-perform all of the above options.
If I encountered a situation where id generation was a bottleneck, I’d ask myself some questions:
“Does knowing the sequence or exact timestamp violate user privacy or pose a security threat?” Yes? Nanoid
“Is local, single machine uniqueness good enough?” Incrementing counter.
No? Ulid

Really fascinating conversation, thanks for all the back-and-forth. I arrived here after I saw that the CUIDv2 wasn't time-sorted like CUIDv1 was; so I was coming in here to ask about that. And, it seems that you have both covered parts of this in great depth (essentially that leaking time-of-creation is in-and-of-itself a security issue). My main concern over this was that if I'm going to use CUIDv2 as a the primary-key in a database, then the clustered index will have to be re-jiggered every time a new record is added. I've never had a non-increasing key before, so I was concerned over the performance of the index maintenance. In your experience, is this index overhead essentially not a problem?

@bennadel
It is not recommended to use a non-sortable value for a clustered index (primary key) as the data in that index is assumed to be sortable. You can do it and some people do (especially with UUIDv4) but it will have some side-effects and increase maintenance.
For our table designs (SQL Server) we usually stick to INT or BIGINT for the Primary Key and store the auxiliary identifier in a nonclustered index or nonclustered columnstore index.

@xaevik in the context of CUIDv2, however, I am not sure that it would be possible to have a sortable clustered index since, it seems, from what I understand, that the "distribution" of key generation across devices is part of the requirement.

@bennadel you're correct, it wouldn't which is why I mentioned that it was not recommended to use it for a primary key due to the very nature of how it is generated. Hence why for us we keep the primary key a sortable int or bigint value and the real identifier as a separate column with a nonclustered index.

@xaevik so, what you're saying is basically that you generate the CUID as the thing the "user sees", but then use an INT/BIGINT behind the scenes when actually doing database IO?

@bennadel correct, under this pattern the primary key is the identifier for the row, not the entity in question. A quick example would be this:

column
type
is_primary

row_id
int
X

entity_id
varchar(32)

created_at
int

So row_id would be treated as the primary key which is a clustered index. It is strictly used internally for maintaining any foreign key relationships, sorting and join operations. Then the CUIDv2 value would be stored under entity_id where then you can attach a nonclustered index. You also have the added benefit of not needing a unique index for entity_id as well.
You could (if using SQL Server) also create a nonclustered columnstore index instead of a nonclustered index which contains entity_id and created_at if you wish to tie those two datapoints together.
Tables should have as few indexes as possible and should allow for natural sorting (e.g,. the primary key). Any business logic sorting should be done through code rather than through the database. The only exceptions to that would be pagination where server-side sorting is required.

@xaevik that makes sense to me (though it seems like entity_id could have a UNIQUE index as well, without any harm (unless you're saying that has a performance impact).

@bennadel I would say that the creation of a unique index is highly subjective to the implementation. There is a performance impact because each time a record is created, hard-deleted, or updated that constraint must be checked and if the dataset is large, it will slow down.
Since CUIDv2 supports variable length creation, the smaller the value, the greater chance of collision. I would say (and @ericelliott  can correct me if I'm wrong) you could forgo needing a unique index so long as you are enforcing that all generated CUIDv2 values in a specific table are of all the same length and are between 24 (the default) and 32 characters.
If you plan on being variable or generating smaller (say between 4 to 6 characters) values, then a unique index may be advised so long as you understand the performance impact should the dataset comprise of 10s of thousands or 100s of thousands of rows.

Here's how I have done it for the last x years:

Use a cloud-native db solution. Stop trying to wrangle your own SQL servers.
All cloud-native solutions I know of have their own internal keys for things like indexing and table joins. Never use or expose them in your actual application. Use your Cuid2 instead.
All cloud-native solutions I know of allow you to create indexes for any field you want in your table (sometimes up to 3-5 such indexes). Add them if you need to improve performance for lookups by that field.

@ericelliott thanks for sharing this. This would make a great blog post that would attract a lot of interest IMO.

I added the following summary of Cuid2 differentiating features to the top of the README:
Cuid2 is:

Secure: It's not feasible to guess the next id, existing valid ids, or learn anything about the referenced data from the id. Cuid2 uses multiple, independent entropy sources and hashes them with a security-audited, NIST-standard cryptographically secure hashing algorithm (Sha3).
Collision resistant: It's extremely unlikely to generate the same id twice (by default, you'd need to generate roughly 4,000,000,000,000,000,000 ids (sqrt(36^(24-1) * 26) = 4.0268498e+18) to reach 50% chance of collision.
Horizontally scalable: Generate ids on multiple machines without coordination.
Offline-compatible: Generate ids without a network connection.
URL and name-friendly: No special characters.
Fast and convenient: No async operations. Won't introduce user-noticeable delays. Less than 5k, gzipped.
But not too fast: If you can hash too quickly you can launch parallel attacks to find duplicates or break entropy-hiding. For unique ids, the fastest runner loses the security race.

@orefalo If you publish your table, please open an issue with a link so we can reference it. I'm leaving this issue open so people can easily find and learn from the discussion.

@orefalo Mistakes in the table:

Number of characters: Cuid2 has variable number of characters, and defaults to 24.
All of the synchronous id-generators should be at least "Fast", but xid, ulid, nanoid, etc. probably all qualify as "Very fast" because they don't do any extra hashing rounds for security (which also hurts their cryptographic security).
On DB performance metrics, all of the non-cryptographically unique standards (including sequential db identifiers) have very high performance cost for distributed dbs because they need to coordinate across the network. All sequential ids have very low single-db instance costs, which is maybe OK for a blog or a non-networked application, but typically unusable for web-scale applications.
"Entropy sources" should probably fall under security.
Collision risks on same unit should be "Very Low" for both Cuid and Cuid2 - default "High" for UUID (can be mitigated by using a CSPRNG, but the assumption that it does is unsafe). Also high for any other random options that don't use CSPRNG and/or multiple entropy sources.
Series, BigSeries, and V4 UUID all require coordination when scaling horizontally.
Series and BigSeries support cursor-based pagination.
V4 UUID has weak cryptographic security (except in certain implementations which use cryptographically secure random - LOTS of them don't).
Cuid v1 has low id predictability. We upgraded to CSRNG years ago after reports that users could deterministically predict random numbers with the broken RNG in Chrome. Even with the old version, generation would need to happen on the same machine in a tight loop to predict another valid id feasibly, and there were zero reports of that ever happening, even though it was one of the most popular id generators for JavaScript, and used as default ids in some database systems.
Auto-incrementing db ids are not "simple" to scale. (See "benefits"). In fact, under "Shortcomings", I'd mention that they're hard to scale and migrate.

Thanks for all these comments,
I am updating the table, will publish and convert to mark down to ease pull requests moving forward. Will also flip the axis, to help column sort.
@ericelliott, I believe there is still a discrepancy in the CUID2 column. "BIT distribution" shows how bits are layout at rest.  cuid2 is 124bit long, 119 of those are used for randomness (via a multitude of hash...etc), what about the remaining 5 bits?

The hash portion is 23 characters by default (119 bits). The initial character is also a source of random entropy, but is not part of the hash entropy. It is 6 bits. (That’s the “+6” part of the random entropy). The entire id requires 125 bits to represent. 125 - 119 = 6. Sorry if I made that confusing.

Sorry if I was unclear,
"a little diagram is easier than a long speech" Napoleon.
For instance:

or

Which for cuid2 translates into,
6 bits for  "Machine Id / Process Id / Prefix Id"
119 for entropy (aka randomness)
I am aware that the comparison table is not perfect as it is:  beyond the size of the ids and their bit particularities lies the algorithm used to generate them.
That's something I am not quite capturing at the moment, as it's really implementation specific.
Getting there, will share the updated table soon,

@ericelliott I much appreciate the deep-dive on database performance. As much as I love to write SQL, I don't know all that much about the low-level database mechanics, so I just try to piece together what I've been told over the years. Sounds like some of my concerns are not longer relevant on modern implementations of database software.

@orefalo Yes - I would call that "id layout", maybe, and those diagrams are way better than text. What did you use to draw those diagrams? They're perfect. 💯
If you're going to do it that way, adding entropy source bits in the security section would be good, to avoid people getting the wrong idea about security from the id layout.

@orefalo @bennadel I added a detailed comparison section to the documentation, including a brief discussion on the issue of K-Sortable ids. Please read it and let me know your thoughts.

I take no credits, I used my best friend google to find the gfx above, not planning to go that far ;-/
Markdown won't make it, the data is too wide.
Here is the latest version https://docs.google.com/spreadsheets/d/1ZsXBH0z7GOJv3N69QbEDKBZt8IeE0CfRI9vhihV4teo/edit?usp=sharing
I also became more knowledgeable about pseudo random generators. Turns out it's a platform/implementation concern -
On Linux for instance, the kernel gathers noisy data from various devices and transfers them to an internal pool of entropy.
In that context... why is cuid2 more secure, than say a nanoid which uses /opt/random?
I read your statement, I like the intro, I agree with some, disagree on many - mainly because it's open statements with no facts (trying to help).

I am not aware of any breakthrough in B-tree indexing: distributed or not, the problem remains: db indexes perform better on ordered data. I don't see how pushing the problem to a cloud vendor can be used as an argument.
Regarding k-order, I am used to seeing application queries such as :

get last orders
get last payments
get last interactions
get changes of the day for backup
....etc
We even use time to bound query results and optimize IO.

All of the above are time bounded, aren't they?
I agree with you - encoding a date in an identifier implicitly makes it less secure because it reduces the entropy size... for the benefit of speeding indexes. But wait... adding an indexed creation_date column to a DB also has a cost. Especially in distributed cloud db → compute, storage, io, network calls.
The conclusion is that picking a GUID is heavily dependent on context and NFR. It is not a one size fit all. (IMHO!)
Cheers,

Cuid2 is more secure than the web crypto API because it does not need to trust the random entropy source (which is not guaranteed to be good or have enough entropy), not does it need to trust the browser’s choice of hashing algorithm (which has historically been proven not cryptographically secure in Chronium and took 3 years to fox after being reported).
We don’t need to trust those things because we supply our own uncorrelated entropy, mixing it with browser random entropy (moving to cryptographically secure rng as well, very soon), and we hash with a security audited, NIST standard cryptographically secure hashing algorithm.

I am not aware of any breakthrough in B-tree indexing: distributed or not, the problem remains

That simply is not true.
The breakthrough is not in the indexing algorithm, but the hardware it runs on. Every cloud provider I know of uses solid state machines with RAM-like or actual RAM access times, sometimes on custom server hardware with custom integrated wide-bandwidth, ultra low latency data buses. Even on a billion records in the most fragmented index, you’re looking at 9 hops worst case and sub-millisecond seek times. Concerns about fragmented index performance are from the days of spinning disks. Those days are long gone in cloud infrastructure.
As I mentioned in the documentation, sequential ids can even cause real performance degredation because sequential inserts cause the btree to unbalance quickly, causing hotspots and rebalancing churn.
Those same hotspots can also cause concurrency locks that slow down writes.
Further, NONE of the use-cases you mentioned should be done by id. Ids should be treated as opaque values by applications. What you’relooking for is createdAt.

@orefalo The sheets link you shared is private.

I posted a Twitter thread with links to a few exploits of the vulnerabilities that motivated Cuid2.

@ericelliott Thanks for adding the section about sorting, I think it makes sense. The only hesitation that I have is the emphasis on "cloud native" database solutions. Only because, I wonder how many people aren't actually in that world yet, perhaps running their DB and PHP and Apache all on a single box (for example). Though, I suppose, at that volume, maybe it doesn't much matter anyway.
I just think of all those poor companies that tried to do Microservices simply because that's what Netflix was doing; and ended up failing because they didn't actually have the need / technical expertise / same problems. Not throwing shade at microservices (or CUID) - only saying that people often enter into a solution space without actually understanding whether or not it applies to them.
That said, this is all very cool and I intend to look closer at the implementation 💪

Only because, I wonder how many people aren't actually in that world yet

@bennadel we're in that group, we still run Microsoft SQL Server via PaaS for numerous architectural reasons. That is the reason behind the conversation we had earlier with the row_id design as the primary key which is what cloud architected database solutions do, except that you cannot see it.

@xaevik 🙊 shhhh, I'm in that world too 😉

I just read Eric's article presenting CUID2, and I'm carefully watching how it evolves.
If your app is a blog sitting on a LAMP stack, you're far from needing to bikeshed on ID generation. If you're not horizontally scaling, simply use UUIDv4 and call it a day.
The discussion should assume horizontal scalability is something you care about (although I don't see any drawbacks to using CUID2 anyway, it's even prettier and URL-safe). A few points already made:

Timestamps as a source of entropy end up leaking information unnecessarily;
Any index overhead is negligible compared to the benefit of safely generating your ID directly from your client application, such as a SPA or mobile app. This is faster than any network round-trip, period.
Last but not least -- and that's the research I'm taking as homework for the next week -- the Web Crypto API is not safe enough yet. That is circumvented by the multilayer approach CUID2 uses. @ericelliott, please correct me if I got this part wrong.

@bennadel unless you're running it on a 12 year old machine in Grandma’s closet, chances are really good you’re using solid state drives and your hosting costs are in the pennies-per-gigabyte range, and you’re also not dealing with enough rows for sparse rows to be problematic.
If you’re too small to scale horizontally, random ids won’t be a problem. If you’re big enough to scale horizontally, random ids will make it easier. Either way, in 2023, just use a random id format, and it should be secure by default for the same reason you wouldn’t put up an HTTP server in 2023.
It’s safe to assume:

Even the slowest path will give you good performance.
Auto-increment is not secure
Sequential ids are not secure
You’re not going to get in trouble using Cuid2

Relying on the DB for collision resistance won’t be a good plan because:

It’s not future proof - if you ever DO need to grow - scale horizontally, or upgrade your database - you’ll need to change ids - are you going to lose all your table join references and have a hard time migrating? Or you can use Cuid2 and you won’t need to change them.
It’s slower for clients - you’ll have to wait for server round-trips to create records. That means every insert has a perceptible ~100ms+ delay. Throw in an extra 100 - 200ms if the client is in China and the server is in New York.
Your app will not function offline. No new record creation without complex client-side juggling, introducing lots of complexity, confusion, and potential for bugs.

@ericelliott appreciate the correction on that information, thank you.

@ivanfilhoz

If you're not horizontally scaling, simply use UUIDv4 and call it a day.

No. UUIDv4 implementations often use untrustworthy random sources. You should consider it insecure in favor of a format that was  implemented from the ground up with cryptographic security in mind. Just use Cuid2.

Public link.  ;-)
https://docs.google.com/spreadsheets/d/1ZsXBH0z7GOJv3N69QbEDKBZt8IeE0CfRI9vhihV4teo/edit?usp=sharing
sorry about that,

Closing because we have added a comparison section to the documentation.
@orefalo Ping here if you publish your chart publicly, write a blog post, etc.

Hi there, I have since published a medium post on the topic.
https://medium.com/@orefalo_66733/globally-unique-identifier-a-fair-comparison-12b114c78ead
I will eventually add new implementations I found in the wild...
Was a great discussion and learned a bunch,
thank you

First, the Vitess documentation points out the security considerations of auto-increment ids, but they are understating the dangers.
The documentation also mentions that a random id solution is a viable alternative. You can think of Cuid2 as a random generator solution that doesn't trust the built in browser random features, and instead uses a cryptographically secure hashing algorithm to hash entropy that has been proven to work at massive production scales for more than a decade.
Interestingly, PlanetScale's API uses NanoId, which is random, not auto-increment. Cuid2 is a good replacement for NanoId in most situations.
As for the JavaScript attempt to figure out the math, what you want is Math.sqrt(36**(4-1)*26) which evaluates to 1101.3882149360415. JavaScript uses the ^ operator to mean bitwise XOR, not exponent.
The default length calculation would be Math.sqrt(36**(24-1)*26) (note: 24-1, not 23-1, which makes a big difference, and the ** operator), and the result of that calculation is<PHONE_NUMBER>824303000  or 4.0268498e+18 as I mention in the documentation.

So comparing those results to the nanoid calculator (removed - & _ from alphabet) nanoid seems to be able to generate way more ids with only 1% of collision probability?

The probabilities you're looking at in the NanoId calculator are not comparable to the birthday paradox calculation we're using here. To do apples-to-apples comparisons between NanoId and Cuid2 entropy, use the formula Math.sqrt(36**l) for NanoId calculations, where l is your desired character length.
It also sounds like you're misinterpreting the results. One of the differences between Cuid2 and NanoId is the character set. NanoId does not guarantee identifier-safe ids by default, meaning you'd need to pass it a custom alphabet to use it in some of the places where Cuids can be safely used (e.g. html classnames), which will change the number of characters you'll need to reach your collision-resistance goals. Since NanoId uses a 64-character alphabet by default, and we use a 36 character alphabet by default, and NanoId doesn't protect against invalid identifier creation, it can store more entropy in fewer characters than Cuid2, but that comes with its own set of tradeoffs.
Keep in mind that both NanoId and Cuid are customizable, which allows you to adjust those calculations for your particular use-case. If you could use NanoId at 8 characters, and you want a Cuid2 with the same or better odds, use Cuid2 with 10 characters instead of 8, which will give you 3x better collision resistance + an identifier-safe id structure + the assurance of using well tested entropy with a cryptographically-secure hashing algorithm (the same one used to secure the Ethereum blockchain). You don't get those assurances with NanoId.

Allow me to provide some context for the benefit of anyone reading this,
Random generators
You have two choices, each with their own risks and benefits, depending on factors such as how many people have reviewed the implementation and how widely it is used.

Trust the cuid2 implementation of randoms.
Trust the runtime implementation of randoms.

The bottom line is to be aware and not to trust blindly.
Alphabet
This is a non-issue because it's customizable.
Nanoid was built to support JavaScript and browsers, so its encoding is optimized for use with URLs.
This is not a bad default choice, in my opinion.
GUID vs. sequences
When dealing with OLTP (Online Transaction Processing) applications, you typically need to focus on security, resilience, scalability, and speed. However, when building an OLAP (Online Analytical Processing) database, the above requirements are less critical.

Sequences are easy to predict and are well indexed.
GUIDs add randomness to minimize the attack surface.

I found this article fascinating: https://brandur.org/nanoglyphs/026-ids.
Once you have considered these factors, the question becomes whether to use ordered GUIDs or random GUIDs.
There are security aspects to consider, such as

entropy (randomness) quality, we mentioned it
dispersion: cuid2 claims to be better in these areas. no evidences.
opacity of the id - if you can live with a "visible" ID generation date or not

There is also the persistent side to consider. Since IDs are typically used as indexes, their randomness can have a negative impact on DB storage and IOs (the Write Ahead Log effect). Eric mentioned that there were "innovations" in data-centers, but please realize that IO and storage do have an impact on cost.
In conclusion

Security always comes at a cost.
Be aware of your specific requirements and choose an implementation accordingly

dispersion: cuid2 claims to be better in these areas. no evidences.

We prove dispersion with multiple kinds of tests with every commit:

20-band histogram
character frequency test
Visual randograms to spot check for patterns (there shouldn't be patterns - should look like random noise)
collision tests - we test id generation in parallel with more than 11 million ids on every commit, in total, we have tested several billion ids for collisions with zero detections

Those tests and evidence are available in the code, and the most recent randogram and histogram are present at the end of the documentation. AFAIK it's the most thorough test suite in the id-gen ecosystem.
As for our entropy sources, they're strengthened versions of the entropy used successfully for more than a decade in Cuid v1, for which there were zero confirmed collision reports in spite of being used in production in products with hundreds of millions of users.
I addressed performance considerations earlier in this thread. TL;DR, it's not a big factor with today's tech.

Morning,
Sorry, I didn't mean to say there were no evidences of cuid2 distributions: there are, and in great details.
There is a lack of evidence that the distribution is better than other implementations.
Put differently, is it a valid selling point?
Eric, please realize I am not trying to criticize your GUID implementation: just trying to provide the proper 360.

entropy sources - fine, inline with my comment above. Not sure if there are metrics which can be used to assess quality. In any case, I would assume Linux would have better quality as it can tap low level random sources.
performance - fine. Indexing a pure random guid translates into IOs and storage, which translates into cost.  This is not limited to cuid2, any random will have the same effect. Therefore, an alternative design pattern consists in using a time based guid to minimize this effect. Of course, just like anything in Architecture - it comes with side effects.

Sincerely,

Both of your assertions are false. In the section on NanoId & Ulid we link to a browser bug that completely invalidated cryptographic security in the "cryptographically secure" random number generator in Chromium, which was a known, open bug between 2015 and 2018 - meaning ALL guids/ulids/nanoids generated during that time are potentially vulnerable to attack. Had they been generated with Cuid2, they would not be. Similarly, there are links in the docs to actual vulnerabilities caused by V4 UUIDs. We don't just show that other id implementations have poor entropy - we link to real examples of exploits in the wild.
As for the assertion that there is "a cost" associated with random ids, while on the surface, obviously theoretically true, is still a very misleading statement which turns out to be false when you consider what happens in real life.
The cost of random indexes - in the age when cloud db storage is pennies per GB and cloud db servers have tens or hundreds of GB memory per host by default - is so tiny as to be completely drowned out by other choices, such as whether or not to optimize your images.
Relative to the large cost of dealing with auto-incrementing ids or worrying about security fallout, the cost slips into positive ROI very easily.
You mention the small cost of random without also mentioning the large cost of the alternatives. That doesn't make sense.

@orefalo I added more context and links in the "why" section of the documentation to make the evidence that Cuid2 is needed more obvious.

Morning Eric,
I read the "why" section and can't disagree.
Yet, there is something I am missing....
Why do you put so much accent on the browser for ID generation?
Channels are insecure, aren't they?
I always generate IDs close to the systems of records, server side.

Authentication -> In an IAM or IDM
Session -> At the edge, typically in the web application
Records id -> in the microservice or the db (depending on)

Now if 99% of ID generation use cases are server side, wouldn't it be better to leverage a CSPRNG who has access to proven sources of entropy? (IO, kernel events..etc)? rather than an opaque implementation coming from the ages of cuid1?
Regarding the "cost" of indexing random UUID in RDBMS, I did an extensive research and couldn't find any scientific assessment.
I mean people talk about the negative impact of the WAL effect, but the articles are operational not comparative.
Maybe there is space to think about a new cuid3 implementation with support for:

an optional variable size timestamp [0-x]
a plugin architecture for entropy generators

Cheers,

Why do you put so much accent on the browser for ID generation?
Channels are insecure, aren't they?

You are right not to trust the client to generate ids that would have system security implications. However, a lot of id generation is safe in the client, assuming all other ids are generated with Cuid2, and assuming that the client is not already compromised by a malicious third party (in which case, that particular user's security already has bigger problems).

Now if 99% of ID generation use cases are server side

This is not a safe assumption, for all the reasons mentioned in the Cuid2 documentation. Some of those requirements include offline operation, which is a non-starter if your id generation relies on server-side access. Remember, there is more than one kind of application in the world, and we still need dependable, distributed, collision-safe, opaque id generation. The use-cases are many. (e.g. most consumer UX productivity applications).
You are right that authentication / session ids usually need to be handled by the service doing the authenticating, but increasingly, that is happening on the client side, too (see decentralized Web3 authentication where the authority is a self-sovereign, client-side wallet).

wouldn't it be better to leverage a CSPRNG who has access to proven sources of entropy? (IO, kernel events..etc)?

No. Where do you think the buggy browser implementations of CSPRNGs got their entropy? We don't completely trust these sources, or we wouldn't need Cuid2 at all. We'd just put a friendly wrapper around  crypto.getRandomValues() and call it a day (and indeed, that's exactly what NanoID does - this is addressed in the documentation).

rather than an opaque implementation coming from the ages of cuid1?

The entropy sources are not opaque, and their reliability as a unique combination of entropy has been proven for more than a decade in some of the largest scale applications in the world, with hundreds of millions of users. Those same sources are so trusted that have been tested in dozens of id generating libraries for many years and proposed in the UUID V6 - V8 specifications.

Maybe there is space to think about a new cuid3 implementation with support for:

an optional variable size timestamp [0-x]

No need. If you don't need the opaque security features of Cuid2, what you need is probably UUID V7 (which is basically a standardized version of Cuid v1).

a plugin architecture for entropy generators

Cuid2 already accepts an entropy generator as a parameter. You can pass in any compliant random() function implementation.
Thanks for your thoughtful questions.

"safe" in terms of security has a different meaning that "safe" in terms of entropy
Listen, this is comment 68 on a closed topic.
While I don't share all of your arguments and positions, I can see where cuid2 fits in.
More importantly, I appreciate the time you took to clarify my doubts and provide references whenever possible. Through this conversation and my own research on GUID, I've learned a lot.
Best regards,

I like facts, I hate open statements..
Here is an article which seems to be confirming my suspicions about fully random UID generators.
https://www.cybertec-postgresql.com/en/unexpected-downsides-of-uuid-keys-in-postgresql/
Enjoy the read,

@orefalo That article is basically reiterating what I said above about monotonically increasing ids. The issue has been well known to me for well over a decade, and inspired the design of the original Cuid. More than a decade of practical use-cases has taught me that it is almost never a real problem in practice, and if and only if it becomes one you can simply designate a different primary key for your database (for table joins/db references) and continue to only expose opaque ids to clients.

You know how much I enjoy arguing with you, right ;-)
I agree with the early points, the article however goes on and provides some interesting insights about the side effects of using UUID as primary keys - namely, page grow which with data locality seems to have a significant negative impact.
On these points, I am not arguing - in the end it's security vs speed.
However, I find the conclusion even more stunning "so if you want to use UUID’s, try to use a sequential variant. UUID v7 is a good option"
Seems like I am not the only one to believe a cuid3 implementation with a optional variable size timestamp [0-x], would solve this long running debate.
As far as using two keys - one external, the other internal ; it doesn't solve the problem as both will need to be indexed to avoid (distributed) table scans.
Sweet dreams

I agree with the early points, the article however goes on and provides some interesting insights about the side effects of using UUID as primary keys - namely, page grow which with data locality seems to have a significant negative impact.

We have different definitions of "significant". I've already explained this several times, but I'll dig into more depth:

The example is manually enumerating 10 million records. This is guaranteed to hit every record in the table every time you run the query. In a production system where that would be a frequent query, you can just cache the count and get sub-millisecond response times on this lookup. Problem solved without messing with IDs or messing with the DB configuration, but read on... there's more.
Typical DB usage does not hit all your records, or even most of your records, but only a few records or a few dozen records in the case of complex queries. Assume your typical query hits 100 records. The UUID V4 version would complete in 0.014 ms - this is not a bottleneck for typical applications and in the last decade, I have never seen this issue cause an actual problem in an app I worked on (I did before SSDs, on 15-year-old-hardware, but we just don't see those problems anymore). As I mentioned before, if you actually DO encounter this issue - just make a monotonic, indexed createdAt field or internal monotonic db primary key for joins that you don't expose to the public, and call it a day. In other words, in the very unlikely case that this is a problem for you, there are good workarounds that don't involve tossing security out the window.
You can fine-tune a DB to optimize for SSD cloud services with hundreds of GB of memory (which is the typical case for scalable cloud situations). Such systems need to tune the db so shared_buffers is ~25% of the total system RAM (which may be dozens or hundreds of GB in a typical cloud DB situation, and app developers are rarely exposed to those parameters at all because services like FaunaDB do it for you). For reference, that is typically enough to store small record data for every user of the internet in the world for less than $100/month. DB admins who are actually worried about these performance implications can compile a specialized build of the db to optimize or even auto-tune the page size to better utilize available RAM and reduce shared_buffers lookup. These optimizations would be good for DB performance whether you use random ids or not.
If you don't have enough data or traffic to be worried about automated horizontal scaling (see the point above), you don't have enough data or traffic to worry about this issue at all, because you will likely not be hitting 10 million+records on a regular basis. Typical primary key lookups (even for random ids) use a b-tree lookup, which is a O(n) operation that completes in sub-millisecond times. In other words, what you're looking at in that comparison is a worst-case lookup - not typical db usage.
If you actually DO have usage that hits tens or hundreds of millions of records per operation, you probably don't want a DB, you want a cloud GPU or TPU cluster.


Set all parameters objects on mc_pos_control to be with the convation

This is small cleanups just to make all the parameters objects to be the same namming convation - the parameter name in CAPITAL_LETTERS
in this model some objects are named with CAPITAL_LETTERS convention where the name on the object is the parameter name, and some wasn't. this PR fix that.
there is no strict convention of that on other models, and what to do about that can be decide to be either make all to CAPITAL_LETTERS or keep current state, or any other sugesttions.

This fix should actually be in the other direction. i.e change caps to _snake_case.

I have chanhed to _snake_case.
Do you like it more?
Do you want me to continue to change more to this convation (I don't want to step on any toes of anyone that puts the CAPITAL_LETTERS....)

Hi @BazookaJoe1900 , This PR looks like a move in the right direction to me.
I see other instances in StraightLine.hpp and PositionControl.hpp, but I think I'd advise to submit work in those as their own PRs.  This PR is good standalone work that is focused on its intent very well. Nice work!

Hi @BazookaJoe1900 , This PR looks like a move in the right direction to me.
I see other instances in StraightLine.hpp and PositionControl.hpp, but I think I'd advise to submit work in those as their own PRs. This PR is good standalone work that is focused on its intent very well. Nice work!

there are much more with the CAPITAL_LETTERS convention. (CollisionPrevention.hpp, FlightTaskAutoLineSmoothVel.hpp, FlightTaskAutoMapper.hpp.... and on and on)
before I will do more work about that, please confirm the _snake_case convantion and merge this pr...

Hi @BazookaJoe1900 , the developer guide shows example usage in the manner you have now pushed up, so I think you're in great shape!

Hi, after a discussion with @bkueng, @MaEtUgR and @Stifael a few weeks ago, we also decided that the snake_case would be more appropriate. We also wanted to add a param_ prefix to the variables in order to be obvious to the reader that this is a parameter and to work nicely with the autocompletion tools.
I started to write a quick bash script that makes the conversion for the whole code base and I just have to cover a few corner cases such as the 2-lines declarations in ecl and it should be good to go.
Did you do a script for those changes? Maybe we should combine them.
@dagar After the changes, shoul we also have a script in CI that checks the correct naming of a parameter to enforce a specific convention?

@bresch, I haven't done that by script (yet) because I first wanted to see the comunity approch to the namming convation.
If you say that you done the script allready, it seems that you done the work... should I close this pr?

Okay, just leave it open with the WIP tag, I'll close it and link it to my PR when I'm done :)

@bresch I removed the [WIP] tag. I think that this pr is ready to merge, later you can do your script work about the parameters over it

@bresch , I like the design pattern/standardization effort of this PR. I hope it can move forward even though you are progressing the work in parallel.

@MaEtUgR can you review and approve?

closed in favor of #11686

@BazookaJoe1900 Good start! I would have done it similarly. Sry had a little downtime. The automated way to enforces the pattern in #11686 is even more future proof.


InvalidPathException

i am encountering this issue where it keeps spamming this code in the therminall, how can i resolve this?
jan. 14, 2024 5:23:50 PM it.auties.whatsapp.util.DefaultControllerSerializer onError SEVERE: java.util.concurrent.CompletionException: java.nio.file.InvalidPathException: Illegal char <:> at index 17<EMAIL_ADDRESS>

i am encountering this issue where it keeps spamming this code in the therminall, how can i resolve this?
`jan. 14, 2024 5:23:50 PM it.auties.whatsapp.util.DefaultControllerSerializer onError
SEVERE: java.util.concurrent.CompletionException: java.nio.file.InvalidPathException: Illegal char <:> at index 17<EMAIL_ADDRESS>
Update to cobalt 0.0.2 and let me know if the issue is fixed


cookie named "_fz_http_key" exceeds maximum size of 4096 bytes

Request: GET /auth/oidc/gate/callback/?code=fa9139cf9de508e0e16f&state=F7cC96pgwIjSuIIb
** (exit) an exception was raised:
** (Protocol.UndefinedError) protocol Phoenix.HTML.Safe not implemented for %Plug.Conn.CookieOverflowError{message: "cookie named "_fz_http_key" exceeds maximum size of 4096 bytes"} of type Plug.Conn.CookieOverflowError (a struct). This protocol is implemented for the following type(s): Atom, BitString, Date, DateTime, Decimal, Float, FzHttp.Types.IPPort, Integer, List, NaiveDateTime, Phoenix.HTML.Form, Phoenix.LiveComponent.CID, Phoenix.LiveView.Component, Phoenix.LiveView.Comprehension, Phoenix.LiveView.JS, Phoenix.LiveView.Rendered, Postgrex.INET, Time, Tuple, URI
(phoenix_html 3.3.1) lib/phoenix_html/safe.ex:1: Phoenix.HTML.Safe.impl_for!/1
(phoenix_html 3.3.1) lib/phoenix_html/safe.ex:15: Phoenix.HTML.Safe.to_iodata/1
(phoenix 1.7.2) lib/phoenix/controller.ex:1010: anonymous fn/5 in Phoenix.Controller.template_render_to_iodata/4
(telemetry 1.2.1) /app/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3
(phoenix 1.7.2) lib/phoenix/controller.ex:976: Phoenix.Controller.render_and_send/4
(phoenix 1.7.2) lib/phoenix/endpoint/render_errors.ex:84: Phoenix.Endpoint.RenderErrors.instrument_render_and_send/5
(phoenix 1.7.2) lib/phoenix/endpoint/render_errors.ex:62: Phoenix.Endpoint.RenderErrors.catch/5
(plug_cowboy 2.6.1) lib/plug/cowboy/handler.ex:11: Plug.Cowboy.Handler.init/2

Hi there, you'll need to use the PHOENIX_HTTP_PROTOCOL_OPTIONS env var to increase the max header size from 4096. See https://www.firezone.dev/docs/reference/env-vars/.
tl;dr: try this:
PHOENIX_HTTP_PROTOCOL_OPTIONS={"max_header_value_length":8192}

Thanks for your reply, but I have added the PHOENIX_HTTP_PROTOCOL_OPTIONS you gave to the.env file and the same exception will still occur.

I'm not sure if using http2 is relevant.

Can you help me with another question first?
https://github.com/firezone/firezone/issues/1621

I tried to enable SSO with zitadel,  but it also occurred the same exception.

Sorry, I've got no fix to provide, but an insight: Browsers "only" allow 4096 Byte cookies to be stored.

Hello,
I have exactly the same issue; it is no longer possible to connect to the interface
It's a rather blocking bug. We are using Keycloak for OIDC authentication.

Hmmm, we haven't been able to reproduce this -- there must be some parameter(s) being saved in the session that happens to be very long. Could you @Kuppit @T-baby report the number of characters you have for the following:

Affected user's email address
OIDC config ID
OIDC discovery document URI
OIDC redirect URI

Affected user's email address: 24 characters
OIDC config ID: 8 characters
OIDC discovery document URI: 69 characters
OIDC redirect URI: 51 characters

Yes it happens with each test, with different browsers, different users and even by deleting the cookies and local storage of the 2 sites.

@Kuppit We store the OIDC ID token from the successful authentication in the cookie. Have you recently added fields/metadata to this token in Keycloak or changed any of the settings there? Are you able to post a sample ID token here so we can check its length?
If that's the culprit, the short-term workaround would be to minimize the size of your ID token in Keycloak. The longer term fix won't be trivial in Firezone -- we'll need to resort to persisting these tokens in the DB since they can grow potentially unbounded.
cc @AndrewDryga for context since you're working on auth for 1.0

Indeed, upon reflection, we added several roles to the users for another tool, and we send the roles by default. I have just made the roles optional for sending, and the connection to Firezone is working again.
Thank you


Adding Neelam College

Hi, Adding my college Neelam College of Engineering from India.
http://neelamcollege.com/

@raviprakashneelamcollege Pull request merged. Thank you!


[Button] Remove label span

Preview
https://deploy-preview-26666--material-ui.netlify.app/components/buttons/#main-content
Changes Summary

remove label classes and element from Button and LoadingButton
fix LoadingButton position: center styling because it rely on label before.

History
button > span > children exists as a workaround for flex container bug on button BUT not anymore, so this PR remove the span.

✅ Chrome 83+ (latest 90)
✅ Safari 11+ (latest 14)
✅ Edge
✅ Firefox 63 (latest 89)

https://github.com/philipwalton/flexbugs#flexbug-9

[x] I have followed (at least) the PR section of the contributing guide.

I believe that merging these changes means closing #26412. AFAIK, the solution was depending on the span, cc @eps1lon to confirm or not.

Regarding the transition when centered, the only solution I could find is adding the span back for the loading label. So it can also support more complex children

In my opinion, adding the span back is a no go as we learned from other components that the structure should not surprise people.
https://user-images.githubusercontent.com/18292247/121457129-61bc2480-c9d1-11eb-8c44-327fc160fec7.mov
Here, I think it is better. remove color from transition only for loadingPosition: 'center'. This way it does not affect other position.

There are two issues with the current animation: 1. the icons disappear instantly (which is quite visible), 2. the border-radius color isn't in sync with the text color. It doesn't feel as smooth as on https://github.com/mui-org/material-ui/pull/26666#pullrequestreview-680214403. Both fixed if we add a <span>.
Another issue of not using a span: this do no longer work:
      <LoadingButton
        onClick={handleClick}
        loading={loading}
        loadingIndicator="Loading..."
        variant="outlined"
      >
        Notifications <Badge>9</Badge>
      </LoadingButton>

PR: https://codesandbox.io/s/loadingbuttonstransition-material-demo-forked-0116b?file=/demo.js:1232-1443

HEAD: https://codesandbox.io/s/zch00?file=/demo.tsx

is a no go as we learned from other components that the structure should not surprise people

Agree with this downside. It's definitely a tradeoff

@siriwatknp I have updated #20012 to check [ ] and link this PR. If you could do it for the other efforts, it will be 👌.


fix(curriculum): add review lessons for arrays in pyramid project

Checklist:

[x] I have read and followed the contribution guidelines.
[x] I have read and followed the how to open a pull request guide.
[x] My pull request targets the main branch of freeCodeCamp.
[x] I have tested these changes either locally on my machine, or GitPod.

Closes #54961

please be careful in resolving the conflicts!

I am really sorry for the trouble! While removing some commits that was made while removing the merge conflicts and force pushing, i accidentally lost all of my changes and this has closed my PR. I will create a new PR with those changes.


[cleanup] Follow-ups from initial mypy PR

Stack from ghstack (oldest at bottom):

-> #25

Summary:
A couple follow-ups from #22:

Add TODO for type checking FLAVA code
Add a couple other type hints missed by mypy in clip_text_encoder (due to https://mypy.readthedocs.io/en/stable/common_issues.html#no-errors-reported-for-obviously-wrong-code)
Some changes to weighted_embedding_encoder to silence mypy errors

Test plan:
$ python -m pytest -v
...
====================================================================================== short test summary info =======================================================================================
FAILED test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_multi_transform - requests.exceptions.MissingSchema: Invalid URL '/data/home/ebs/torchmultimodal/torchmultimodal/test/a...
FAILED test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_single_transform - requests.exceptions.MissingSchema: Invalid URL '/data/home/ebs/torchmultimodal/torchmultimodal/test/...
================================================================== 2 failed, 48 passed, 3 skipped, 26 warnings in 76.48s (0:01:16) ===================================================================

(Failures are pre-existing and will be fixed with a separate commit)
$ mypy
Success: no issues found in 29 source files

@ebsmothers has imported this pull request.  If you are a Facebook employee, you can view this diff on Phabricator.


No refresh after mocking an item

When mocking an item in the network tab it doesn't refresh when moving to the scenarios tab

Fix in #124


Add security group name

Hello !
This PR replicates the already existing fix to the terraform011 branch.
Thanks.

hi @akofman - we are no longer maintaining terraform 0.11 unfortunately


jekyll new -> invalid option: --future

Hi,
I am trying to use the docker image jekyll/jekyll:pages to bootstrap a jekyll GitHub pages site.
When I run :
 docker run --rm -ti --label=jekyll --volume=$(pwd):/srv/jekyll -p 4000:4000 jekyll/jekyll:pages jekyll new . --force

it returns :
jekyll 2.5.3 | Error:  Whoops, we can't understand your command.
jekyll 2.5.3 | Error:  invalid option: --future
jekyll 2.5.3 | Error:  Run your command again with the --help switch to see available options.

Can't find what goes wrong...I searched for --future invalid option for jekyll, but nothing found but a reference to a bug in 2.x version with --future flag : http://jekyllrb.com/docs/upgrading/2-to-3/
What's wrong ? Thanks.
Clément

You've done nothing wrong, I have a bug in my shim, I'll release a fix for you after I test another bug discovered while working on this bug.

Even  though the commit was never made my build server pushed out a fix last night for you, the reason a commit was never made as there were dozens of other changes in the works to make this repository more robust for people who need more flexibility.
Either way I'm gonna hold this open until I make that commit but a push was made last night.

Perfect ! It works. Thanks


upgrade python-hcl2 to version 3.0.3

This release fixes a bug I encountered and fixed:
https://github.com/amplify-education/python-hcl2/pull/87

Current dependencies on/for this PR:

main

PR #1   (this PR)

This comment was auto-generated by Graphite.


feat(custom-webpack): Export function in webpack configuration

Add the possibility to export a function in webpack custom configuration.
The function takes the angular build option as parameter.
The need for this feature was mentioned in the issue : #135
exemple :
module.exports = (buildOptions) => ({
    devtool: buildOptions.sourceMap ? 'source-map' : false
});

Thanks for the contribution!
Would you mind adding UTs that cover the functionality you've added (the function/object thing)?

Closed by #258


Close button on searchbar hides entered text

As you can see on the screeshot below, when the text is long enough to fill the input field, it goes under the close button. It would be better if the input stops before the close button.

Use this codepen to illustrate your problem.

Can you post it in a codepen and I'll try to help.

Here is the codepen : http://codepen.io/anon/pen/LEwyQJ
Thanks for your help.

@Dynodzzo I found a simple solution for you bud, here is my solution. I hope this works for you? I used the maxlength attribute for the search field. @Dogfalo is the an alternative to this at your end?

Yeah i thought of that but it isn't really a solution as it limits the input. It would be nice to be able to enter a many characters as I want and limiting the input width.

Another solution could be
.input-field input[type=search] {
    width: calc(100% - 8rem);
}
.my-input {
    padding-right: 56px;
}


Post release automated changes for keyvault releases

Post release automated changes for azure-keyvault-admin

Two things I had to manually do:

I had to run node ./eng/tools/versioning/increment.js --artifact-name azure-keyvault-secrets --repo-root . for keyvault secrets because the perf-test was still pinned to the older version. IMO the perf tests is one place where you don't want to pin to a specific version - it makes incrementing harder, and we'd want to run it against the latest pre-release anyway
Post release automated steps generated samples for Key Rotation which is in beta, so I had to manually rebase and edit those files out of the commit


test: polish

remove a test case "throws errors" which seems duplicated
fix some typos and remove some comments which seem out-of-date
test the getter sum as well in the test case "keeps getters reactive"

Codecov Report

Merging #1280 (1af6d20) into v2 (e5843ac) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##               v2    #1280   +/-   ##
=======================================
  Coverage   97.58%   97.58%
=======================================
  Files           9        9
  Lines         414      414
  Branches      108      108
=======================================
  Hits          404      404
  Misses          5        5
  Partials        5        5

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5843ac...1af6d20. Read the comment docs.


Add "file an issue" button to the "Rating and Review" section

Sometimes, issues are filed as VS Marketplace reviews that are supposed to be GH issues, which lowers the extension ratings. Even though the "Issues" link exists in the righthand column of the extension page, it doesn't seem as discoverable. Would it be possible to make filing an issue more prominent in the "Rating and Review" section by adding it as a button? Something like:

Related: https://github.com/microsoft/vscode-jupyter-internal/issues/262

Similarly, other app stores have links for support/filing issues near where users write a review:
Apple App Store

Chrome

Steam
In some cases if you write a negative review, some apps asks the user if they'd like to contact support first:

This is something we have on our backlog; it will be a while we get to it :)

We have implemented the minor enhancements on marketplace for more details refer to the discussion thread, https://github.com/microsoft/vscode-discussions/discussions/329.


How do you create projects?

Hi I have a question, how do you create a project?
Thanks in advance!

Duplicate of #79.
You have to add project: true to the post header.  Post your posts in the usual place...no special folders named "project" or anything like that.


Force fact gathering even when using --limit

Having this task run on localhost meant that a --limit would limit out
localhost.
Change-Id: I6bd871f554ea5a46e38ed38a77741935d238684e

retest

👍


Add ascii art to cli

Thanks to @bongobongoland here is a nice ascii art for the cli

what happened to the bite marks? :)

Oh lol I didn't knew what they are so I removed them

It took me quite some time to arrange them. Every bite mark was unique :)

Really cool logo!


feat: RemoveEffect interactive example

Description
Added interactive example for RemoveEffect.
Added a simple way to re-add the flower to the example by a simple tap like you can reset the other examples.

Checklist

[x] The title of my PR starts with a Conventional Commit prefix (fix:, feat:, docs: etc).
[x] I have followed the Contributor Guide when preparing my PR.
[ ] I have updated/added tests for ALL new/updated/fixed functionality.
[x] I have updated/added relevant documentation in docs and added dartdoc comments with ///.
[x] I have updated/added relevant examples in examples or docs.

Breaking Change?

[ ] Yes, this PR is a breaking change.
[x] No, this PR is not a breaking change.

Related Issues

Closes #1957

Text not working at the moment, will look more tomorrow.

Should work properly now :)


Switch to temporarily un-hide all hidden giveaways

Is your enhancement request related to a problem? Please describe.
When on any giveaway listing page there should be a switch in e.g. the search filters box that allows us to temporarily unhide all hidden giveaways, so they are displayed between all the other giveaways without having their "hidden" tag removed.

This is not possible. SteamGifts does not include the hidden giveaways in the page, You have to use the SteamGifts option for this.

I'm not talking about all giveaway for specific games that are hidden on Steamgifts but about the specific giveaways hidden by esgst.

Ah ok, you mentioned a hidden tag, so I thought you were talking about Game Categories. Single giveaways hidden don't have tags.

"Tag" I meant just to describe the fact that they are hidden. So that they don't get generally un-hidden just because they are shown one.
As said, a temporary un-hide switch for hidden specific giveaways is what I meant.

Where can I find the requested feature in the settings? Seems that you have added something but I can't find it.
I would have expect it under 2.21.1, as it temporarily undoes the Single Filters, but I also can't find it anywhere else.

It's enabled by default. The button is at the footer. I know it's not a
practical place, but it was the easiest to implement. I'll probably move it
to the page heading when I can.
On Mon, May 13, 2019, 08:38 Eiion<EMAIL_ADDRESS>wrote:

Where can I find the requested feature in the settings? Seems that you
have added something but I can't find it.
I would have expect it under 2.21.1, as it temporarily undoes the Single
Filters, but I also can't find it anywhere else.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/gsrafael01/ESGST/issues/1273#issuecomment-491787270,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGCT34LCHA2LBN2S34H3EE3PVFHKBANCNFSM4HEG53SQ
.

There's no button - only things down there are: "Powered by Steam", Archive, Stats, Roles, Users, Steam Group, Chat, Privacy Policy and Terms of Service.

Sorry, wrong wording. It's at the end of the page, where you can see "Displaying 1 to 50 (5 hidden by filters - unhide)" or something like that.

That works.
I guess a good place to put it in the future would be either in the filter box or the left main page heading.

It will be in the main page heading in the next version (you'll have to enable the option under "Single Filters").


Use ABAP CDS Syntax Highlighter for .acds files

With https://github.com/github/linguist/pull/4614 Github understands "asddls" files und uses the syntax coloring provided by https://github.com/FreHu/abap-cds-grammar
I tried to extend this to also the "acds" file extension but failed due to too low usage of that file extension.
But we could add an override for this repo so that acds files just use the existing coloring for "asddls".
This is explained in: https://github.com/github/linguist/blob/master/docs/overrides.md

ref https://github.com/github/linguist/pull/5954

Coloring is not perfect. See https://github.com/FreHu/abap-cds-grammar/issues/1


docs: split guide

@mubaidr added your composable here https://deploy-preview-83--vuetify-nuxt-module.netlify.app/guide/server-side-rendering.html#vuetify-themes , any other suggestion (I'm also adding some reference for Vuetify useDisplay)?


import styling rule

There doesn't appear to be a way to set a style on naming imported modules similar to how you can set the style of variable names. This may be an extension of an existing rule instead of an all together new rule.
If I have the following two imports, I'd like to check to see if they are in camelCase for example.
import * as Q from "q";
import * as Path from "path";

I'd expect the following "Foo" likely not to be affected by the naming convention.
import { Foo } from "foo";

In considering this, I referenced variable-name rule as a comparison.
"variable-name": [true, "ban-keywords", "check-format", "allow-leading-underscore"],

Isn't the format of module names more semantic than syntactic though?
For example:
import * as Cat from "cat";
import * as path from "path";

const myCat = new Cat();
const somePath = path.resolve("some/path");

In other words, wouldn't you name your imported modules based off of how they are used?

Potentially, though I see value in being able to distinguish a given item was imported instead of created locally. Similar to the value you get from a naming convention distinguishing private members/functions versus public ones (which you might find in a checkstyle rule configuration).

@trodi this is covered by tslint-microsoft-contrib's import-name rule. This issue is really a subset of #1142 for determining which rules should be ported over.


Question - Scattered Plot

I am looking for a control to replace and old implementation that we cannot longer support.
The control displays scattered points.Ideally we want the 'points' to be a string, such as "p1", etc.. but we are finding out that all of the implementations using WPF only support shapes, circles, etc..
We can live without having the strings, and instead, perhaps displaying the x,y,z coordinate on a point click event.
So my questions, is there a .NET sample somewhere for creating a scatter plot using nzy3d, and are there click events defined that we can use...?
Thanks in advance for your help.

ScatterPlot where not supported previously
I just commited the code for it right now.
See issue #9 where I attached a sample of displaying points.
No labels so far, and no click event unfortunately.


BeatLeader caused crash after every map until I logged it in

Just an FYI:  If you install the BeatLeader qmod but don't login to a BeatLeader account it'll cause the game to crash pretty reliably after completing any given (modded) map.  Seems like the type of bug that'd be hard for a developer to notice since you're always going to be logging in as yourself and... why would you play a song to completion without logging in first? :smile:

Fixed👍


M3-3088 Change: Use new Image select for rebuild flow

Description
POC for how this should work; the existing distro/version select pair doesn't work great for a full list of images so we did a tab-separated flow.
Type of Change

Breaking change ('break', 'deprecate')

Note to Reviewers

Closing in favor of #5431


[BEAM-7999] Fix BigQuery timestamp handling for Schema Aware PCollection

This fixes the handling of timestamps when reading for BigQuery. As it
was assumed BigQuery always outputs a Double, reading a table directly
outputs a textual version in the JSON. This is now handled. Also the
from Beam Row also outputs the BigQuery textual version, iso the ISO
standard representation, this makes the implementation symmetric.
Tested on DirectRunner and DataflowRunner

Abandoning... as rebase seems to be incorrect


discord_rpc was not found, does it need to be installed?

Describe your problem here.
every time i type lime test windows to compile, i get that error. i have everything installed, but it's just not working. is there a solution, or do i just gotta wait it out?

Are you modding a build from source or with Lua?
Source
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
all i've done is add a new character so far, to test if it works.

did u followed all FNF's original repo? U need to install discord_rpc with the code haxelib git discord_rpc https://github.com/Aidan63/linc_discord-rpc

i have. the exact command will not work. it says it's installing and already installed, but when i run lime test windows, i get the error
Sent from my iPhone

On Dec 1, 2021, at 15:54, HiroMizukiOFC @.***> wrote:

did u followed all FNF's original repo? U need to install discord_rpc with the code haxelib git discord_rpc https://github.com/Aidan63/linc_discord-rpc
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.

i have. the exact command will not work. it says it's installing and already installed, but when i run lime test windows, i get the error
…
Sent from my iPhone
On Dec 1, 2021, at 15:54, HiroMizukiOFC @.***> wrote:  did u followed all FNF's original repo? U need to install discord_rpc with the code haxelib git discord_rpc https://github.com/Aidan63/linc_discord-rpc — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

ok, i think you will need to uninstall and install it again
1 - Use the command haxelib remove discord_rpc
2 - The same command i have written before, haxelib git discord_rpc https://github.com/Aidan63/linc_discord-rpc

i have. the exact command will not work. it says it's installing and already installed, but when i run lime test windows, i get the error
…
Sent from my iPhone
On Dec 1, 2021, at 15:54, HiroMizukiOFC @.***> wrote:  did u followed all FNF's original repo? U need to install discord_rpc with the code haxelib git discord_rpc https://github.com/Aidan63/linc_discord-rpc — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

ok, i think you will need to uninstall and install it again
1 - Use the command haxelib remove discord_rpc 2 - The same command i have written before, haxelib git discord_rpc https://github.com/Aidan63/linc_discord-rpc

this is the new error. source/ModsMenuState.hx:388: characters 6-28 : {+ length : Int } should be String
source/ModsMenuState.hx:392: characters 6-42 : {+ length : Int } should be String
source/ModsMenuState.hx:396: characters 36-47 : Array access is not allowed on {+ length : Int }
source/ModsMenuState.hx:396: characters 36-47 : For function argument 'Red'
what do i do now

now, read the issue #1524, people have fully answered here

now, read the issue #1524, people have fully answered here

i'll try it once i get back home. i'm getting starbucks atm.

now, read the issue #1524, people have fully answered here

i'll try it once i get back home. i'm getting starbucks atm.

ok. Oh wait, i've found a better answer, it is from #1430

now, read the issue #1524, people have fully answered here

i'll try it once i get back home. i'm getting starbucks atm.

ok. Oh wait, i've found a better answer, it is from #1430

i don't exactly understand that one. i need in depth explaining because i can't comprehend stuff lolz

now, read the issue #1524, people have fully answered here

i'll try it once i get back home. i'm getting starbucks atm.

ok. Oh wait, i've found a better answer, it is from #1430

i don't exactly understand that one. i need in depth explaining because i can't comprehend stuff lolz

lmao. Basically, u will need to deactivate the codes with the word "mods" at MainMenuState (lines 38 and 235-236)

now, read the issue #1524, people have fully answered here

i'll try it once i get back home. i'm getting starbucks atm.

ok. Oh wait, i've found a better answer, it is from #1430

i don't exactly understand that one. i need in depth explaining because i can't comprehend stuff lolz

lmao. Basically, u will need to deactivate the codes with the word "mods" at MainMenuState (lines 38 and 235-236)

OH shit i get it now, i'll try this and see if it works. if it doesn't, i probs will leave the question open for a bit

now, read the issue #1524, people have fully answered here

i'll try it once i get back home. i'm getting starbucks atm.

ok. Oh wait, i've found a better answer, it is from #1430

i don't exactly understand that one. i need in depth explaining because i can't comprehend stuff lolz

lmao. Basically, u will need to deactivate the codes with the word "mods" at MainMenuState (lines 38 and 235-236)

OH shit i get it now, i'll try this and see if it works. if it doesn't, i probs will leave the question open for a bit

ok bro lol

now, read the issue #1524, people have fully answered here

i'll try it once i get back home. i'm getting starbucks atm.

ok. Oh wait, i've found a better answer, it is from #1430

i don't exactly understand that one. i need in depth explaining because i can't comprehend stuff lolz

lmao. Basically, u will need to deactivate the codes with the word "mods" at MainMenuState (lines 38 and 235-236)

OH shit i get it now, i'll try this and see if it works. if it doesn't, i probs will leave the question open for a bit

ok bro lol

it ran for about 10 min, then froze. just started back up, idk if its working tho

now, read the issue #1524, people have fully answered here

i'll try it once i get back home. i'm getting starbucks atm.

ok. Oh wait, i've found a better answer, it is from #1430

i don't exactly understand that one. i need in depth explaining because i can't comprehend stuff lolz

lmao. Basically, u will need to deactivate the codes with the word "mods" at MainMenuState (lines 38 and 235-236)

OH shit i get it now, i'll try this and see if it works. if it doesn't, i probs will leave the question open for a bit

ok bro lol

hey, could you help with a different thing? if so, email me at<EMAIL_ADDRESS>


[SPARK-35550][BUILD] Upgrade Jackson to 2.12.3

What changes were proposed in this pull request?
This pr upgrade Jackson version to 2.12.3.
Jackson Release 2.12.3: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.12.3
Why are the changes needed?
Upgrade to a new version to bring potential bug fixes like https://github.com/FasterXML/jackson-modules-java8/issues/207  and avro's master has been upgraded to Jackson to 2.12.3
Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass the Jenkins or GitHub Action

Test build #139039 has started for PR 32688 at commit 72c425c.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43557/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43557/

Test build #139039 has finished for PR 32688 at commit 72c425c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

@LuciferYang, the test failure should be fixed in the latest master branch.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139039/

thx @HyukjinKwon

Test build #139047 has started for PR 32688 at commit f5bafee.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43567/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43567/

Out of curiosity, what is the actual impact for Spark?

Out of curiosity, what is the actual impact for Spark?

There may not be any real benefits at the moment, if we are going to upgrade avro version in the future, we will need upgrade Jackson to this version because the master of avro already used Jackson 2.12.3. Of course, we can wait until really need to upgrade.

I see. I would suggest doing the upgrade until we have a better reason for this.

Test build #139047 has finished for PR 32688 at commit f5bafee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139047/

I think it's OK to upgrade - we tend to want Jackson to stay pretty up to date anyway

@srowen I see. I am OK to merge this as well.

Merged to master.

thx all ~

Hi all @LuciferYang @HyukjinKwon @gengliangwang @srowen  --
While attempting to run a job on k8s with a SNAPSHOT of today's master branch, I found there appears to be a conflict in our dependencies. The error occurs at the start of the job, and the stack trace is:
Exception in thread "main" java.lang.NoSuchMethodError: 'com.fasterxml.jackson.core.util.JacksonFeatureSet com.fasterxml.jackson.core.JsonParser.getReadCapabilities()'
	at com.fasterxml.jackson.databind.DeserializationContext.<init>(DeserializationContext.java:211)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.<init>(DefaultDeserializationContext.java:50)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext$Impl.<init>(DefaultDeserializationContext.java:391)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext$Impl.createInstance(DefaultDeserializationContext.java:413)
	at com.fasterxml.jackson.databind.ObjectMapper.createDeserializationContext(ObjectMapper.java:4656)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4585)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3593)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:247)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:200)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:495)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:451)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:416)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:397)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:933)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:220)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:186)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:84)
...

The root cause is that kubernetes-client has a dependency on Jackson 2.11.3: https://github.com/fabric8io/kubernetes-client/blob/d810aa1da99ff5cdb81cabea4f5a6dcf55ff0218/pom.xml#L87
When it initializes DeserializationContext, getReadCapabilities() is called here https://github.com/FasterXML/jackson-databind/blob/b5a60a816b9333edbba830bb252a1417dcd2686b/src/main/java/com/fasterxml/jackson/databind/DeserializationContext.java#L212
but that method has only existed since 2.12: https://github.com/FasterXML/jackson-core/blob/080eee156d82481625a38eda5a14ba4fbe2c4a46/src/main/java/com/fasterxml/jackson/core/JsonParser.java#L604
I will report back if downgrading the pom to 2.11.4 resolves the issue.

This upgrades from 2.12.2 to 2.12.3 so the compatibility w/ 2.11 is unlikely from this PR. Does that happen after this PR?

It's likely from https://github.com/apache/spark/pull/31878 if it's an issue. cc @wangyum and @xclyfe

cc @dongjoon-hyun too FYI

Thank you for pinging me!

BTW, Apache Spark K8s IT has been healthy.
Maybe, it seems that we need to add some test coverage for the failed API.

@dchristle Could you elaborate about the following a little bit more?

The error occurs at the start of the job,

@dongjoon-hyun Sure - the error triggers right after the job jar is added, i.e. there are no stages that start, and it's so early that the Spark UI is not accessible.
I tried rebuilding my Docker image from scratch and the error persists.  I am using the GCS connector and Snowflake connectors for this job, which are technically only on 3.1. I am not sure how to debug whether there is something specific about my build (aka a problem locally) or if there truly is a dependency conflict in the master branch.

Thank you for the detail. Let me build the vanilla Spark distribution from the master branch manually and play with K8s a little. I'll post the result here.

BTW, this is the community Jenkins success result with this PR
[SPARK-35550][BUILD] Upgrade Jackson to 2.12.3

https://amplab.cs.berkeley.edu/jenkins/view/Spark K8s Builds/job/spark-master-test-k8s/792/

I am able to build master branch and run TPCDS without any issue.
$ k get pod | head
NAME                  READY   STATUS    RESTARTS   AGE
tpcds-dhyun           1/1     Running   0          2m37s
tpcds-dhyun-exec-1    1/1     Running   0          2m3s
tpcds-dhyun-exec-10   1/1     Running   0          2m2s
tpcds-dhyun-exec-11   1/1     Running   0          2m1s
tpcds-dhyun-exec-12   1/1     Running   0          2m1s
tpcds-dhyun-exec-13   1/1     Running   0          2m1s
tpcds-dhyun-exec-14   1/1     Running   0          2m1s
tpcds-dhyun-exec-15   1/1     Running   0          2m1s
tpcds-dhyun-exec-16   1/1     Running   0          2m
...

$ k exec -it tpcds-dhyun -- /bin/bash
185@tpcds-dhyun:~/work-dir$ cd ../jars/
185@tpcds-dhyun:~/jars$ ls jackson*
jackson-annotations-2.12.3.jar	jackson-databind-2.12.3.jar	    jackson-datatype-jsr310-2.11.2.jar
jackson-core-2.12.3.jar		jackson-dataformat-cbor-2.12.3.jar  jackson-mapper-asl-1.9.13.jar
jackson-core-asl-1.9.13.jar	jackson-dataformat-yaml-2.12.3.jar  jackson-module-scala_2.12-2.12.3.jar
185@tpcds-dhyun:~/jars$ ls kube*
kubernetes-client-5.4.0.jar			  kubernetes-model-common-5.4.0.jar	   kubernetes-model-metrics-5.4.0.jar
kubernetes-model-admissionregistration-5.4.0.jar  kubernetes-model-coordination-5.4.0.jar  kubernetes-model-networking-5.4.0.jar
kubernetes-model-apiextensions-5.4.0.jar	  kubernetes-model-core-5.4.0.jar	   kubernetes-model-node-5.4.0.jar
kubernetes-model-apps-5.4.0.jar			  kubernetes-model-discovery-5.4.0.jar	   kubernetes-model-policy-5.4.0.jar
kubernetes-model-autoscaling-5.4.0.jar		  kubernetes-model-events-5.4.0.jar	   kubernetes-model-rbac-5.4.0.jar
kubernetes-model-batch-5.4.0.jar		  kubernetes-model-extensions-5.4.0.jar    kubernetes-model-scheduling-5.4.0.jar
kubernetes-model-certificates-5.4.0.jar		  kubernetes-model-flowcontrol-5.4.0.jar   kubernetes-model-storageclass-5.4.0.jar

Given the above, could you share your docker image's jar directory, @dchristle ? Do you have both Jackson 2.11.x and 2.12.3?

Or, is your Spark app using kubernetes-client library directly for some extension? It would be great if you can provide an reproducible example to us. We want to help you because K8s is GA in Apache Spark community. :)

Or, is your Spark app using kubernetes-client library directly for some extension? It would be great if you can provide an reproducible example to us. We want to help you because K8s is GA in Apache Spark community. :)

That was the root cause. I refactored my workflow to compile Spark from the master branch inside my Docker image, so I could be sure the environment was clean. Replacing the GCS jar I was using with the -shaded variant seems to have fixed the problem, which suggests that jar was pulling in a slightly older incompatible Jackson version. Thank you for the help!

@dongjoon-hyun Thank you for your help, busy with release spark 3.1.x used in our company recently, haha