Increase the number of dims for KNN vectors to 2048 [LUCENE-10471]

The current maximum allowed number of dimensions is equal to 1024. But we see in practice a couple well-known models that produce vectors with > 1024 dimensions (e.g mobilenet_v2 uses 1280d vectors, OpenAI / GPT-3 Babbage uses 2048d vectors). Increasing max dims to 2048 will satisfy these use cases.
I am wondering if anybody has strong objections against this.

Migrated from LUCENE-10471 by Mayya Sharipova (@mayya-sharipova), 6 votes, updated Aug 15 2022
Pull requests: https://github.com/apache/lucene/pull/874

Lots of things happened since Aug, like the arrival of ChatGPT, and people's increased desire to use OpenAI's state of the art embeddings which are of size 1536. Can you at least please increase it to 2048 for now, while you discuss upper limits?

Lots of things happened since Aug, like the arrival of ChatGPT, and people's increased desire to use OpenAI's state of the art embeddings which are of size 1536. Can you at least please increase it to 1536 for now, while you discuss upper limits?

Actually it is a one-line change (without any garantees), see https://github.com/apache/lucene/pull/874/files
If you really want to shoot you in the foot: Download source code of Lucene in the version you need for your Elasticsearch instance (I assume you coming from https://github.com/elastic/elasticsearch/issues/92458), patch it  with #874, and then run './gradlew distribution'. Copy the JAR files into your ES districution. Done.
But it is not sure if this will blos up and indexes created by that won't read anymore with standard Lucene

Why I was doing that suggestion: If you are interested, try it out with your dataset and your Elasticsearch server and report back! Maybe you will figure out that performance does not work or memory usage is too high.

I'll preface this by saying I am also skeptical that going beyond 1024 makes sense for most use cases and scaling is a concern. However, amidst the current excitement to try and use openai embeddings the first cut at choosing a system to store and use those embeddings was Elasticsearch. Then the 1024 limit was run into and so various folks are looking at other alternatives largely because of this limit.
The use cases tend to be Q/A, summarization, and recommendation systems for WordPress and Tumblr. There are multiple proof of concept systems people have built (typically on top of various typscript, javascript, or python libs) which use the openai embeddings directly (and give quite impressive results). Even though I am pretty certain that reducing the dimensions will be a better idea for many of these, the ability to build and prototype on higher dimensions would be extremely useful.

@uschindler @rmuir FWIW We are interested in using Lucene's kNN with 1536 dimensions in order to use OpenAI's embeddings API. We benchmarked a patched Lucene/Solr. We fully understand (we measured it :-P) that there is an increase in memory consumption and latency. Sure thing.
We have applications where dev teams have chosen to work with OpenAI embeddings and where the number of records involved and requests per second make the trade offs of memory and latency perfectly acceptable.
There is a great deal of enthusiasm around OpenAI and releasing a working application ASAP. For many of these the resource cost of 1536 dimensions is perfectly acceptable against the alternative of delaying a pilot to optimize further.
Our work would be a lot easier if Lucene's kNN implementation supported 1536 dimensions without need for a patch.

I'm reminded of the great maxBooleanClauses debate.  At least that limit is user configurable (for the system deployer; not the end user doing a query) whereas this new one for kNN is not.
I can understand how we got to this point -- limits often start as hard limits.  The current limit even seems high based on what has been said.  But users have spoken here on a need for them to configure Lucene for their use-case (such as experimentation within a system they are familiar with) and accept the performance consequences.  I would like this to be possible with a System property.  This hasn't been expressly asked?  Why should Lucene, just a library that doesn't know what's best for the user, prevent a user from being able to do that?
This isn't an inquiry about why limits exist; of course systems need limits.

Hi @dsmiley  I updated the dev discussion on the mailing list:
[Proposal] Remove max number of dimensions for KNN vectors
And proceeded with a pragmatic new mail thread, where we just collect proposals with a motivation (no discussion there):
Dimensions Limit for KNN vectors - Next Steps
Feel free to participate!
My intention is to act relatively fast (and then also operate Solr side).
It's a train we don't need/want to miss!

The rabbit hole that is trying to store Open AI embeddings in Elasticsearch eventually leads here.  I read the entire thread and unless I am missing something, the obvious move to make the limit configurable (up to a point) or at a minimum, increase the limit to 1536 to support the text-embedding-ada-002 model. In other words, there should be a compelling reason not to increase the limit beyond the fact that it will hard to reduce in the future.

Cross posting here because I responded to the PR instead of this issue.

...why is it then that GPT-4, which internally represents each token with a vector of more than 8192, still inaccurately recalls information about entities?

I think this comment actually supports @MarcusSorealheis argument?  e.g., What's the point in indexing 8K dimensions if it isn't much better at recall than 768?

If the real issue is with the use of HNSW, which isn't suitable for this, not that highe-dimensionality embeddings have value, then the solution isn't to not provide the feature, but to switch technologies to something more suitable for the type of applications that people use Lucene for.

I may be wrong but it seems like this is where most of the lucene committers here are settling?
Over a decade ago I wanted a high dimension index for some facial recognition and surveillance applications I was working on. I rejected Lucene at first only because of it being written in java and I personally felt something like C++ was a better fit for the high dimension job (no garbage collection to worry about). So I wrote a high dimension indexer for MongoDB inspired by RTree (for the record it's implementation is based on XTree) and wrote it using C++ 14 preview features (lambda functions were the new hotness on the block and java didn't even have them yet). Even in C++ back then SIMD wasn't very well supported by the compiler natively so I had to add all sorts of compiler tricks to squeeze every ounce of vector parallelization to make it performant. C++ has gotten better since then but I think java still lags in this area? Even JEP 426 is a ways off (maybe because OpenJDK is holding these things hostage)? So maybe java is still not the right fit here? I wonder though, does that mean Lucene shouldn't provide dimensionality higher than arbitrary 1024? Maybe not. I agree dimensional reduction techniques like PCA should be considered to reduce the storage volume. The problem with that argument is that dimensionality reduction fails when features are weakly correlated. You can't capture the majority of the signal in the first  N components and therefore need higher dimensionality. But does that mean that 1024 is still too low to make Lucene a viable option?
Aside from conjecture does anyone have empirical examples where 1024 is too low and what specific Lucene capabilities (e.g., scoring?) would make adding support for dimensions higher than 1024 really worth considering over using dimensionality reduction? If Lucene doesn't do this does it really risk the project becoming irrelevant? That sounds a bit like sensationalism. Even if higher dimensionality is added to the current vector implementation (I'd actually argue we should explore converting BKD to support higher dimensions instead) are we convinced it will reasonably perform without JEP 426 or better SIMD support that's only available in newer JDKs? Can anyone smart here post their benchmarks to substantiate their claims? I know Pinecone (and others) have blogged about their love for RUST for these kinds of applications. Should Lucene just leave this to the job of alternative Search APIs? Maybe something like Tantivy or Rucene? Or is it time we explore a new optional Lucene Vector module that supports cutting edge JDK features through gradle tooling for optimizing the vector use case?
Interested what others think.

Nice little rewrite ChatGPT did there.

❗ ❗ ❗

I also vote for the increase of dimensions. I use GPT Ada2 for embeddings and I have to recompile Lucene just to have the constant value increased. Having a system property for changing it would be nice.

Just adding a perspective, we're evaluating options for indexing vectors and need to support OpenAI's second-generation text embedding that outputs 1536 dimensions.
Raising Lucene's max to something like 2048 makes a lot of sense to me. Things are moving very quickly in the LLM space at the moment. I can see there are concerns about performance and scale, but today people don't choose Lucene to index vectors when they need to leverage GPU performance and have massive amounts of vector data. Anyway, just my two cents. :-)

Here's an example why making the vector dimensions configurable is a bad idea: #12281
This issue shows that each added dimension makes the floating point errors larger and sometimes also returns NaN. Do we have tests when multiplying vectors causes NaN?

Copying and pasting here, just for visibility:
"> Here's an example why making the vector dimensions configurable is a bad idea: #12281 This issue shows that each added dimension makes the floating point errors larger and sometimes also returns NaN. Do we have tests when multiplying vectors causes NaN?
I may sound like somebody who contradicts another just for the sake of doing so, but I do genuinely believe these kind of discoveries support the fact that making it configurable is actually a good idea:
We are not changing a production system here but we are changing a library.
Enabling more users to experiment with higher dimensions increase the probability of finding (and then solving) this sort of issues.
I suspect we are not recommending anywhere here to go to prod with un-tested and un-benchmarked vector sizes anyway
"

Enabling more users to experiment with higher dimensions increase the probability of finding (and then solving) this sort of issues.

It also shows that this causes a long-tail of issues:

If we would fix the mentioned issue in the way proposed there, the performance would be going down, as a widening conversion to double would break hotspot optimizations and would also not be possible with SIMD in a performant manner. So this is a reason to for now stay with the current limit.

In addition if we raise the number of dimensions people will then start claiming for higher precision in calculations, completely forgetting that Lucene is a full text search engine to bring results in milliseconds not 2 hours. Score calculations introduce rounding anyways and making them exact is (a) not needed for lucene (we just sort on those values) and (b) would slow down the whole thing so much.
So keep with the current limit and NOT make it configurable. I agree to raise the maximum to 2048 (while recommending to people to use Java 20 for running Lucene and enable incubator vectors).
At same time close any issues about calculation precission and on the other hand get the JDK people support half float calculations.

I think a library should empower a user to discover what works (and doesn't) for them, rather than playing big brother and insist it knows best that there's no way some high setting could ever work for any user.  Right?  By making it a system property that does not need to be configured for <= 1024, it should raise a red flag to users that they are venturing into unusual territory.  i.e. they've been warned.  They'd have to go looking for such a setting and see warnings; it's not something a user would do accidentally either.

if we raise the number of dimensions people will then start claiming for higher precision in calculations,

LOL People may ask for whatever they want :-) including using/abusing a system beyond its intended scope.  So what?  BTW I've thoroughly enjoyed seeing several use cases of my code in Lucene/Solr that I had never considered yet worked really well for a user :-D.  Pure joy.  Of course not every request makes sense to us.  I'd rather endure such than turn users away from Lucene that we can support trivially today.

In addition if we raise the number of dimensions people will then start claiming for higher precision in calculations, completely forgetting that Lucene is a full text search engine to bring results in milliseconds not 2 hours. Score calculations introduce rounding anyways and making them exact is (a) not needed for lucene (we just sort on those values) and (b) would slow down the whole thing so much.
So keep with the current limit and NOT make it configurable. I agree to raise the maximum to 2048 (while recommending to people to use Java 20 for running Lucene and enable incubator vectors).
At same time close any issues about calculation precission and on the other hand get the JDK people support half float calculations.

@uschindler , I am not convinced but it's fine to have different opinions!
I do agree we should improve all the improvable and at the same time, in parallel, give users flexibility to experiment:

break it
make it super slow with enormous vectors
make it super slow with enormous field content
make it super slow with a great number of fields
...
I just think that the more you let your users do (with a reasonable effort), the more we'll gain users and consequentially improvements (it's a fact that more people you involve in a project as users the more you increase the probability of developing contributors and contributions).

We may have different opinions here and that's fine, but my intent as a committer is to build the best solution for the community rather than the best solution according to my ideas.
You know, if we wanted sub-ms responses all the time we could set a hard limit to 1024 chars per textual field and allow a very low number of fields, but then would Lucene attract any user at all?

I would like to renew the issue in light of the recent integration of incubating Panama Vector API, as indexing of vectors with it much faster.
We run a benchmarking test, and indexing a dataset of vectors of 1536 dims was slightly faster than indexing of 1024 dims. This gives us enough confidence to extend max dims to 2048.
Test environment

Dataset:

nq dataset with text field embedded with OpenAI text-embedding-ada-002 model, 1536 dims

KnnGraphTester

maxConn: 16, beamWidthIndex: 100

Apple M1 laptop

Test1:

Lucene 9.7 branch
Panama Vector API not enabled
vector dims=1024 (OpenAi vectors that were cut off to first 1024 dims)
Results: Indexed 2680961 documents in 3287s

Details
 java -cp  "lib/*:classes" -Xmx16g -Xms16g org.apache.lucene.util.hnsw.KnnGraphTester -dim 1024 -ndoc 2680961 -reindex -docs vectors_dims1024.bin -maxConn 16 -beamWidthIndex 100
creating index in vectors_dims1024.bin-16-100.index
MS 0 [2023-06-26T11:10:24.765857Z; main]: initDynamicDefaults maxThreadCount=4 maxMergeCount=9
IFD 0 [2023-06-26T11:10:24.782017Z; main]: init: current segments file is "segments"; deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@646d64ab
IFD 0 [2023-06-26T11:10:24.783554Z; main]: now delete 0 files: []
IFD 0 [2023-06-26T11:10:24.784291Z; main]: now checkpoint "" [0 segments ; isCommit = false]
IFD 0 [2023-06-26T11:10:24.784338Z; main]: now delete 0 files: []
IFD 0 [2023-06-26T11:10:24.785377Z; main]: 0 ms to checkpoint
IW 0 [2023-06-26T11:10:24.785523Z; main]: init: create=true reader=null
IW 0 [2023-06-26T11:10:24.790087Z; main]:
<EMAIL_ADDRESS>lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c039ac6
index=
version=9.7.0
analyzer=org.apache.lucene.analysis.standard.StandardAnalyzer
ramBufferSizeMB=1994.0
maxBufferedDocs=-1
mergedSegmentWarmer=null
delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
commit=null
openMode=CREATE
similarity=org.apache.lucene.search.similarities.BM25Similarity
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=4, maxMergeCount=9, ioThrottle=true
codec=Lucene95
infoStream=org.apache.lucene.util.PrintStreamInfoStream
mergePolicy=[TieredMergePolicy: maxMergeAtOnce=10, maxMergedSegmentMB=5120.0, floorSegmentMB=2.0, forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022208E12, noCFSRatio=0.1, deletesPctAllowed=20.0
readerPooling=true
perThreadHardLimitMB=1945
useCompoundFile=false
commitOnClose=true
indexSort=null
checkPendingFlushOnUpdate=true
softDeletesField=null
maxFullFlushMergeWaitMillis=500
leafSorter=null
eventListener=org.apache.lucene.index.IndexWriterEventListener$1@2173f6d9
writer=org.apache.lucene.index.IndexWriter@307f6b8c

IW 0 [2023-06-26T11:10:24.790232Z; main]: MMapDirectory.UNMAP_SUPPORTED=true
DWPT 0 [2023-06-26T11:19:47.652040Z; main]: flush postings as segment _0 numDocs=460521
IW 0 [2023-06-26T11:19:47.653761Z; main]: 1 ms to write norms
IW 0 [2023-06-26T11:19:47.653954Z; main]: 0 ms to write docValues
IW 0 [2023-06-26T11:19:47.654032Z; main]: 0 ms to write points
IW 0 [2023-06-26T11:19:49.152263Z; main]: 1498 ms to write vectors
IW 0 [2023-06-26T11:19:49.166472Z; main]: 14 ms to finish stored fields
IW 0 [2023-06-26T11:19:49.166642Z; main]: 0 ms to write postings and finish vectors
IW 0 [2023-06-26T11:19:49.167167Z; main]: 0 ms to write fieldInfos
DWPT 0 [2023-06-26T11:19:49.167954Z; main]: new segment has 0 deleted docs
DWPT 0 [2023-06-26T11:19:49.168030Z; main]: new segment has 0 soft-deleted docs
DWPT 0 [2023-06-26T11:19:49.169572Z; main]: new segment has no vectors; no norms; no docValues; no prox; freqs
DWPT 0 [2023-06-26T11:19:49.169670Z; main]: flushedFiles=[_0_Lucene95HnswVectorsFormat_0.vem, _0.fdm, _0_Lucene95HnswVectorsFormat_0.vec, _0.fdx, _0_Lucene95HnswVectorsFormat_0.vex, _0.fdt, _0.fnm]
....
Indexed 2680961 documents in 3287s

Test2

Lucene 9.7 branch with FloatVectorValues.MAX_DIMENSIONS set to 2048
Panama Vector API enabled
dims=1536
Results: Indexed 2680961 documents in 3141s

Details
java --add-modules jdk.incubator.vector -cp  "lib/*:classes" -Xmx16g -Xms16g org.apache.lucene.util.hnsw.KnnGraphTester -dim 1536 -ndoc 2680961 -reindex -docs vectors.bin -maxConn 16 -beamWidthIndex 100

WARNING: Using incubator modules: jdk.incubator.vector
creating index in vectors.bin-16-100.index
Jun 26, 2023 10:34:29 A.M. org.apache.lucene.store.MemorySegmentIndexInputProvider <init>
INFO: Using MemorySegmentIndexInput with Java 20; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
MS 0 [2023-06-26T14:34:29.271516Z; main]: initDynamicDefaults maxThreadCount=4 maxMergeCount=9
IFD 0 [2023-06-26T14:34:29.329779Z; main]: init: current segments file is "segments"; deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@64f6106c
IFD 0 [2023-06-26T14:34:29.336415Z; main]: now delete 0 files: []
IFD 0 [2023-06-26T14:34:29.338546Z; main]: now checkpoint "" [0 segments ; isCommit = false]
IFD 0 [2023-06-26T14:34:29.338654Z; main]: now delete 0 files: []
IFD 0 [2023-06-26T14:34:29.347243Z; main]: 2 ms to checkpoint
IW 0 [2023-06-26T14:34:29.348255Z; main]: init: create=true reader=null
IW 0 [2023-06-26T14:34:29.368686Z; main]:
<EMAIL_ADDRESS>lockFactory=org.apache.lucene.store.NativeFSLockFactory@319b92f3
index=
version=9.7.0
analyzer=org.apache.lucene.analysis.standard.StandardAnalyzer
ramBufferSizeMB=1994.0
maxBufferedDocs=-1
mergedSegmentWarmer=null
delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
commit=null
openMode=CREATE
similarity=org.apache.lucene.search.similarities.BM25Similarity
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=4, maxMergeCount=9, ioThrottle=true
codec=Lucene95
infoStream=org.apache.lucene.util.PrintStreamInfoStream
mergePolicy=[TieredMergePolicy: maxMergeAtOnce=10, maxMergedSegmentMB=5120.0, floorSegmentMB=2.0, forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022208E12, noCFSRatio=0.1, deletesPctAllowed=20.0
readerPooling=true
perThreadHardLimitMB=1945
useCompoundFile=false
commitOnClose=true
indexSort=null
checkPendingFlushOnUpdate=true
softDeletesField=null
maxFullFlushMergeWaitMillis=500
leafSorter=null
eventListener=org.apache.lucene.index.IndexWriterEventListener$1@10a035a0
writer=org.apache.lucene.index.IndexWriter@67b467e9

IW 0 [2023-06-26T14:34:29.369224Z; main]: MMapDirectory.UNMAP_SUPPORTED=true
Jun 26, 2023 10:34:29 A.M. org.apache.lucene.util.VectorUtilPanamaProvider <init>
INFO: Java vector incubator API enabled; uses preferredBitSize=128
DWPT 0 [2023-06-26T14:40:36.945965Z; main]: flush postings as segment _0 numDocs=314897
IW 0 [2023-06-26T14:40:36.949748Z; main]: 2 ms to write norms
IW 0 [2023-06-26T14:40:36.950336Z; main]: 0 ms to write docValues
IW 0 [2023-06-26T14:40:36.950452Z; main]: 0 ms to write points
IW 0 [2023-06-26T14:40:38.639069Z; main]: 1688 ms to write vectors
IW 0 [2023-06-26T14:40:38.669749Z; main]: 29 ms to finish stored fields
IW 0 [2023-06-26T14:40:38.670044Z; main]: 0 ms to write postings and finish vectors
IW 0 [2023-06-26T14:40:38.670847Z; main]: 0 ms to write fieldInfos
DWPT 0 [2023-06-26T14:40:38.672893Z; main]: new segment has 0 deleted docs
DWPT 0 [2023-06-26T14:40:38.673016Z; main]: new segment has 0 soft-deleted docs
DWPT 0 [2023-06-26T14:40:38.675915Z; main]: new segment has no vectors; no norms; no docValues; no prox; freqs
DWPT 0 [2023-06-26T14:40:38.676120Z; main]: flushedFiles=[_0_Lucene95HnswVectorsFormat_0.vem, _0.fdm, _0_Lucene95HnswVectorsFormat_0.vec, _0.fdx, _0_Lucene95HnswVectorsFormat_0.vex, _0.fdt, _0.fnm]
DWPT 0 [2023-06-26T14:40:38.676311Z; main]: flushed codec=Lucene95
DWPT 0 [2023-06-26T14:40:38.677609Z; main]: flushed: segment=_0 ramUsed=1,945.012 MB newFlushedSize=1,863.46 MB docs/MB=168.985
DWPT 0 [2023-06-26T14:40:38.680696Z; main]: flush time 1735.77025 ms
IW 0 [2023-06-26T14:40:38.682741Z; main]: publishFlushedSegment seg-private updates=null
IW 0 [2023-06-26T14:40:38.683738Z; main]: publishFlushedSegment _0(9.7.0):C314897:[diagnostics={source=flush, lucene.version=9.7.0, os.version=13.2.1, os.arch=x86_64, os=Mac OS X, java.vendor=Oracle Corporation, java.runtime.version=20.0.1+9-29, timestamp=1687790438678}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=717x28qrd00q2ke3d17eerf4x
BD 0 [2023-06-26T14:40:38.687864Z; main]: finished packet delGen=1 now completedDelGen=1
IW 0 [2023-06-26T14:40:38.691420Z; main]: publish sets newSegment delGen=1 seg=_0(9.7.0):C314897:[diagnostics={source=flush, lucene.version=9.7.0, os.version=13.2.1, os.arch=x86_64, os=Mac OS X, java.vendor=Oracle Corporation, java.runtime.version=20.0.1+9-29, timestamp=1687790438678}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=717x28qrd00q2ke3d17eerf4x
IFD 0 [2023-06-26T14:40:38.692639Z; main]: now checkpoint "_0(9.7.0):C314897:[diagnostics={source=flush, lucene.version=9.7.0, os.version=13.2.1, os.arch=x86_64, os=Mac OS X, java.vendor=Oracle Corporation, java.runtime.version=20.0.1+9-29, timestamp=1687790438678}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=717x28qrd00q2ke3d17eerf4y" [1 segments ; isCommit = false]
IFD 0 [2023-06-26T14:40:38.693268Z; main]: now delete 0 files: []
IFD 0 [2023-06-26T14:40:38.693464Z; main]: 1 ms to checkpoint
MP 0 [2023-06-26T14:40:38.700301Z; main]:   seg=_0(9.7.0):C314897:[diagnostics={source=flush, lucene.version=9.7.0, os.version=13.2.1, os.arch=x86_64, os=Mac OS X, java.vendor=Oracle Corporation, java.runtime.version=20.0.1+9-29, timestamp=1687790438678}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=717x28qrd00q2ke3d17eerf4y size=1863.460 MB
MP 0 [2023-06-26T14:40:38.701368Z; main]: findMerges: 1 segments
MP 0 [2023-06-26T14:40:38.701645Z; main]:   allowedSegmentCount=10 vs count=1 (eligible count=1)
 ...

Indexed 2680961 documents in 3141s

We run a benchmarking test, and indexing a dataset of vectors of 1536 dims was slightly faster than indexing of 1024 dims. This gives us enough confidence to extend max dims to 2048 (at least when vectorization is enabled).

I found this very strange at first :)
But then I read more closely, and I think what you meant is indexing 1024 dims without Panama (SIMD vector instructions) is slower than indexing 1536 dims with Panama enabled?  Which is really quite impressive.
Do we know what gains we see at search time going from 1024 -> 1536?

@mikemccand Indeed, exactly as said, sorry for being unclear. We have not checked search, will work on that.
@uschindler Thanks, indeed, we need tests on other machines. +1 for raising dims to 2048 in HNSW codec.

Can we run this test with lucene's defaults (e.g. not a 2GB rambuffer)?
We are still talking about an hour to index < 3M docs, so I think the performance is not good.
As i've said before, i never thought 1024 was a good situation either. 768 is also excruciating.
Purpose of the vectorization is just to alleviate some of the pain. It is like giving the patient an aspirin, it doesn't really fix the problem.

I am extremely curious, what should we consider a good performance to index <3M docs?
I mean, I agree we should always try to improve things and aim for the stars, but as maintainers of a library who are we to decide what's acceptable and what's not for the users?
Is it because of a comparison with other libraries or solutions?
They may have many reasons for being faster (and definitely we should take inspiration)
If we look to : https://home.apache.org/~mikemccand/lucenebench/indexing.html , we clearly improved the indexing throughput substantially over the years, does this mean that Lucene back in 2011 should have not committed additional features/improvements because for some people (people from the future) "it was slow"?

Leaving a higher number of segments dodges the merge costs, I think.

This benchmark really only measures the flushing cost, as ConcurrentMergeScheduler is used, so merges run in background threads. So the improvement makes sense to me as the cost of adding vectors into a HNSW graph increases as the size of the HNSW graph increases. If we want to get a sense of the number of docs per second per core that we support with a 2GB RAM buffer vs. the 16MB default, using SerialMergeScheduler would be a better choice.

Hi,
actually this issue is already resolved, although the DEFAULT did not change (and won't change due to performance risks), see here: https://github.com/apache/lucene/pull/12436 - this PR allows users of Lucene to raise the limit (at least for HNSW codec) on codec level.
To implement (on your own risk), create your own  KnnVectorsFormat and let it return a different number fog getMaxDimensions(). Then construct your own codec from it and index your data.

@mayya-sharipova: Should we close this issue or are there any plans to also change the default maximum? I don't think so.

Yes, thanks for the reminder. Now Codec is responsible for managing dims, we can close it.


Load order issue

The changes introduced in #96 cause difficult to debug results if one requires multiple gems that themselves require api_auth.
Lets say I have 2 gems, gem A and gem B. Gem A requires api_auth but does not require active_resource. Gem B requires api_auth and active_resource and uses the active_resource mixin. If, in my gemfile, I require gem A before gem B then gem A will fail to load with an error saying that the mixins like with_api_auth are not defined.
Would it be possible to revert to the old behaviour? Otherwise if multiple libraries use api_auth then one needs to be careful not to load api_auth before the rails bits that it's modifying.

The old behaviour of requiring an "optional" library just due to the library existing on the system is undesirable.
The point of a railtie is, of course, to "magically" tie in to rails if rails happens to be defined.
However, I understand that something like active resource is not part of the standard rails install anymore, so it is very easy to get into the situation where api-auth is loaded before active resource.
I think maybe the path forward is to break railtie.rb apart into the ActionController bits and the ActiveResource bits. Then railtie.rb can try to load both parts automatically, but if the active resource one desn't get loaded due to load order "issues", it would also be possible to load just the active resource file via a rails initializer or someone else as well. Would you be open to making those changes?

Yeah I can take a stab at it.

I have this issue with ruby '2.5.1' and gem 'rails', '~> 5.2.0'
In my Gemfile the order is like this:

gem 'activeresource'
gem 'api-auth'

and works other way i get undefined method `with_api_auth' for BaseResource:Class.

I also had this issue with Ruby 2.3.8 and rails Rails <IP_ADDRESS>
As you no longer need to gem 'activeresource' anymore, I just put gem 'api-auth' at the very end of the Gemfile and things worked.


Replace VerifiedProcedure with VerifiedNetwork

It looks extraneous to wrap each RemoteProcedure in VerifiedProcedure. Let's check JSON structure in a decorator for SimpleNetwork and throw an error there.

@biboran/z please, pay attention to this issue

Job gh:blockstoreapp/web3swift#106 is not assigned, can't get performer

The job is not in WBS, won't close the order


Reload all

Create a PR so that we can talk about this. :)

I've tried to use this one personally, and it's definitely a game changer, now we don't need to envy the ReactNative guys for they js live reloading. Because what we have is even better.

Not just styling updates anymore. the whole deal:

Adding, removing views on the fly.
Touch event logic handling update on the fly.
...

Let's create an issue to discuss it.


Conda forge package has broken version

Version command on conda-forge package returns version 0.0.0

Duplicate of #562


add support for strikethrough (not supported on all terminals)

Not sure if this was omitted on purpose. I added to the tests and the cmd line tool as well but wasn't 100% sure what the output was supposed to look like. Let me know if you want this cleaned up somehow.

Just implemented the same just to find that threre is a PR open for more than a year...
@mgutz in the project dead?
@ilowe would you mind to update doc.go and README.md as well in ilowe:master?

It's not dead just very stable. I could not confirm it worked on my terminal back then (rxvt-unicode) and forgot about it.

@mgutz Thanks for merging! Glad to see that since I just started using your lib in a project...


✨update: add 132nd CatchUp Summary

Sorry for the delay. I don't have any particular excuse. I was out of my pace for a few days in a row.

It's okay. All of us are struggling :'))

Okay, I'm squashing and merging this PR as well. Thank you everyone!


Force obseravable artifacts to be strings in TheHive

It follows #254 discussion about TheHive alerter.
Without casting,  if matched field is an integer as ES schema, TheHive APIs throw an error as they expect strings as data values.

Thanks for the contribution. A unit test to validate that the change is working, and doesn't get reverted by a future commit, would be a good addition. Review this PR if you'd like to see how the other fix tested this: https://github.com/jertel/elastalert2/pull/206/files

Hi @jertel. I've tried to follow what were done with that PR.
Hope the change in unit test is enough to validate the request.


Feature Request: wrap the position and ring dials on Enigma

It would be nice to wrap the position and ring dials when using Enigma so that when they reach 26 we don't have to press minus 25 times or type in the field to change the amount to 1.
My use case has been testing the effect of moving the position one letter at a time on the decryption result.

Hi @pinduli! I just implemented rotation into the number setting. Position and ring settings now wrap as you described. Thank you for your feedback!

Awesome, thanks!


Build a Tokenizer that works well with Duckling and SpaCy

Description of Problem:
In the financial demo, SpaCy tokenizer would be needed for SpaCy to extract entities correctly, but using that degrades the performance of Duckling's entity extraction in comparison to the WhitespaceTokenizer. We need a tokenizer that works well for SpaCy and Duckling. See Slack thread.
Overview of the Solution:
Find out what causes the performance drop in Duckling entity extraction when SpaCy tokenizer is used and build a custom tokenizer that fixes this.
Definition of Done:

[ ] We have a new SpacyAndDucklingTokenizer component that works for both SpaCy and Duckling
[ ] Tests are added
[ ] Feature described the docs
[ ] Feature mentioned in the changelog

Ah yeah, I hit this issue before too with tools like dateparser. There's a bunch of great parsing tools that assume access to the full text as opposed to tokens.
Do we really want to have a SpacyAndDucklingTokenizer in our stack though? I'm wondering if instead it makes more sense to build a spaCy compatible plugin for duckling.

Hi @JEM-Mosig , Just curious, How does any tokenizer affect duckling performance? Because a per code here https://github.com/RasaHQ/rasa/blob/690f8b891305e5bc36c22febd6e56da26a66be6e/rasa/nlu/extractors/duckling_entity_extractor.py#L182 we are simply passing the user text and not the tokens to Duckling.
Correct me if I have missed some things.


feature request - configurable nameserver bind address

It would decrease fragility in some deployment scenarios to have a configurable nameserver bind address. My first thought is that it should allow any interface address and network interface to be specified, and not necessarily be restricted to the DNS intercept range.
This would allow creative use of the tunneler in Docker and Kubernetes environments where there are typically multiple virtual networks in play.

@ekoby

@ekoby Let me know if you like this idea?


Query: What is maximum number of records facets can handle?

What will be the maximum no of records facets can handle/ how we can determine the limit to be used.

The limit will depend on the RAM of your machine, and the limit will be different for Facets Overview and Facets Dive.
Do you have a dataset that has failed in one of the visualizations due to size?

@jameswex I wanted to try with some big record sets. Before doing that Just thought of checking with you guys :smile:

Facets dive freezes on a dataset of 1 million+ rows,  250 columns. I have 256 GB RAM.

@vatsan not surprising. We haven't used dive with more than 50k rows.

Closing as the question has been answered. Please re-open if further discussion is needed.


Adress failing "Install from conda-forge" workflow

Starting with the 15th of January the "Install from conda-forge" workflow started failing on windows-latest-anaconda.

Closed with https://github.com/iiasa/message_ix/pull/541.


Headless Chrome version not parsed correctly

Using a newer version of Chromium in headless mode, useragent does not report the version correctly.
Agent {   family: 'HeadlessChrome',   major: '0',   minor: '0',   patch: '0',   source: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/64.0.3240.0 Safari/537.36' }

uap-core had incomplete regex, I made a PR to fix that in https://github.com/ua-parser/uap-core/pull/263, it's just beem merged.
I think now useragent just needs to pull the latest uap-core.

This works in useragent 2.3.0.


Semantic highlighting for format macros

The {} in println!("{}", 1) is highlighted specially, which is a very nice feature. However, it only works for some of the format-like macros:

[ ] asm!
[x] assert!
[x] assert_eq!
[x] assert_ne!
[x] debug_assert!
[x] debug_assert_eq!
[x] debug_assert_ne!
[x] eprint!
[x] eprintln!
[x] format!
[x] format_args!
[x] format_args_nl!
[x] panic! *
[x] print!
[x] println!
[x] todo! *
[x] unimplemented! *
[ ] unreachable!
[x] write!
[x] writeln!

Arguably, support for asm is more complicated, but it seems close enough to format_args style input in the first argument that I think that applying the usual syntax highlighting will work well enough. Here's a rust snippet containing all the macros for quick testing:
fn main() {
    asm!("mov {}, 0", out(reg) _);
    assert!(true, "{}", 1);
    assert_eq!(1, 2, "{}", 1);
    assert_ne!(1, 2, "{}", 1);
    debug_assert!(true, "{}", 1);
    debug_assert_eq!(1, 2, "{}", 1);
    debug_assert_ne!(1, 2, "{}", 1);
    eprint!("{}", 1);
    eprintln!("{}", 1);
    format!("{}", 1);
    format_args!("{}", 1);
    format_args_nl!("{}", 1);
    panic!("{}", 1);
    print!("{}", 1);
    println!("{}", 1);
    todo!("{}", 1);
    unimplemented!("{}", 1);
    unreachable!("{}", 1);
    write!((), "{}", 1);
    writeln!((), "{}", 1);
}

Tested on rust-analyzer v0.2.760 on VSCode with rustc 1.57.0-nightly (11491938f 2021-09-29). The macros marked with * are recent regressions, which work on stable but not on the latest nightly, presumably due to a change in the implementation of panic! relating to rust 2021.

#10397 fixes panic!, assert! and debug_assert!

todo! and unimplemented! both expand to
$crate::panic!("not (yet) implemented: {}", $crate::format_args!($($arg)+))

which we should already handle, so there's another bug in there.
unreachable! on the other hand expands to
$crate::panic!($crate::concat!("internal error: entered unreachable code: ", $fmt), $($arg)*)

which is hard to handle correctly, since the actual format string passed to (const_)format_args! differs from the one the user sees

Oh, todo! and unimplemented! are working as intended - they don't take any format string, so the first arg is just a plain string literal. I have removed them from the list.

No, they do:
fn main() {
    todo!("TODO: {}", 123);
}

thread 'main' panicked at 'not yet implemented: TODO: 123', src/main.rs:2:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

#10412 highlights asm! as a format string (note that there's also https://github.com/rust-analyzer/rust-analyzer/issues/6031 for actual assembly syntax highlighting)


Migrates from ReactiveCocoa 2.x API to RxSwift

Well, I might need to make myself a cup of tea for this one.

@orta Well, you've reviewed each merge into develop :stuck_out_tongue:

Alright, I've given this a pretty solid look over, again. :ship:


Planned SDK 2.0 Release (Important Dates and Information)

Description
Quick Summary
We're planning to release a 2.0 version of stable SDK packages. This release will include breaking changes that aim to make the stable SDK packages easier to maintain and use - which will improve development velocity in the long run.
We plan to feature-freeze 1.x and 0.x on Dec 14 2024, any feature PRs merged later than that will target SDK 2.0.
We plan to release SDK 2.0 on Feb 17 2025 - between feature-freeze and SDK 2.0 release 1.x and 0.x packages will only receive bugfix releases.
Why we're bumping the SDK to 2.0
The 1.x SDK release supports Node.js versions that are out-of-date, such as Node.js v14. As more and more of our dependencies and devDependencies drop support for this Node.js version we amass more and more technical debt. It forces us to develop workarounds which take lots of time to maintain - this takes time away from feature development and is not sustainable long-term. Dropping support for Node versions is a breaking change and therefore requires us to bump the major version of our packages.
Similar to the outdated Node.js versions, we also support TypeScript >4.4, which limits us in which tools and language features we can use. Bumping TypeScript will cause our type-output to change and may break users that also use older TypeScript versions. Therefore this change requires us to bump the major version.
We have identified problems with our current exports. We usually export TypeScript classes, which, in the context of a library means that we expose private fields as part of the public API. This makes it extremely difficult to innovate as every small change is potentially breaking for our users. We also export some types that were unintentionally exported. Removing these types from the public API is a breaking change. It does, however, make it easier to make internal changes and reduces the risk for unintentional breaking changes for end-users.
We currently do not specify a minimum supported browser version this means that we have to deal with bugs that only occur on very old, and long-out-of-support browsers. Removing this support is a breaking change, but by dropping support for these old versions, we free up bandwidth to work on other things.
Some features are difficult for end-users to understand due to the way they are implemented and it is possible to do things in an incorrect order. However, in many cases we cannot change the public interface to make it easier to use without introducing a breaking change. In SDK 2.0 we aim to iron out some of these issues and make the SDK easier to use in these areas.
How (and when) we're moving forward

[!IMPORTANT]
@opentelemetry/api follows a different versioning scheme and is NOT included in the SDK 2.0 efforts and will therefore not receive any breaking changes. No pull-requests for breaking changes targeting the @opentelemetry/api package will be accepted.

Currently development for SDK 2.0 takes place on the next branch. There we stage these changes to ensure that feature development can continue for the time being. Maintainers periodically merge changes from main to next to ensure features and bugfixes are kept in sync.
On Dec 14 2024 we will enter feature-freeze for 1.x. From this point onward:

only bugfixes will be accepted for 1.x
any unmerged features will only be accepted for 2.0, users may have to rebase their PRs to ensure that they remain mergable

On Feb 17 2025 we will release 2.0.
In the time from Dec 14 2024 to Feb 17 2025

we may accept breaking changes to the SDK and other packages as long as these changes loosely follow the goals set out in the above sections and follow the specification.
we will not release any new features, but we will still accept feature PRs to the 2.0 branch
we will back-port and release bugfixes for the state that entered feature-freeze on Dec 14 2024 (the 1.x and 0.x line of packages)
contrib development will continue as-usual

What this means for you
End-Users
You will still receive bugfixes, but new features merged after Dec 14 2024 will only be released on Feb 17 2025.
There will be no breaking changes in @opentelemetry/api as it follows a different versioning scheme and has different stability guarantees then the SDK 2.0 packages. Any code written against @opentelemetry/api should continue to work as usual. You may have to update to the latest feature version of @opentelemetry/api to keep compatible with SDK 2.0 packages.
Upon release you may have to adjust your SDK setup code to the new interface. To prepare for this, we recommend you periodically update @opentelemetry/* packages to the latest version, and if you are using interfaces that are marked as @deprecated we recommend you take the transition path outlined for that feature.
Instrumentation authors
@opentelemetry/instrumentation may receive breaking changes, so you may have to update your instrumentation upon release to keep compatible with newer releases of the package. However, you will still be able to register your instrumentation with an older versions of @opentelemetry/instrumentation
@opentelemetry/api WILL NOT receive breaking changes. Your actual instrumentation code (writing metrics, traces, interacting with context and propagation) using @opentelemetry/api should remain the same even when SDK 2.0 is released.
If you're using any other @opentelemetry/* packages, please ensure that you're not using any deprecated features. You may still have to adjust some of your code to the new interface upon SDK 2.0 release. If you SDK components in testing, you may have to adjust tests when upgrading to SDK 2.0.
Thrid-party distributors of OpenTelemetry JavaScript SDKs
If you're a third-party distributor of OpenTelemetry JavaScript SDKs you should follow the same steps as outlined in the End-User section. Upon release, you may have to update your distribution's code to adjust to the new public interface of the OpenTelemetry JavaScript SDK. Depending on if your distribution exposes/accepts SDK 1.x types or not, this may also require you to bump the major version of your distribution.
If you implement a thrid-party SDK based on @opentelemetry/api, the planned changes in the SDK do not affect you.

In the time from Dec 14 2024 to Feb 17 2025
...
we will not release any new features

We will not release new features, but we will still accept feature PRs to the 2.0 branch I think correct?

This sounds good! Are there plans to also release (some of?) the currently experimental packages - mainly thinking of @opentelemetry/instrumentation - as stable? IMHO this would be extremely helpful and important.
And to clarify, what are the exact versions you are looking to Support?

Node 16+
Typescript 5.0+

or others?

This sounds good!

🙌

Are there plans to also release (some of?) the currently experimental packages - mainly thinking of @opentelemetry/instrumentation - as stable? IMHO this would be extremely helpful and important.

Please see #5149 for announcements of us starting to work on that. As stated in #5149 our priority is currently:

ship SDK 2.0 in Feburary
ship OTLP Exporters as GA (likely with 2.0) - work on that started a few months ago already and was needed to simplify some changes for SDK 2.0

Once any of these are done we'll chose another topic to move up into Focus Topics. We have been doing "everything at once" for a quite a while now and it's very clear to everybody that it's not working and we're not shipping things at the velocity we'd like. So focusing on fewer things is what we're doing deal with that.
I made a write-up of what we need to do to @opentelemetry/instrumentation to get it stable here (#4586) - it's already in the backlog in #5149 and it's going to be a lot of work.

And to clarify, what are the exact versions you are looking to Support?
Node 16+
Typescript 5.0+
or others?

We're looking to support Node 18+
Typescript 5.6 (ish), currently still being decided - see #5145
Also plans for the future are to cut major releases more often (possibly on a yearly cadence) to drop support for old versions so that we don't fall back into the same problems that we're dealing with today.

@mydea - update: we'll liklely adopt definetly-typed's policy, so that means<EMAIL_ADDRESS>for the 2.0 SDK.


Webpack has issues with loading nested npm module

I am using npm for managing client libs. My application has dependency to legacy lodash (2.4.1) and I am trying to add redux (3.3.1), which also has own lodash (^4.2.1) dependency. Accordingly to NPM 3.3.12 I have following structure
-node_modules
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>While I am trying to build my application with dependency to redux I get an error:
log.txt
Please see the example attached:
bug.zip
Just extract and run:

npm i
webpack

So, the issue is that during building my application webpack does not aware about nested/own dependencies of redux package.

    resolve: {
        root: __dirname,
        modulesDirectories: [
            __dirname,
            path.join(__dirname, 'node_modules')
        ]
    },

should be
    resolve: {
        root: __dirname,
        modulesDirectories: ['node_modules']
    },


Links to Colab in "Lightning in 2 steps" are dead

📚 Documentation
Under "Examples of systems are:" all the Colab links are dead.

Dear @enolan,
Thanks for reporting this issue. Would you be interested in making a contribution ?
Best,
T.C

Hi, @tchaton shall I work on this!?

Dear @seanbenhur,
Yes, please !
Best,
TC

The notebooks are not available on the master branch, it is available here, https://github.com/PyTorchLightning/lightning-tutorials/tree/6100885854c803458886c731cddd6bd67498c0a1, is it okay to use this branch!?


[Remote Store] Remove reliance of checkpoints from replica shards if remote translog is enabled

Is your feature request related to a problem? Please describe.
We need to identify all such interaction points with checkpoints and mutates it while in primary shard. Currently checkpoints are used in a lot of areas directly or indirectly like Index commits, translogs, segments, insync shards, etc. But primarily it keeps track of  how far are the replicas behind the primary. It also helps in deciding from what sequence number should the recovery happen on replicas since a definition of global / local checkpoint guarantees durability of data upto the sequence number it refers to.
Describe the solution you'd like
With no-op replication, the calls would not be reaching to the engine of the replica anymore. The calls would return to the primary before that itself. That kinds of makes the need of relying on the checkpoints unnecessary for primary and as well as replica. In this issue, we plan to remove the reliance only for replica shards
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Taken cared off with #5282.


Documentation query: Sharing files and folders

The wiki page on sharing files and folders between ChromeOS and the chroot gives a method to share an (encrypted) home directory between the chroot and the ChromeOS host.

encrypted/home ~

It is not clear, however, what encrypted means in this context.
I had expected it would mean the same as it does for the chroot: that I would have to decrypt the directory by entering some sort of password before being able to use it. However, it appears not to be the case.
It would be helpful to improve the documentation, even if it just involves adding a link to an existing description. How do the encrypted/ directories work?
Thanks!

@dbaynard,
You're right, the documentation could be more clear for sure, if I understood it well enough I'd attempt it but I don't.
IIUC, the part about mapping outside the chroot that says these are the only folders crouton recognizes:
downloads: ~/Downloads
shared: /mnt/stateful_partition/crouton/shared
encrypted: ~/crouton/shared

The encrypted: ~/crouton/shared part is talking about creating this folder in CrOS:

/home/chronos/user/crouton/shared

If you creat a sub-folder called home it can be accessed by your chroot(s) in your home folder '~/' if you add this entry the the chroot's /etc/crouton/shared file -

encrypted/home ~

I know that's not much clearer at all but it might help to look at the source code startng here: enter-chroot#L404, it helped me.
Hope this helps,
-DennisLfromGA

Thanks @DennisLfromGA. That was helpful. It treats the ~/crouton/shared folder just as the ~/Downloads folder. And so I've found #3329 (which I had seen, but didn't understand).
So to clarify:
So let's say I set up a (default) crouton chroot. Any logged in ChromeOS user may decrypt and log in to that chroot?
Then, of the three directory roots:
downloads: ~/Downloads
shared: /mnt/stateful_partition/crouton/shared
encrypted: ~/crouton/shared

downloads is encrypted using the ChromeOS user profile — meaning it is avaliable iff that ChromeOS user is logged in
encrypted is the same
shared is available to any ChromeOS user who is logged in

In no case do any of those three directories need decrypting in the same way the chroot does?

So let's say I set up a (default) crouton chroot. Any logged in ChromeOS user may decrypt and log in to that chroot?

You can access the chroot even without logging in, by going into guest mode or using frecon (ctrl-alt-f2).  If you've encrypted your chroot, they'll need a password (and the keyfile if you're stored it elsewhere).

downloads is encrypted using the ChromeOS user profile — meaning it is avaliable iff that ChromeOS user is logged in

yep.  The Downloads folder also comes with the feature that ChromeOS may delete old files from there if you start to run out of disk space.

encrypted is the same

yep!  But this one isn't subject to automatic clean-up.

shared is available to any ChromeOS user who is logged in

similar to the chroot itself, you can also access it without logging in.

In no case do any of those three directories need decrypting in the same way the chroot does?

that's correct.  They're all stored separately from the chroot and exist in folders managed one way or another by ChromeOS.

Thanks, @dnschneid! I'll put the detail in the wiki and then close the issue.

Closing as I've edited the wiki. Thanks!


Current project status

Hi, what is the current status of this project? Is it still maintained?

@xtianmiller ?


Revisit behavior of registry access failures during stack activation

Two components are currently relying on the "digest at activation" from the Operator,  pipelines and the command line.   I want to revisit behavior of registry access failures during stack activation.
When this error occurs, it makes it very difficult for the consuming components to determine the appropriate behavior in all the cases and give meaningful error messages to the user, since the issue is occuring in another component outside of the pipelines.
My desire is that we fail activation of the stack(s) if the registry cannot be accessed at that time.

There was an issue opened against pipelines that was caused by this issue:  https://github.com/kabanero-io/kabanero-pipelines/issues/378

I guess the tactical solution here is to fail stack activation if we could not retrieve the digest.
The better solution is to inspect the governance policy and allow the stack to activate if the configuration doesn't care about what the digest is.  I'm not sure if the pipeline will be able to handle that.  It will require the Stack to know what's in the Kabanero CR instance.

I agree with @kaczyns, if Governance policy is activeDigest or strictDigest we should fail the activation. In other cases, we should let the activation and pipelines to be successful. There could be reasons when Champ does not care about Governance.

merged, closing

@kaczyns @mtamboli With respect to activating a stack if the current stackPolicy would not be impacted by the lack of an activation digest, I think the issue here is that the stackPolicy can be altered at any time. I am sure that you could retry the digest if they changed it, but then what would the operator do if it couldn't retrieve the digest?  Disallow the stackPolicy change?
The container image registry actually contains the stack images that are used for building the applications.  If the image registry is not accessible at any point,  is the stack really active?  Even if we sidestep all stackPolicy issues, the image registry has to be accessible at all times so that we can build applications on them.

@smcclem I agree with you at the 10,000 foot level, but configuring the operator to retrieve the digest and configuring the pipeline to retrieve the image are two different things, configured two different ways with two different secrets (at least last I checked).  I am saying that we should not fail stack activation just because the operator was not able to read the digest, if the governance policy would not have what the digest was anyway.
If the governance policy changed then the stack controller would have to go retrieve all the stack image digests, and if it couldn't retrieve one, then that stack version would transition from active to error (or inactive or whatever we decided).
I'm trying to save champ the work of creating and labeling/annotating a second secret for the operator to use, it the user doesn't care about the digest.
@stephenkinder maybe you should weigh in here.  I'm happy to back down if I'm being stupid.

@kaczyns Yes, configuring the secrets for the operator and pipelines are two different things.  We talked about a unified registry access service, but I don't think that's even possible (since we don't control access registry access directly in every case i.e. buildah),  but at best I would like to see singular place / service to obtain the configured credentials from.
Just as an FYI, once this goes in :  https://github.com/kabanero-io/kabanero-pipelines/pull/386 , the pipelines behavior will be:
If the stackPolicy = none,  The operator is not queried for anything.  For any other stack policy, I query the operator for only status="active" stacks, but I do expect the version and digest to be present and valid.  I am not currently validating that the digest is not empty, I assume active = digest is valid and present.  It will use an empty string.


getWindowHandles returns "500 internal server error" with excel

Hi,
we are facing a strange issue while automating microsoft excel with winappdriver. While trying to work around the splash screen with excel we are seeing "500 internal server error" while trying to fetch the window handles. I have mentioned the code, webdriver trace and the server logs below:
                DesiredCapabilities capabilities = new DesiredCapabilities();
		capabilities.setCapability("app", "C:/Program Files (x86)/Microsoft Office/root/Office16/EXCEL.EXE");
		excelSession = new WindowsDriver(new URL("http://<IP_ADDRESS>:4723"), capabilities);
		excelSession.manage().timeouts().implicitlyWait(1, TimeUnit.SECONDS);

		/** Work around to ignore the splash screen */
		Thread.sleep(5000);
		allWindowHandles = excelSession.getWindowHandles();
		excelSession.switchTo().window(allWindowHandles.iterator().next());

		/** open a blank document */
		excelSession.manage().window().maximize();
		excelSession.findElementByAccessibilityId("AIOStartDocument").click();

Webdriver exception:
org.openqa.selenium.WebDriverException: Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
System info: host: 'DELL3480-2241', ip: '<IP_ADDRESS>', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_211'
Driver info: io.appium.java_client.windows.WindowsDriver
Capabilities {app: "C:\Program Files\Microsoft..., javascriptEnabled: true, platform: WINDOWS, platformName: WINDOWS}
Session ID: B0747E7B-6F1B-4282-9374-2B0D3A0EAF0D
Winappdriver logs:
POST /session/B0747E7B-6F1B-4282-9374-2B0D3A0EAF0D/timeouts HTTP/1.1
Accept-Encoding: gzip
Connection: Keep-Alive
Content-Length: 38
Content-Type: application/json; charset=utf-8
Host: <IP_ADDRESS>:4723
User-Agent: selenium/3.141.59 (java windows)
{
"type": "implicit",
"ms": 1000
}
HTTP/1.1 200 OK
Content-Length: 63
Content-Type: application/json
{"sessionId":"B0747E7B-6F1B-4282-9374-2B0D3A0EAF0D","status":0}
Selenium and java-client:

org.seleniumhq.selenium
selenium-java
3.141.59

io.appium
java-client
7.0.0

org.seleniumhq.selenium
selenium-java

This happens only with specific set of windows machine where as in few other machines we were not able to reproduce this issue. And with in the system where this issue was reported when we clean, close and reopen the server and excel and try again, it works 1 out of 10 times.

I have the same issue with Outlook. It was working fine on the machine and now give this error and I Can't seem to get past it. Can someone please have a look into this


Getting several 'Module not found: Error: Can't resolve 'lodash-es' errors when compiling

I have installed it without any problems but as soon as I add it to any vue file and run npm I get several errors outputted such as. I am using Laravel mix.
ERROR in<EMAIL_ADDRESS>10:0-35
Module not found: Error: Can't resolve 'lodash-es/findKey' in
Did you mean 'findKey.js'?
BREAKING CHANGE: The request 'lodash-es/findKey' failed to resolve only because it was resolved as fully specified
(probably because the origin is strict EcmaScript Module, e. g. a module with javascript mimetype, a '*.mjs' file, or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.

Using:
Laravel 9.11
VueJS 3.2.31
Tailwindcss: 3.0.0

It seems the solution is to use the preview version.
"@protonemedia/inertiajs-tables-laravel-query-builder": "^2.0.0"


Trim ZWeb3

This PR trims ZWeb3 by removing all the functions which are just wrappers around web3 functionality. I additionally made ZWeb3.eth a getter (it used to be a function) so that the syntax for using it is nicer (without parentheses).
For example ZWeb3.accounts() has been removed and it now becomes ZWeb3.eth.getAccounts().
I've also removed many tests that were only testing the wrapped functionality, because they were just testing the web3 dependency. We might want to keep them though, given that web3 is kind of unstable, but IMO they would not be enough to detect any significant issues.
Do we consider this a breaking change?

I don't consider this a breaking change, but I'm reluctant to removing many of the methods from ZWeb3 that would allow us to switch to a different library in the future (if we need to). Let's discuss it live.


feat: implement GraphQL Client (WIP)

Description

replace manual GraphQL structs with derived graphql_client::GraphQLQuery

I was having trouble acquiring the appropriate query data from GitHub while pushing the derived queries from graphql_client through to the octocrab::instance().graphql() method.
The response contains only errors from GitHub (not data). So the octo.graphql method is accepting the serde_json::to_value(QuerySearch::build_query) result, but I just cannot figure out how to pass it in in a way that GitHub is okay with.
Query Struct
#[derive(graphql_client::GraphQLQuery)]
#[graphql(
    schema_path = "schema.docs.graphql",
    query_path = "query_1.graphql",
    response_derives = "Serialize,Debug"
)]
struct QuerySearch;

Calling Code
    let query_string: String = format!(
        r#"repo:{org_name}/{repo_name} is:pr created:>{date_str}"#,
        org_name = org_name,
        repo_name = repo_name,
        date_str = date_str,
    );

    let q = QuerySearch::build_query(query_search::Variables {
        query_string: query_string,
    });
    let q_body = serde_json::to_value(&q)?;
    let q_pretty = serde_json::to_string_pretty(&q_body).unwrap();

    println!("{}", &q_pretty);

    let octo = octocrab::instance();
    let response: Response<query_search::ResponseData> = octo.graphql(&q_pretty).await?;
    println!("{:#?}", response);

This is what println!("{}", &q_pretty); prints:
{
  "operationName": "QuerySearch",
  "query": "query QuerySearch($query_string: String!) {\n    search(query:$query_string, type:ISSUE,last:1) {\n        issueCount\n    }\n}\n",
  "variables": {
    "query_string": "repo:rust-lang/rust is:pr created:>2021-05-20"
  }
}

println!("{}", response); prints:
Response {
    data: None,
    errors: Some(
        [
            Error {
                message: "Parse error on \"operationName\" (STRING) at [2, 3]",
                locations: Some(
                    [
                        Location {
                            line: 2,
                            column: 3,
                        },
                    ],
                ),
                path: None,
                extensions: None,
            },
        ],
    ),
}


VERSION: Bump to 0.3.1

Signed-off-by: Fernando Fernandez Mancera<EMAIL_ADDRESS>

Merge this after releasing 0.3.0


[Feature request] Automatically identify host device

Is your feature request related to a problem? Please describe.
There exist a multitude of OS specific releases of RaspberryMatic.

It is sometimes difficult to asses the host hardware without physical access.
Users are often unaware that they could check the host device by

Enabling SSH in the frontend
Querying /proc/device-tree/model

Example for my device:
# grep Model /proc/device-tree/model
Raspberry Pi 2 Model B Rev 1.1

This makes the OS update process unnecessarily difficult and error prone.
Describe the solution you'd like
Automatically query /proc/device-tree/model for the type of host device RaspberryMatic is running on.
Present the user with the appropriate link to the OS update package for their specific model.
Bonus:
Automate the download and installation of updates.

This refs and is a partly a duplicate of #616

The related issue #616 has a significantly larger scope than what I am proposing here.
A small script that queries the device-tree and then displays the Model information on the frontend would be a nice first step.
A link to the binary could be a second.
The Bonus (see previous comment) is partially duplicated in #616.
I'd implement this myself, but I can't find any documentation on the architecture of RaspberryMatic.
This is all that I could find.
Am I looking in the wrong places?
Some pointers would be nice.

@jottr Sorry, but instead of simply adding a link and/or device model info the WebUI, I would like to give #616 more priority as I feel if we touch this area now anyway we should better go for the bigger than and integrate it nicely. Users have lifed with it so long already and given they know the hardware they are using simply displaying a link and/or device tree model info doesn't help a lot. Furthermore, with the upcoming 64bit version of RaspberryMatic RaspberryPi2 users have to switch installation archives and reinstall their SD card anyway. Thus, no easy way to go.
And last not least I have heard that eQ3 is working on such an auto-download mechanism themself. Thus, we should probably better wait what they will come up with in the next upcoming firmware updates.
So sorry, but I still see no benefit in working on this right now.

Thank you for the insight!
Where can I find further information on the architecture?
It is quite difficult to understand all the moving parts without prior knowledge of the project.

What do you mean exactly about "architecture"? RaspberryMatic is using buildroot/Linux for the OS stuff, and OCCU for the homematic/CCU stuff which is published by eQ3. And sorry, no big time for documentation here (as usual). So feel free to ask specific questions (but probably better in the german speaking HomeMatic Forum, if possible).


More specific TemplateStringsArray type for tagged templates

This introduces a more specific type over TemplateStingsArray when invoking a tagged template. This would then allow developers to interpolate the literal types provided to the tagged template invocation, along with the provided arguments:

type Interpolate<T extends readonly string[], A extends any[], R extends string = ''> =
    T extends readonly [infer TH extends string, ...infer TT extends readonly string[]] ?
        A extends [infer AH extends string, ...infer AT extends string[]] ?
            Interpolate<TT, AT, `${R}${TH}${AH}`> :
            Interpolate<TT, [], `${R}${TH}`> :
        R;

declare function interp<
  T extends TemplateStringsArray,
  A extends string[]
>(strs: T, ...args: A): Interpolate<T, A>;

const a = interp`a${"b"}c`; // "abc"
const b = interp`1${2}3`; // "123"

The type we manufacture for the tagged template invocation is an instantiation of TemplateStringsArrayOf<Cooked, Raw>:
type TemplateStringsArrayOf<
  Cooked extends readonly string[],
  Raw extends readonly string[]
> =
  Cooked & { readonly raw: Raw };

This would then allow you to perform custom interpolation against either the "cooked" values (where escapes are normalized), and "raw" values (where escapes are preserved):
declare function raw<
  T extends TemplateStringsArray,
  A extends string[]
>(strs: T, ...args: A): Interpolate<T["raw"], A>;

const a = raw`a\n${"b"}\nc`; // "a\\nb\\nc"

In theory, more complex interpolation could also be achieved, but that is an exercise left to the reader:
type QueryResult<T extends readonly string[], A extends any[]> =
   // some type that parses T and interpolates A...
  todo;

declare function query<T extends TemplateStringsArray, A extends any[]>(strs: T, ...args: A): QueryResult<T, A>;

interface Person { firstName: string, lastName: string }
declare const people: Iterable<Person>;

const result = query`
  for x in ${people}
  where x.firstName === ${"Bob"}
  select x.lastName
`;

result; // Iterable<string>

To support interpolation over a TemplateStringsArray, I also needed to make a change to inferFromObjectTypes. Prior to this change you could not infer to tuple element positions if the source type was an intersection of a tuple and another type:
type X<T extends any[]> = T extends [infer H, ...infer _] ? H : never;

// before
type T1 = X<[1, 2]>; // 1 (ok)
type T2 = X<[1, 2] & { foo: true }>; // never (😞)

However, this becomes a problem when trying to pick apart a generic TemplateStringsArrayOf type. To address this, this PR makes a small change in how we perform this inference. Nothing changes when a source type is already an array or tuple type, but if the source type is instead an intersection of an array or tuple type and another type, we will also now permit tuple element inference as long as the intersected types do not shadow tuple-specific properties (i.e., numeric indexers, numeric-stringed properties, or the "length" property):
type X<T extends any[]> = T extends [infer H, ...infer _] ? H : never;

// after
type T1 = X<[1, 2]>; // 1 (ok)
type T2 = X<[1, 2] & { foo: true }>; // 1 (ok)

Fixes #33304
Fixes #31422
Related #29432
Related #16551

I tinkered with an example of a tail-recursive parser for jest.each here, though it can probably be drastically simplified.

Here's a much simpler version of Jest's each:
type IDStart =
    | "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" | "j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" | "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z"
    | "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" | "I" | "J" | "K" | "L" | "M" | "N" | "O" | "P" | "Q" | "R" | "S" | "T" | "U" | "V" | "W" | "X" | "Y" | "Z"
    | "$" | "_"
    ;

type IDPart =
    | "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
    | IDStart
    ;

type Trim<S extends string, Chars extends string = " " | "\t" | "\v" | "\n"> =
    S extends (Chars extends "" ? S : never) ? S :
    S extends `${Chars}${infer R}` ? Trim<R, Chars> :
    S extends `${infer R}${Chars}` ? Trim<R, Chars> :
    S;

type ParseHeaders<S extends string> =
    S extends "|" ? [] :
    S extends `${infer H}|${infer T}` ? [Trim<H>, ...ParseHeaders<T>] :
    [Trim<S>];

type CheckHeader<TIn extends string, TWork extends string = TIn, Valid extends IDPart = IDStart> =
    TWork extends `${infer Ch}${infer R}` ?
        Ch extends Valid ? CheckHeader<TIn, R, IDPart> :
            JestEachParseError<`Invalid identifier: '${TIn}'`> :
    TIn extends '' ? JestEachParseError<`Identifier expected.`> :
        TIn;

type CheckHeaders<TIn extends string[], TWork extends string[] = TIn> =
    TWork extends [infer TH extends string, ...infer TT extends string[]] ?
        CheckHeader<TH> extends infer E extends JestEachParseError<any> ? E :
            CheckHeaders<TIn, TT> :
        TIn;

type ParseRows<A extends any[], S extends readonly string[], Row extends any[] = [], Rows extends any[][] = []> =
    [A, S] extends [[infer AH, ...infer AT], readonly [infer TH extends string, ...infer TT extends string[]]] ?
        Trim<TH, " " | "\t" | "\v"> extends "|" ? ParseRows<AT, TT, [...Row, AH], Rows> :
        Trim<TH, " " | "\t" | "\v"> extends "\n" | "" ? ParseRows<AT, TT, [], [...Rows, [...Row, AH]]> :
            JestEachParseError<`Unexpected character: '${Trim<TH>}'`> :
    [A, S] extends [[], readonly []] ? Rows :
        JestEachParseError<`Mismatched elements`>;

type JestEachArgument<Headers extends string[] | JestEachParseError<any>, Rows extends any[][] | JestEachParseError<any>> =
    Headers extends string[] ?
        Rows extends any[][] ?
            {
                [P1 in keyof Rows]: {
                    [P2 in keyof Headers as P2 extends `${number}` ? Headers[P2] : never]:
                        P2 extends keyof Rows[P1] ? Rows[P1][P2] : undefined;
                };
            }[number] :
            Rows :
        Headers;

type JestEachFunction<Arg> =
    Arg extends JestEachParseError<any> ? Arg :
    (name: string, cb: (arg: Arg) => void, timeout?: number) => void;

declare const JestEachParseError: unique symbol;

interface JestEachParseError<Message extends string> {
    [JestEachParseError]: Message;
}

type JestEach<T extends readonly string[], A extends any[]> =
    T extends readonly [infer TH extends string, ...infer TT extends readonly string[]] ?
        JestEachFunction<JestEachArgument<CheckHeaders<ParseHeaders<TH>>, ParseRows<A, TT>>> :
        JestEachParseError<`Mismatched elements`>;

declare function each<T extends readonly string[], A extends (string | symbol | number | bigint | boolean | null | undefined | object)[]>(strs: T, ...args: A): JestEach<T, A>;

// jest
it.each`
    foo    | bar
    ${"a"} | ${"b"}
    ${"a"} | ${"c"}
`("test", ({ foo, bar }) => {
//           ^? any
//                ^? any
});

each`
    foo    | bar
    ${"a"} | ${"b"}
    ${"c"} | ${"d"}
`("test", ({ foo, bar }) => {
//           ^? "a" | "c"
//                ^? "b" | "d"
});

There is no longer any order dependence. The behavior for inferring from an intersection of two tuple types now falls back to pre-existing behavior.

Also, I've added a test for Jest's each based on the example above.

@DanielRosenwasser: Are we comfortable taking this for 4.8 or would you rather wait?

@typescript-bot perf test
@typescript-bot run dt
@typescript-bot test this
@typescript-bot user test this

@typescript-bot perf test

reference playground

This is a feature I've been waiting for a long time, but I forgot why. 😄

Hi, this PR looks fantastic (I've been hoping for this feature for a long time)! Wondering what the status is?

Is there any progress for this feature?

I hope the pending reviews can be done soon so we will see this released soon. This would open the door to a lot of interesting things.

This PR is currently on hold. The main use cases for this feature require a fair amount of complexity that is likely to run any implementation against the recursion depth limiter. Such types are expensive to compute and check, and often result in degraded performance in an editor. It's unlikely that we will ship a feature that introduces this much complexity without first making significant improvements in overall type checking performance.
I do plan to keep this PR alive and will periodically update it, but I do not foresee it shipping in the near future.

Another year has passed, and version 5.x has now been released. This version has optimized the performance of comparison. Is it possible to consider introducing this feature in the new version? This helps us a lot, thank you very much for your work.

i wrote a working type level parser for my library's html template string DSL so it can be typesafe but it's useless without this PR. JSX isn't an option due to lack of ability to cache static parts with default tooling. i'd love to see this merged.

@rbuckton Any updates? Aren't v5's perf enhancements sufficient now?

@rbuckton Any updates? Aren't v5's perf enhancements sufficient now?

No, unfortunately. The performance concerns we have are related to the complex conditional types and inference that are necessary to "parse" a string literal type, and that still isn't very efficient at the moment.

@rbuckton Has ArkType not proven it can be done efficiently at this point?
Even with the overhead of type-level string parsing (including syntactic and semantic validation), types are often 3-10x less expensive than equivalent Zod.
It would be a shame not to ship what would be an awesome feature just because it is possible to use inefficiently.

We need this merged yesterday ❤ Typescript users are harassing me into changing my library APIs to be different just because they can't do what this PR would allow them.
Why not let the resposibility of making fast implementations, and avoiding slow ones be up to the users?

This PR is currently on hold. The main use cases for this feature require a fair amount of complexity that is likely to run any implementation against the recursion depth limiter. Such types are expensive to compute and check, and often result in degraded performance in an editor. It's unlikely that we will ship a feature that introduces this much complexity without first making significant improvements in overall type checking performance.
I do plan to keep this PR alive and will periodically update it, but I do not foresee it shipping in the near future.

@rbuckton Any updates? Aren't v5's perf enhancements sufficient now?

No, unfortunately. The performance concerns we have are related to the complex conditional types and inference that are necessary to "parse" a string literal type, and that still isn't very efficient at the moment.

FWIW, @jantimon's use case, as well as a similar use case I'm working on regarding SQL queries, do not make use of conditional types or literal type parsing. If I understand him correctly, the use of literal types we have in mind boils down to generating overloads, like so:
function template(strings: ["select ", " + 1"], parameter: number): number;
function template(strings: ["select ", " || ' world!'"], parameter: string): string;
...

No literal parsing required. I'd argue that tools like this will be the main use cases in the near future, since users will avoid libraries that do complex parsing within TypeScript because they are too slow.

What about those of us who just wants a way to map types against a tagged template literal function starting with specific words?

What about those of us who just wants a way to map types against a tagged template literal function starting with specific words?

It sounds like that would be supported by this PR. And potentially simple enough that it wouldn't run into TypeScript performance constraints, and so, under my theory, be adopted widely by users and be one of "the main use cases", unlike more complex use cases that may need to wait for a more performant TypeScript to gain adoption.
My point is that performance is an important part of DX, and this is something that library authors and users already have to take into account. For example, uom for Rust is an awesome library, but it is not as widely used as it could be because it has quite slow compile times.

"npm<EMAIL_ADDRESS>
Is it possible to release this pr with 5.7?


[AWS::SSM::Association] - [BUG] - cfn-lint doesn't recognize Values: expected type: JSONArray, found: String

CloudFormation Lint Version
cfn-lint 0.87.1
What operating system are you using?
Windows 11
Describe the bug
cfn-lint doesn't recognize the following error message:
Resource handler returned message:
"Model validation failed (#/Targets/0/Values: expected type: JSONArray, found: String)"
(RequestToken: REQUESTTOKEN, HandlerErrorCode: InvalidRequest)

for the following example:
  SSMAssociation:
    Type: AWS::SSM::Association
    Properties:
      Name: "SSM Document Name"
      ScheduleExpression: rate(2 days)
      Targets:
        - Key: InstanceIds
          Values: !GetAtt EC2Instance.InstanceId

This is how it should look like:
  SSMAssociation:
    Type: AWS::SSM::Association
    Properties:
      Name: "SSM Document Name"
      ScheduleExpression: rate(2 days)
      Targets:
        - Key: InstanceIds
          Values:
            - !GetAtt EC2Instance.InstanceId

Expected behavior
cfn-lint finds this error.
Reproduction template
See above.

Hmm, should be able to find this one.  I wonder if we aren't handling JSON values well.  I'll look into this.

Already fixed in v1 but is a bug in v0.


Refactor bible command to handle long passages using chunked responses

This Pull Request modifies the code to implement chunking for the /bible command. In simple words, it splits everything into chunks (up to 5), with each being 4000 characters at max. Please let me know if something is wrong or if you'd like something changed.

I'll make sure to review everything & restart the bot if it works

Okay then, good luck. Happy to help! :)

@Blue-Robin-Taken Applied the changes, please have a look.

Thanks!


How make confirm ALL (cs:go/dota/etc) trades?

I wanna to confirm all trades from my account in ArchiSteamFarm. How make?

Learn to read.
Read the f* manual.

Read.
Enable MatchEverything in TradingPreferences  (https://github.com/JustArchi/ArchiSteamFarm/wiki/Trading) result - Ignored Trades

cs:go/dota/etc trades

SteamTradeMatcher

y u no read?

Please review the guidelines for contributing to this repository.

https://github.com/JustArchi/ArchiSteamFarm/blob/master/CONTRIBUTING.md

If my issue is not meeting contributing guidelines specified above, especially if it's a question or technical issue, please close my issue immediately - I won't complain.


Remove final from methods

Please remove final from AsciiTable.add___() methods.  The final keyword prevents me from overriding the class with default settings I want to set on each row.

What you want is default settings! That's a missing feature. There should be an easy way to set default contexts for rows and cells, inherited by all added rows/cells. I'll look into how to add this feature and keep this issue open until then.
For the final methods: The final on "addRow(...)" is to protect the table's contract in the provided implementation. You can build/provide your own implementation for the table by either implementing IsTable or extending AsciiTable with your own "addRule(...)" methdods, but overwriting the basic add rule methods should not be necessary.
thanks/


Replacement for glyph.isEmpty.

The isEmpty method has been deprecated because it is too vague. In ye olden days, it simply stated if a glyph had contour or component data. But, guidelines, anchors, lib and more are also data, so it's ambiguous. Testing for the presence of an outline is beneficial to scripters. They can get the same result as isEmpty with not glyph.contours and not glyph.components but that is very verbose. So, we need a convenience property that indicates if outline data is present.
Candidates so far:

glyph.outline
glyph.hasOutline
glyph.hasGeometry
glyph.anything
glyph.hasVisibleData
glyph.emptyOutline
glyph.visible
glyph.paints

isEmpty is just fine. Document that it's about the outline and be done.

Suggesting isClear

Unnecessary change. Is all I'm saying.

Okay. Let's bring back isEmpty().

hurray! should reflect outlines (contours and components) only

Should this be a property, not a method?

See #280

Closed by #280


Handle big text ingestion

Recently, I saw below error and feel that we need a mechanism to handle large docs
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:439} WARNING - Error during upsert. Rolling back all inserts for docs with errors.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:328} INFO - Removing id f9c660b6-a393-54ef-9f95-20a616c3edfe for rollback.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:333} INFO - UUID 767b916a-f6ef-51e1-bf23-cea43a801baa does not exist. Skipping deletion during rollback.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:328} INFO - Removing id c7ccb038-c35a-501e-8404-1ce26782f0f7 for rollback.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:328} INFO - Removing id 8a084e49-5e06-5827-8165-69c05d29716e for rollback.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:328} INFO - Removing id 9a413399-6b68-5660-99d8-4bda030f9496 for rollback.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:467} ERROR - Errors encountered during ingest.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:468} ERROR - <generator object AskAstroWeaviateHook.ingest_data.<locals>.<genexpr> at 0x7fc00df27e00>
[2023-12-19, 05:42:31 UTC] {taskinstance.py:1937} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/airflow/decorators/base.py", line 221, in execute
    return_value = super().execute(context)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/operators/python.py", line 192, in execute
    return_value = self.execute_callable()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/operators/python.py", line 209, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/include/tasks/extract/utils/weaviate/ask_astro_weaviate_hook.py", line 469, in ingest_data
    raise AirflowException("Errors encountered during ingest.")
airflow.exceptions.AirflowException: Errors encountered during ingest.

https://clmkpsyfc000301kcjie00t1l.astronomer.run/d2uulcqu/dags/ask_astro_load_airflow_docs/grid?dag_run_id=scheduled__2023-12-18T05%3A00%3A00%2B00%3A00&task_id=ingest_data&tab=mapped_tasks&map_index=0

I handle in case forum by truncating it. not a perfect solution just in case want to check
https://github.com/astronomer/ask-astro/blob/441def6bb62ccdad2e50d35a65a8b4a71a6e26a4/airflow/include/tasks/extract/astro_forum_docs.py#L88-L109

On a related note, I also noticed some of our inserts/upserts have issues with having too long of a chunk being attempted to be ingested. see error log below. This needs further investigation as it could mean some docs aren't properly ingested
[2024-01-16, 05:02:04 UTC] {weaviate.py:430} INFO - Error occurred in batch process for 00d97bbe-cec4-51bc-b0d2-11f1326f167e with error {'error': [{'message': "update vector: connection to: OpenAI API failed with status: 400 error: This model's maximum context length is 8192 tokens, however you requested 20585 tokens (20585 in your prompt; 0 for the completion). Please reduce your prompt; or completion length."}]}
[2024-01-16, 05:02:04 UTC] {weaviate.py:430} INFO - Error occurred in batch process for d1eace29-205f-53bf-9fc5-6645f133877c with error {'error': [{'message': "update vector: connection to: OpenAI API failed with status: 400 error: This model's maximum context length is 8192 tokens, however you requested 28319 tokens (28319 in your prompt; 0 for the completion). Please reduce your prompt; or completion length."}]}
[2024-01-16, 05:02:04 UTC] {weaviate.py:430} INFO - Error occurred in batch process for a82bb7bc-f2fc-5698-ad79-c28185a33e82 with error {'error': [{'message': "update vector: connection to: OpenAI API failed with status: 400 error: This model's maximum context length is 8192 tokens, however you requested 13851 tokens (13851 in your prompt; 0 for the completion). Please reduce your prompt; or completion length."}]}
[2024-01-16, 05:02:04 UTC] {weaviate.py:430} INFO - Error occurred in batch process for 9075559f-b0ce-5d8f-8162-28fd974a99e1 with error {'error': [{'message': "update vector: connection to: OpenAI API failed with status: 400 error: This model's maximum context length is 8192 tokens, however you requested 15193 tokens (15193 in your prompt; 0 for the completion). Please reduce your prompt; or completion length."}]}
[2024-01-16, 05:02:04 UTC] {weaviate.py:440} INFO - Total Objects 6066 / Objects 6009 successfully inserted and Objects 22 had errors.


[DOC]: Missing full-stop in the doc area

Type
Docs Bug Report
Feedback
I noticed that there is a missing full stop at the end of the message under the SSI Service API. Could you please consider me  adding it for proper punctuation and completeness?

I am thinking that there should be a full-stop after Verifiable Credentials
Screenshots
Paste relevant images or screenshots here
developer.tbd.website URL
DOCS page
Browser
Microsoft Edge
Relevant log output
No response

please can I work on it.

@philling1 🥳 it's been assigned to you, another great catch.

Ok thank you, let me get on with it.

I have opened a PR for this changes #826 and I have remove the Unwanted file from my previous PR on this issue. Please help me to review it and see if there is any further changes to my PR.

I don't know how I should remove this file  "site/src/util/code-snippets-map.json" from my PR could you please direct me on how to do that.


gist to install grpc + cgrpc

I'm using this Swift3 dev dockerfile
https://gist.github.com/johndpope/01a74ffb93921df48e334097117be1b3
Gist to install gprc
https://gist.github.com/johndpope/503029706ed56d5375d1e9469f0135d4
Patch out of date?
cd ~/
git clone https://github.com/Huawei-PTLab/CGRPC
cd /usr/local/include/
sudo patch -p1 < ~/CGRPC/grpc-header-hack.diff
(getting collision here on patch on sync.h)

cd ~/
git clone https://github.com/Huawei-PTLab/SwiftGRPC
swift build

it appears GRPC is working

it seems like the patch is causing problems
@@ -36,6 +36,7 @@
#include <grpc/impl/codegen/byte_buffer.h>
#include <grpc/impl/codegen/status.h>
+#include <grpc/impl/codegen/time.h>
#include <stddef.h>
/usr/local/include/grpc/byte_buffer.h:37:10: note: in file included from /usr/local/include/grpc/byte_buffer.h:37:
#include <grpc/impl/codegen/grpc_types.h>
^
/usr/local/include/grpc/impl/codegen/grpc_types.h:41:10: error: 'grpc/impl/codegen/time.h' file not found
there is a grpc++/impl/codegen/time.h file. Did you have an alias setup?


Coexistence with Java plugin

I ran into an issue when attempting to apply both the Java and Terraform plugins, and declaring an implementation dependency:
plugins {
    id("java")
    id("org.jetbrains.gradle.terraform") version "1.4.2"
}

repositories {
    mavenCentral()
}

dependencies {
    implementation("org.junit.jupiter:junit-jupiter-api:5.8.2")
}

This fails with the following error:
Could not create task ':generateMainExecutionContext'.
Could not resolve all files for configuration ':terraformMainRuntimeElements'.
No variants of org.junit.jupiter:junit-jupiter-api:5.8.2 match the consumer attributes:
  - org.junit.jupiter:junit-jupiter-api:5.8.2 variant samplessources:
      - Incompatible because this component declares a runtime of a component, packaged as a jar and the consumer needed a usage of 'terraform' of a component, with the library elements 'tf-zip-archive'

I guess it is because the Terraform plugin uses the implementation configuration (like the Java plugin does), and becomes confused when a non-Terraform project is added.
In the ideal world, I would like to have a main module, which depends on the required Java/Kotlin modules, and also describes the terraform structure needed to deploy the code, along with the appropriate Gradle tasks for doing so.
Is it intentional that the plugin can't coexist with the Java plugin?

Indeed it is intentional. I would recommend you to create separate modules, one for you Java project and one for Terraform. This would also make it easier to share lambdas.


Add clarification about how composite service discovery works.

In another one.
On Fri 11 Oct 2024 at 15:21, Aurea Muñoz Hernández @.***>
wrote:

@.**** commented on this pull request.
In docs/docs/service-discovery/composite.md
https://github.com/smallrye/smallrye-stork/pull/956#discussion_r1796950000
:

Remember to define the services that make up your composite service.

+Be aware that Stork doesn't work as a standalone service discovery cluster.
+Instead, it processes composite configurations—meaning it handles multiple configurations that may use different service discovery implementation.
+For each specific configuration, Stork delegates the service discovery task to the appropriate service discovery provider.
+If one of these providers doesn't respond or fails, it affects Stork's ability to resolve that specific configuration, but it doesn't mean Stork itself is faulty;
Yes, we can better manage the error case.
Do we do it in this PR? or shall we create an issue and do it in another
one?
—
Reply to this email directly, view it on GitHub
https://github.com/smallrye/smallrye-stork/pull/956#discussion_r1796950000,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADCG7IFX4TTRHVFA6MNHEDZ27GERAVCNFSM6AAAAABPYQGMI2VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGNRSG44TQMRVG4
.
You are receiving this because your review was requested.Message ID:
@.***>


API crashed when put bucket.

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:
/kind bug

What happened:
API crashed when put bucket:

What you expected to happen:
API should not crashed.
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:

Gelato(release/branch) version:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

It's fixed in PR#842


Is redux-zero compatible with React Native ?

Let's find out...

I started a new app yesterday and the counter example was working fine for me.

@guilhermefloriani could you please send this example in a PR? This will help a lot!
Please take a look at this issue #35


Select2 4.0 in bootstrap modal

Hi, I need to use select2 (multiple and single) with search in bootstrap modal, but when i try using it, i can't see options, it seems like select2 open, but modal hide them. How can i fix it?

This is what I did; it seems to work on 4.0.x:
element.on("select2:open",function(event){
			var modal = element.closest('.modal');
			if(modal[0]){
				var modalZ=modal.css('z-index');
				var newZ=parseInt(modalZ)+1;
				element.data('select2').$container.css('z-index',newZ);
				element.data('select2').$dropdown.css('z-index',newZ);
				element.data('select2').$element.css('z-index',newZ);
				element.data('select2').$results.css('z-index',newZ);
				element.data('select2').$selection.css('z-index',newZ);
			}
		});

I think this might depend on when you trigger the select2 creation to happen, I've not seen this issue myself but then I only create the select box when the modal is being called using the
    $('#myModal').on('shown.bs.modal', function () {
        $('#select').select2()
    })

script. That way it's always in the correct position I've found.

This is a duplicate of https://github.com/select2/select2/issues/600#issuecomment-102857595.

I use the way that cigaar propose. It work in Chrome, but in Firefox in search input don't focus, can anyone help with this?

fix using -  dropdownParent: "#modal_id" .

It's not works on Bootstrap 3.3.7


Updating benign smart card errors

Updating error 610 and 621 as benign since they do not impact end use of smart cards in most scenarios.

@Dani


adds objects for TB-3302

What 🕵 🔍

See TB-3302 in the main app
Adds the necessary xayn_design updates to support it

References 📝 🔗

TB-3302

Codecov Report

Merging #65 (30a8397) into main (2f33f1c) will decrease coverage by 0.17%.
The diff coverage is 63.63%.

@@            Coverage Diff             @@
##             main      #65      +/-   ##
==========================================
- Coverage   49.13%   48.96%   -0.18%
==========================================
  Files          54       54
  Lines        1974     1983       +9
==========================================
+ Hits          970      971       +1
- Misses       1004     1012       +8

Impacted Files
Coverage Δ

lib/src/linden/xstyles.dart
31.87% <0.00%> (-1.68%)
:arrow_down:

...b/src/widget/settings/data/settings_card_data.dart
80.00% <100.00%> (+5.00%)
:arrow_up:

lib/src/widget/settings/widget/settings_card.dart
100.00% <100.00%> (ø)

...b/src/widget/settings/widget/settings_section.dart
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f33f1c...30a8397. Read the comment docs.


How to use flash attention for inference speed boosting in bert like models

is flash attention support bert like models(bert,distilbert,roberta...etc) for reducing inference time latency?
If it supports, how to use flash attention for inference speed boosting in bert like models?
share the sample code

Please search for "bert" in the repo. If you don't put in effort I don't think you'll get help.


ci: adding validation checks for contracts.json

Adding contracts.schema.json
Adding github workflow to ensure that contracts.json matches the required schema

For testing it out,

Adding Liquidity Council instance address

Closed in favour of https://github.com/archway-network/contract-register/pull/3


update deps and specs

closes #16
@pwFoo can you verify this works

Hi @jessfraz,
I cloned the branch and built the docker image. Get the following error if I execute riddler:
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x560a3f8c4c2c]

goroutine 1 [running]:
panic(0x560a3fd022a0, 0xc420012060)
	/usr/lib/go/src/runtime/panic.go:500 +0x1a5
github.com/jessfraz/riddler/parse.Config(0xc4202d02c0, 0x560a3fe848c8, 0x0, 0x0, 0xc420316000, 0xc420318000, 0x560a3fa63d5b, 0x5, 0x560a3fa63cb1, 0x5, ...)
	/go/src/github.com/jessfraz/riddler/parse/config.go:261 +0xecc
main.main()
	/go/src/github.com/jessfraz/riddler/main.go:163 +0x17c

@jessfraz Maybe you want to swap to OCI's runtime-tools (https://github.com/opencontainers/runtime-tools) which provides a library to generate runtime configurations meaning you won't have to write all of the management code yourself.

yeah thats a good idea
On Fri, Apr 21, 2017 at 12:05 PM, Aleksa Sarai<EMAIL_ADDRESS>wrote:

@jessfraz https://github.com/jessfraz Maybe you want to swap to OCI's
runtime-tools (https://github.com/opencontainers/runtime-tools) which
provides a library to generate runtime configurations meaning you won't
have to write all of the management code yourself.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jessfraz/riddler/pull/17#issuecomment-296232567, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABYNbOYgXyLxB32_6tKobMgrZ3zhkCO_ks5ryNO8gaJpZM4NB12W
.

--
Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC  511E 18F3 685C 0022 BFF3
pgp.mit.edu http://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3

If you want, I can write up a PR for this in the coming days.

whoa thanks yeah!
On Fri, Apr 21, 2017 at 3:51 PM, Aleksa Sarai<EMAIL_ADDRESS>wrote:

If you want, I can write up a PR for this in the coming days.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jessfraz/riddler/pull/17#issuecomment-296290662, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABYNbPcz1K6M3y2DruLmEUEu9LMy3XrDks5ryQjVgaJpZM4NB12W
.

--
Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC  511E 18F3 685C 0022 BFF3
pgp.mit.edu http://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3