Improve topN algorithm

Description
@leerho proposed how the topN algorithm can be improved here: https://github.com/apache/incubator-druid/issues/7161#issuecomment-468890906
Motivation
Avoid potentially significant errors, see #1134 and #1135.

Objective

Replace the current internal heuristic algorithm for "TopN" with the DataSketches Frequent Items Sketch (FIS).

Motivation

Error vs. Size: The FIS has user configureable parameter that allows tradeoff of the sketch size and its resulting accuracy (or what we call its error threshold, or epsilon).
Error Clarity: The FIS has well-defined error bounds that can be applied to each item in the result set.
Query Flexibility: At query time the user can choose two different query modes NO_FALSE_POSITIVES (more strict) or NO_FALSE_NEGATIVES (less strict).  The FIS will not return items below the noise floor. So you know that the items it does return are, in fact, significant.
Streaming: The FIS is a streaming algorithm, just pass the input data into the sketch once.  No need to presort the input stream.  This will save processing cycles and time.
Mergeable: The FIS is mergeable.  Each node would sketch its own data independently, then the sketches from each node are passed to a central "broker" node where they are merged into a final sketch, which can then be queried. This could reduce the amount of data passing between the nodes and the broker, but it will depend on how the sketch is configured.  There is no loss of accuracy due to the merging process.
Data Independent: The FIS error properties are independent of how the input data is partitioned into separate streams (the nodes) and how the values and their weights are distributed within the streams themselves.  The FIS can be somewhat input data order sensitive, in that the weights and order of the items in the result set may vary slightly from run to run, but they will always be within the error specification of the chosen sketch configuration.

Making it Happen!
We on the sketches team are not knowledgeable about Druid internals.  We certainly can advise and consult on the proper use of the sketches, but it must be someone on the Druid team to integrate the sketch library into Druid.

It sounds like the behavior of FIS is different enough from topN that 'replace' is not in the cards. But it does sound like something that would be really nice to expose in Druid somehow. Maybe as a new query type? Can the 'weight' of an item be any Comparable object? (This would allow it to be compatible with any Druid aggregator.) Or is it limited to ints?

@gianm
I think we may have some confusion here.  FIS does not depend on Comparable at all, only Identity. And the weights are positive longs.   (The Quantiles sketch depends on Comparable).
Imagine you are Apple and have a stream of song title objects, each with a price as purchased.  Different instances of the same title may have a different price due to store discounts or whatever.
The task is to identify the largest revenue song titles, by title and total revenue.
The input stream is a stream of {Object songTitle, long purchasePrice} (Price in cents)
You create the sketch:
 ItemsSketch<SongTitle> fisSketch = new ItemSketch<SongTitle>(1024);
Note: the sketch size is configurable, depending on how accurate you want it to be.
You update the sketch with
while (remainingItems > 0) {
  SongTitle songTitle = next();
  fisSketch.update(nextItem, (long)(songTitle.getPrice()));
}

Note that the sketch does the item aggregation for you, and no need to sort the input.
The purchasePrice is the "weight".  When duplicate songTitles appear in the stream, their purchasePrice is accumulated in the sketch.
When all the sketches at each of the nodes are complete, you send the sketches to the Broker where they are all merged together.
The final merged sketch is then queried:
ItemSketch.Row<T>[] rows = getFrequentItems(ErrorType.NO_FALSE_POSITIVES);
Each row of the resulting array of rows contains:

A reference to the frequent Item
An Estimate of the total weight (total revenue) for that item
The upper bound of what the true total Weight is
The lower bound of what the true total Weight is.
plus a few other convenience methods.

The array is sorted by the accumulated weights, so they are in approximate rank order.  The sketch will return all items that have a weight above the threshold.
This is effectively a superset of the typical "TopN" heuristic as it does a lot more than the current heuristic, and it provides accuracy guarantees that the existing heuristic cannot.
Fundamentally the Druid "TopN" heuristic is data sensitive and it can fail without warning where this will not.
So I am not clear where you think would not replace the current functionality?  What is the functionality that you think is missing?
It should certainly be worth a try :)
Lee.

The current topN algorithm can be used for any query shaped like SELECT x, AGG(y) FROM tbl ORDER BY AGG(y) DESC LIMIT N when useApproximateTopN is enabled (when disabled, groupBy is used instead, which is exact).
It sounds like FIS is limited in a couple ways that prevents it from being used for all queries of that form:

"AGG" must be "SUM" and y must all be positive. (topN will accept any aggregation function and any y)
"LIMIT" will not necessarily be adhered to. (topN will always return LIMIT results, or the number of distinct values of x, whichever is lower)

Now, you might argue that this level of flexibility is bad, because it involves throwing caution to the wind, and sometimes falls flat on its face. And you would have a good point. But, nevertheless, the flexibility is there, and is even useful sometimes.
Maybe it makes sense for the FIS to be exposed via its own native query type ("frequentItems" instead of "topN"). If we do that, I'm wondering how it makes sense to expose it through SQL. Maybe a context flag like useFrequentItemsForTopN that would plan a SQL query like SELECT x, AGG(y) FROM tbl ORDER BY AGG(y) DESC LIMIT N as a "frequentItems" query if "AGG" is "SUM", and would be documented to relax "LIMIT" such that fewer than N items might be returned. I'm not sure how to satisfy the "y must all be positive" requirement; maybe we need to track metadata on the brokers about min/max values for columns, and only use the frequentItems query if y is known to all be positive.

I'm not sure how widely it's used but TopN's also supports ascending order. Curious if the sketch supports returning the inverse "most infrequent" items too?

Also curious on people's thoughts on user's being explicit in their queries that can return approximate results versus optimising queries based on context/configuration.
SELECT AGG(y), x FROM tbl where x in (SELECT FREQ_I_SKETCH(x, y, 100) from tbl)
versus trying to optimise a order by limit.

Also curious on people's thoughts on user's being explicit in their queries that can return approximate results versus optimising queries based on context/configuration.

That's probably better, given that FIS is doing a very particular thing, and SQL users might appreciate being able to trigger its activation explicitly.

This is very helpful and I think I see where we are talking past each other.  There appear to be several different use cases for the concept of "TopN".  The starting point for these use cases is after the selection and other possible SQL manipulation, where you end up with a set of distributed lists of items where the task is to efficiently merge together these lists and obtain the "TopN" of these items.  It is the next operations that follow that distinguish the several use cases.

Case 1:  The items are Comparable with respect to some property y, which does not have to be numeric.  It could be a string.   If y is numeric, it could be either positive or negative, it doesn't matter. The other significant property is that these items would not be aggregated together, e.g. it would not be meaningful to do an operation like ItemC = ItemA + ItemB.  To finish, the following operations would be performed:  (Assume N = 100).  Note that the "order" here could be either direction so that one could also obtain the "BottomN".
-- Order the items in each node .
-- Select top 100 from the list in each node (Note: we don't need more than N from each node)
-- Merge the lists together in the Broker
-- Order the items in the Broker, choose the top 100.

Case 2: The items have the Identity property such that one can check if ItemA == ItemB, and they have a weight (or importance) property that is independent of the identity property.  In other words, ItemA == ItemB and possibly ItemA.w1 != ItemB.w2.   This weight is numeric, positive and obviously comparable.  These objects can be "merged" together when the identity is satisfied as in ItemC.w3 =  ItemA.w1 + ItemB.w2 where w3 = w1 + w2.    (Assume finalN = 100).  The current algorithm would be the following
-- Order the items in each node .
-- Select top (nodeN=1000 from the list in each node (Note: we must select nodeN >> finalN)
-- Merge the lists together in the Broker
-- Order the items in the Broker, merging identical items, choose the top 100.
The sketch equivalent of this process would be
-- Feed the items in each node into a FIS.
-- Each node sends its sketch to the Broker.
-- The Broker merges the sketches together into a single sketch
-- The result "Most frequent Items" or "Heaviest Items" are returned from the sketch.

Case 3: This is the same as Case 2, except that negative weight values would be allowed. This, unfortunately cannot be solved with our current sketch anyway.   We will have to think about this.

I think this addresses all the use cases (this was kinda slapped together).  But for both cases 1 and 2, improvements can be made in how the final steps are performed. For case 3 I would like to ask: "Is this a real case and how often does it occur?

I should point out that Case 1 above is also streamable and solvable with significantly less resources than you are using now.  And I think we have a solution for Case 3 also, but that may require a bit more effort.

@leerho Going back to your song example, could FIS support getting the top song titles by number of unique listeners over the past week? The current topN can do that using HLL sketches as metric (weight). That's one example of the usefulness of accepting any Druid aggregation or post-aggregation as a metric. (Interestingly, this specific case could benefit from using a HLL Map sketch.)
Also, I don't think that the current topN algorithm orders all distinct items on each node (or segment) prior to selecting the top 1000, since that would be unnecessarily wasteful when the number of distinct items is >> 1000. Would be great if someone more familiar with the topN implementation could confirm.

getting the top song titles by number of unique listeners over the past week

@peferron Now that is very interesting!  The current FIS doesn't handle this case, but I think it could be extended to do that, or to have a configurable aggregate function.  I'll have to think about it.
I agree with you that having the ability to specify any Druid aggregation or post-aggregation as a metric is pretty powerful and I'm not trying to replace that functionality.
My principal concern is the way the final steps are performed to obtain the Top 100(for example) items:

for each node sort all items in the node // cost: n log n //where n = total items in the node
pass top 1000 items to the Broker //network cost
The Broker sorts (numNodes(M) * 1000 items): cost: M * 1000 log (M*1000)
The Broker chooses the top 100.

This heuristic approach is error prone, data sensitive and compute-wise costly.  This, I think, can be improved, but not by substituting FIS!
I am not familiar with how the pre- and post-aggregation steps intertwine themselves with this process.  But perhaps this heuristic could be accomplished with a Min-Heap?  It would be faster, smaller, not data sensitive, and generally more robust.  It would change the compute cost from a n log n to a n log k, which could be significant.
This (I realize now) has nothing to do with FIS.  As I tried to state above, that Druid's concept of "TopN" consists of several very different types of operations.  And the current FIS would only help in a subset of them.)
But where FIS could apply (many duplicates (with weights) in the stream and you wish to return the heaviest weighted items (or the most frequent items).  Now the the compute cost reduces to just n and makes it suitable for real-time queries.  This type of query is fairly common and it might deserve a special query type of its own.
The idea of a compound FrequentItem<Item, Aggregator> sketch sounds very interesting, but we don't have it now, so it would have to be developed.

@peferron
By the way, we do have an HLL Map sketch,  but it was designed with a different use-case in mind so doesn't quite fit what we need here.

@leerho As mentioned earlier, from my understanding of the topN code, there is no n log n sorting of all items in the segment (a node can contain multiple segments, which contain usually up to 5M rows each and can be processed in parallel; per-segment results are merged locally on the node, then the result is passed to the broker for another round of merging with other node results). The process is:

Aggregate values of all n items in the segment, grouping by item key. Time O(n) and space O(m) where m is the number of distinct item keys. Worst case is m = n so this can actually take a lot of space compared to FIS if cardinality is high.
Take top k items by aggregated value. This is currently done with a priority queue, so time O(m log k) and space O(k). Time could be improved to O(m) with quickselect but it might not make a difference in practice.

Regarding the HLL Map sketch, I was indeed referring to the one in the Yahoo Datasketches lib (should have made this clearer, sorry). If a topN includes a HLL aggregation, the current implementation keeps a dedicated HLL sketch for each distinct item key, so you can end up with millions of low-cardinality HLL sketches. It looks like the HLL Map sketch is designed to improve this scenario. But it probably would be quite a lot of work, as a naïve implementation that duplicates the set of item keys would waste most of the space gained.

Thanks for your clarifications.  I was led to believe that the TopN process was the one I described.  A PriorityQueue is a Min-heap so it sounds like it is already doing close to what I was suggesting.
As for the HLL-Map Sketch, one of its requirements was that it keep every key and its latest count estimate on every unique key that it has seen to allow arbitrary queries of any key.  This consumes a huge amount of space.  And there is no requirement for this here.  What I imagine would be a FIS-HLL combination that would be considerably smaller.  It’s size would be O(k).

@peferron

How is the group by (your step 1) performed under the covers?  Hash Table?  What do you do if n or m don't fit into memory?  (or is that out of scope by definition?)
"per-segment results are merged on the node".  How?  Merging of Priority Queues from each segment?   Size k? or bigger than k?
What exactly is passed from the node to the broker, effectively a Priority Queue? And how big is it WRT k?

How is the group by (your step 1) performed under the covers? Hash Table? What do you do if n or m don't fit into memory? (or is that out of scope by definition?)

The TopN engine's intra-segment group-by will use either an array (if it can) or hashtable (if it can't use an array). It assumes the m keys will fit in memory, which is considered to be a fair assumption since the highest it could reasonably be is a few million. (Creating segments of more than a few million rows is not considered a good idea.)

"per-segment results are merged on the node". How? Merging of Priority Queues from each segment?  Size k? or bigger than k?

All intra-segment resultsets (which are exact) are then sorted (using a priority queue) and truncated to max(k, 1000). Call that K. Then the historical merges all those per-segment resultsets pairwise by first combining the 2K elements based on the aggregation key, then sorting those 2K elements (again, using a priority queue) into a new priority queue of size K.

What exactly is passed from the node to the broker, effectively a Priority Queue? And how big is it WRT k?

It's a priority queue of size max(k, 1000).

@gianm
Where in this process is the "pre-aggregation" performed?
Where in this process is the "post-aggregation" performed?
And by "aggregation", I presume that means, for example, two identical items (each of weight 1) will be replaced by one of the items with a weight of 2.  Is that correct?
I think you may still have a problem, but I can't prove it yet :)

What do you mean by pre-aggregation and post-aggregation in this context?

From @peferron 's comment above  "That's one example of the usefulness of accepting any Druid aggregation or post-aggregation as a metric . "https://github.com/apache/incubator-druid/issues/7187#issuecomment-470386898.
Just wondering if "Druid Aggregation" occurs in the node and "post-aggregation" occurs on the broker?

From @peferron 's comment above "That's one example of the usefulness of accepting any Druid aggregation or post-aggregation as a metric . "#7187 (comment).
Just wondering if "Druid Aggregation" occurs in the node and "post-aggregation" occurs on the broker?

Ah, usually, aggregation happens on data nodes and post-aggregation happens on the broker. But if you're sorting on a post-aggregation (like an average, which in Druid is a post-aggregation division) then that particular post-aggregation is done on data nodes right before the early-limiting step. Sometimes this works great and sometimes it's wildly inaccurate. You do need to be a little careful with this particular feature.

Ok, this is what I've been suspecting.
If I understand what has been said here (and I still may be missing something), then WRT the TopN operations:

In general, anytime you do limiting prior to an aggregation step you can produce errors, and the size of the resulting error will be unknown since it is data sensitive, and there will be no warning.  This will be true whether it is performed on the node or on the broker.

Also, the max(k, 1000) step may be wasteful in the absence of aggregation, and may not provide enough data in the presence of aggregation.  And, again, there will be no warning.

I don't think you're missing anything. It is definitely 'well known' that the topN algorithm has data-sensitive behaviors and only works if the top things are somewhat consistent from segment to segment. (This is discussed in the documentation.) People have still found it quite useful, though. I think they would also find something FIS-based to be useful; it sounds like it's a tool that is applicable to fewer problems, but better suited for those problems it is applicable to.

Perhaps it is "well-known" amongst the Druid developers, but I think the Druid team is taking a serious risk with Druid's reputation.  Druid's customers may not fully understand that the current TopN functionality may return garbage, even if it is discussed in the documentation (who reads the manual, anyway  :)  ).
Quoting @peferron above:

Going back to your song example, could FIS support getting the top song titles by number of unique listeners over the past week? The current topN can do that using HLL sketches as metric (weight). That's one example of the usefulness of accepting any Druid aggregation or post-aggregation as a metric.

It is not clear from his statement whether he realizes that "accepting any Druid aggregation or post-aggregation" may not work as intended and, in fact may produce meaningless or misleading results.
Data sensitivity is not a good thing especially given that there is no easy way for a customer to determine whether the data he/she has processed is producing trustworthy results or not.
I would consider the current TopN functionality to be broken and unreliable, period.
Here are two alternatives (there may be others):

In any "TopN" operation do not allow any aggregation steps after the data set has been "limited".  This has to be true both on the nodes and on the broker.  7
Adopt the Frequent Items sketch for "TopN" operations.  Accept that the ordering weight metric is positive integers (longs) and that the aggregation operation is add().  (If you feel you really need real-valued weights, then we can develop for you an FIS based on doubles.  That is not a big deal.)

My point is that a better, more efficient, and free solution exists for the most common TopN type operations. Why not use it?

@leerho It's well understood that FIS would be better than the current algorithm for a subset of topN queries, and I'm sure that most Druid users would love having that.
Regarding the subset of topN queries that FIS cannot replace, though, removing that functionality from Druid without an alternative is not realistic, even though the data sensitivity and lack of error bounds are big issues.

Just to be clear and independent of FIS, I'm not advocating removing TopN.
What is very problematic in the current implementation is allowing aggregation functions after limiting / truncation of the data.  This can lead to wildly wrong results such that it is misleading to even call it "TopN" anymore.  This is very easy to prove, by the way, that you could be entirely missing the Top-1 and with a little extra math, prove that it is not too hard to miss all of the TopN.
You can still have TopN and you can still provide the aggregation functions as long as they are all performed prior to any limiting/truncation.  If you follow this, It changes the max(k, 1000) step to just k, which means you will be sending far less data to the broker.  And now when the broker  applies a PriorityQueue(k), you will have a TopN that is no longer data sensitive and quite robust.  Although it will likely be slower.
I realize that speed is all important to Druid and that is why I love Druid as I am a speed freak too :)
But allowing functionality that effectively corrupts the intent of the query is not a good idea as it can come back to bite no matter how much you caveat it in the documentation.
"If it doesn't have to work, it can meet any requirement."
"If you don't care about quality, you can achieve any objective."
-- G.M. Weinberg
Yes, perhaps the FIS should be a completely separate function, that is up to you.  But it is the only mechanism that will allow a simple form of aggregation and "truncation" at the same time, is fast, data insensitive, and with known error.

And, I should point out that for the current implementation of "TopN" if any of the weight parameters are negative, the aggregations MUST be performed prior to truncation otherwise the results will be absolute junk.

@leventov @gianm @peferron @Dylan1312
I removed two comments I had here, because after thinking about this some more, I am confident that I can make even stronger statements to make my point clear.
I the "good old days" of SQL if you issued a query shaped like SELECT x, AGG(y) FROM table ORDER BY AGG(y) DESC LIMIT N, the DataBase would oblige by performing the operations brute-force on all the data, and would eventually return with the result set of N items that you could call a "TopN", but as we will discover that is a misnomer.  The bottom of that list could have some items of equal weight that could represent noise in the sense that they may not have a weight that is greater than the weight of all remaining items not on the list.  Nonetheless, this would be considered the "exact" solution.
However, in the attempts to speed up the horribly slow brute-force SQL query above, and based on what was described above in this thread, the Druid "TopN" algorithm was created to do this (I think this is right):
M Historical Nodes:

"First Truncation": All intra-segment resultsets (which are exact) are sorted (using a priority queue) and truncated to max(k, 1000). Call that K.
"Pre-Aggregation": Then the historical node merges all those per-segment resultsets pairwise by first combining the 2K elements based on the aggregation key, then sorting those 2K elements (again, using a priority queue) into a new priority queue of size K.
Each H-Node passes a sorted resultset of size max(k, 1000) to the broker.

Broker Node:

2nd Truncation and "Post-Aggregation": The data from all M H-Nodes is gathered, and sorted and truncated again using a Priority Queue with a size max(k, 1000).
K items are reported as the result.

These truncations and aggregations change everything!
The first belief that:

The current topN algorithm can be used for any query shaped like SELECT x, AGG(y) FROM table ORDER BY AGG(y) DESC LIMIT N when useApproximateTopN is enabled.

This belief is a total fiction and this claim will fail miserably.
The second belief:

Going back to your song example, could FIS support getting the top song titles by number of unique listeners over the past week? The current topN can do that using HLL sketches as metric (weight). That's one example of the usefulness of accepting any Druid aggregation or post-aggregation as a metric.

This claim of the ability to plug-in HLL sketches as the aggregator will also fail miserably.
The third belief:

It sounds like FIS is limited in a couple ways that prevents it from being used for all queries of that form: (because)
"AGG" must be "SUM" and y must all be positive. (topN will accept any aggregation function and any y)

The view here is that somehow FIS is "limited", and is a "subset case" because it can only sum positive values is seriously misguided.
In fact the only "aggregation" operation that can possibly produce any meaningful results with the Druid TopN algorithm is a sum of positive weights. And this  case that kinda-sorta works is when the data in the H-nodes is pre-aggregated (only sum of positive weights allowed) prior to the 1st truncation (Priority Queue). Then the PQs are merged, summed and PQed again, before passing to the Broker.  The process in the Broker would be similar.  But this still could have significant errors.
The forth belief:

"LIMIT" (using FIS) will not necessarily be adhered to. (topN will always return LIMIT results, or the number of distinct values of x, whichever is lower).

This is also misguided.  For a set of items where all have the same weight, there is no TopN!  Any random selection of N values from the set could be a "valid" result, which makes the name "TopN" meaningless.  In order for a set of values to have a "Top", there must be some skew in their weights.
Suppose the set has 1000 items with weight 1, and 2 items of weight 10.  The top-100 will have the two heavy items, the other 98 could be randomly chosen from the remainder of the original set .  Those 98 add no insight as they are essentially noise.  So to require that there be N items returned from a TopN query is severely misguided.  The Druid TopN also doesn't tell you which one of the small end of the TopN are noise, making it even less useful.  The better term to use is either "Frequent Items" or "Heavy Hitters", both of which are used in the scientific literature.

In conclusion:

The belief that a query of the shape SELECT x, AGG(y) FROM table ORDER BY AGG(y) DESC LIMIT N can actually produce meaningful results using the Druid TopN algorithm is misguided.  It is especially misguided if the item weights are negative  or those aggregations are non-linear functions such as an HLL sketch.

The only query that might kinda-sorta-sometimes produce meaningful results is when the weights are positive and the aggregations are sum.  But if this is the case, why not use the FIS, which operates with positive weights and the "aggregation" is a sum.   And it will produce far superior results.

In other words the FIS is not a subset of what can be usefully queried, it is the ONLY query set that can produce meaningful results.

@leerho

The first belief that:

The current topN algorithm can be used for any query shaped like SELECT x, AGG(y) FROM table ORDER BY AGG(y) DESC LIMIT N when useApproximateTopN is enabled.

This belief is a total fiction and this claim will fail miserably.

You can use topN with any aggregation and get back results. This is a fact, not a belief. The misguided belief would be that these results are always meaningful, but I don't recall anybody here saying that; it has been repeated again and again that topN is data-sensitive.

The second belief:
This claim of the ability to plug-in HLL sketches as the aggregator will also fail miserably.

Same answer as above. You can absolutely plug a HLL sketch as a topN aggregation. Whether the results are meaningful or not depends on the data.

The third belief:
[...]
In fact the only "aggregation" operation that can possibly produce any meaningful results with the Druid TopN algorithm is a sum of positive weights.

Since this assertion is worded very strongly, it can be disproven by finding a counter-example where meaningful results are produced using topN in conjunction with, say, a HLL sketch.
That's easy to do for almost anyone with access to a Druid cluster. That happens to be my case, so I just ran queries against about a billion rows of real-world data. The queries are SELECT x, COUNT(DISTINCT y) GROUP BY 1 ORDER BY 2 DESC LIMIT 5.
In the first test, x is a dimension with a cardinality of 30k, and y is another dimension with high cardinality. topN returns the same results as groupBy (which is the exact counterpart of topN), but 30% faster.
In the second test, x is a dimension with a cardinality of 2M. groupBy cannot return results because it exceeds resource limits. After reducing the query interval enough to bring the cardinality of x down to levels that groupBy can handle, topN again returns the same results as groupBy, but 75% faster.
So, here we are—meaningful results produced using topN with HLL, and with significant performance gains.
These are the first queries I tried, but I'm sure I could find some where topN fails miserably. That's not the point, though—the point is that topN works well in some cases, as demonstrated. You seem to think that Druid users, including participants in this discussion, are incapable of identifying these cases, and are using topN only because they don't understand how badly it can fail. That's certainly your prerogative, and you may be right for the possibly large number of Druid users who don't dig into how the system works or read the documentation. As a somewhat power user though, I'm happy to have a tool like topN at my disposal, even if it's up to me to use it responsibly.
That being said, I certainly hope that Druid goes towards a dirrection where error bounds are systematically provided, as discussed in #7160. So thank you and @AlexanderSaydakov for starting this movement by integrating the DS library.

@peferron
Thank you for your response and for performing a test with an HLL aggregator.
The intent of my statements in the preceding entries is to protect the end users of Druid, and ultimately to protect the reputation of Druid as a trustworthy platform.
In my statements above, my use of the word "meaningful" is with respect to an end user and is meant to imply that the results of any query of the above shape could be interpreted usefully (i.e., correct interpretation and similar to what a brute-force exact computation would produce).
My use of the word "belief" is with again with respect to the end-user:  That he/she might believe (because the Druid specifications say that it can be done) that the results of any query of the shape stated above would be meaningful (useful and correct).  Of course the query can be executed, that is beside the point, and doesn't require much trust or belief.
Unfortunately, running one test does not prove that an algorithm or process works correctly.  Nor does running a hundred or thousand such tests.  However, all one has to do to prove that an algorithm or process does not work is to provide one counter example test that does not work.
Relying on the fact that the documentation states that the TopN process is data sensitive is a very weak argument, as most end-users probably don't read the documentation, or don't think about what that phrase might mean.
The fact that a data sensitive query might run 30% or 75% faster is a misleading "sales pitch" to the end users, because in order for an end user to actually trust that the results from such a query is usefully correct, the end user would have to run each query in brute force, exact mode and check its results with the results from the "faster" query.  The total time to do both queries is clearly longer than just running the brute force exact query by itself.   If you think this is ridiculous,  it derives directly from the "data sensitive" statement.  Because you cannot prove that a "data sensitive" process works on all data, because you can't possibly have access to all data.  More importantly, you don't have access to the end-user's data!  The statement "data sensitive" is a huge admission that the process may not work, and without running brute-force tests on all runs, the user would never know whether he/she can trust the results or not.
It appears that it is your opinion that just documenting the process with the words "data sensitive" is enough, and if the user wishes to shoot themselves in the foot that that is OK.  I don't share that view, and perhaps nothing I say will change your mind, nonetheless, there are a couple of thoughts that perhaps you might consider.

Allowing positive and negative values as weights in the aggregations.  This actually can work if there is no limiting nor truncations of the data followed by aggregations.  However, with truncations followed by aggregations, negative and positive weights can easily lead to cancellations of weights of items across segments or nodes within the limited region and nullifying any earlier TopN results from individual segments or nodes.  This, I hope you can see, would easily lead to nonsensical results.

Allowing non-linear processes like unique counting (HLL), as an aggregation option, can also lead to very strange results both within the truncated regions and outside the truncated regions.  This is not as serious a problem as allowing negative weights.  It turns out that the use of a unique-counting algorithm like HLL as an aggregator inside a Heavy-Hitter / Frequent-Item sketch has been written about and studied quite a bit in the scientific literature.   However, the algorithms that have been proposed so far are pretty nasty and it is not at all clear that they are practical (e.g., not mergeable, complex, the results are probabilistic, and other issues).

For a mergeable process that incorporates truncation / limiting followed by aggregation, to date we know of no algorithm that would produce trustworthy results if the aggregations allow negative values or non-linear calculations (e.g. unique counting).
To lead the user to believe (or hope) that an expression like
SELECT x, AGG(y) FROM table ORDER BY AGG(y) DESC LIMIT N when useApproximateTopN is enabled would produce meaningful, trustworthy results with arbitrary aggregation plug-ins is fundamentally misleading.  And hiding behind the "data-sensitive" caveat doesn't change that.
The only algorithms that do incorporate truncation / limiting followed by aggregation and do provide trustworthy results are the sketching algorithms described in the scientific literature: the main ones are Misra-Gries, Space-Saver, Heavy-Hitter, and Frequent-Items algorithms.  And all of these algorithms require that the weight aggregations must be linear, additive with positive numbers.   These are not a subset of a larger working set of working algorithms. They are the only set that, to date, work.

@leerho I think we're on the same page regarding the behavior of the current topN. Our disagreement comes from what we consider "meaningful", which is more subjective.
For you, "meaningful" means returning useful and correct results regardless of the data being queried.
For me, it's OK if correctness depends on the data; for example, I consider binary search to be useful and capable of returning meaningful results, even though it returns garbage on unsorted data.
You probably also think that binary search is useful. So the main difference is, I think, that it's usually simple to ensure that data fed to binary search is sorted, but much harder to ensure that data fed to topN is suitable along all combinations of aggregations/filters/intervals. That's the main practical problem with topN IMO. But the difficulty here depends on the size of the query space, which is application-specific, so I don't think there can be a single yes/no answer regarding whether one should use topN or not.
It's interesting that you mention running brute force queries to verify topN correctness. It's actually quite viable, because the topN query quickly returns results to the user while the brute force (groupBy) query runs in the background using spare cluster resources. So the user experience is improved, even though the total computation time and resources are indeed larger than running only the brute force query.

@peferron Thank you for your thoughtful comments.  Clearly we have to leave it up to an informed user to decide.  And all we can do is do our best to make sure that he/she is informed.

Bouncing back to the top of this thread, we are developing a new sketch that we are tentatively calling "Frequent Unique Nodes" (FUN).
Suppose you have a stream that contains pairs {IP address, UserID}, and you wish to identify the IP addresses that have the largest number of unique users.  In this context think of a large graph where the IP addresses and users are nodes in the graph.  Consider Node1 = IP and Node2 = ID, then we want to identify the Node1s that have the largest number of unique Node2s.  Conversely, it might also be interesting to identify the Node2s (IDs) that have the largest number of unique Node1s (IPs).   Conceptually, this can also be extended to more that just 2 nodes (although don't go nuts with this!).
With this new sketch you will be able to perform these types of queries and have some guarantees of accuracy as well.
If this is of interest, please let me know, as we could use your help in characterizing and performance testing of this, if possible.
Lee.

From your description, it sounds like it could handle the "top songs by unique viewers" query we discussed earlier, by processing pairs of {SongID, UserID}. Essentially like a topN with HLL but with accuracy guarantees, and still better performance than an exact query. If that's the case then it sounds extremely useful. I'd love to see how you plan to implement that.
It would be especially useful if a additional aggregations could be computed at the same time, such as computing the total (non-distinct) count for each key: SELECT COUNT(DISTINCT UserID), COUNT(*) GROUP BY IPAddress ORDER BY 1 DESC LIMIT 10. In my experience it's common to compute multiple metrics in the same query in Druid. This looks like something that would be implemented in the Druid extension rather than the sketch itself though, although the sketch may need to expose some APIs to support that.
Once you have a draft extension ready I'll be happy to run it against some of our real-world datasets if that helps. I'll even pick some dimensions where topN fails badly for comparison 😄. I won't have time to help developing the extension itself, unfortunately.

@peferron

top songs by unique viewers
Yes.

additional aggregations could be computed at the same time
Yes, with a caveat.  For the example you gave: SELECT IPAddress, COUNT(DISTINCT UserID), COUNT(*) GROUP BY 1 ORDER BY 2 DESC LIMIT 10 A sketch could be constructed to handle that specific query.  However, handling arbitrary additional aggregations is more challenging as we would have to assume that the additional fields are simple counters, and restrict the type of aggregation to a simple addition.

The more generic we try to make this, the more challenging it will be to configure and the performance will be impacted.

Can we start with just the "top songs by unique users" and characterize that first?

Will you need an actual published artifact Jar to test this.  Or would a jar generated from master be OK for your testing?

The more generic we try to make this, the more challenging it will be to configure and the performance will be impacted.

Understood. Extra aggregations can still be computed in a follow-up query anyway, such as SELECT COUNT(*), AVG(SomeColumn) WHERE IPAddress IN ('a', 'b', 'c') where a, b and c are top IP addresses previously returned by the FUN query. This can work decently well with Druid indexes. One advantage of this approach is that the FUN query returns faster since it's not loaded with extra aggregations, so you can immediately show up the top items & unique counts to the user, with extra metrics following up later. In some cases that's better UX than a longer initial wait followed up by showing up everything at once. That's a bit off-topic but hopefully there's some value in listing the different ways in which this sketch could be used within Druid.

Can we start with just the "top songs by unique users" and characterize that first?

Sure. Are you looking for any specific patterns in the test data? I assume that Yahoo already has large datasets & Druid clusters that could be used, so I'm trying to see what I could bring to the table here.

Will you need an actual published artifact Jar to test this.  Or would a jar generated from a branch be OK for your testing?

The best would be a branch that I can check out to build the extension from source.

Just to let you know I am almost done with this new Sketch .  I decided that the FUN acronym was a bit too whimsical so I have renamed it Frequent Distinct Tuples, which I think better reflects what it is doing.  Not as easy to say, I realize, but we will get used to it.
This new Sketch is experimental still, as I don’t know how it will actually perform yet.  But it is better than nothing, at least until a better algorithm comes along :)

@peferron
The new FDT sketch is in master and should be stable enough for you to start looking at it and playing with it.  I'm sure you will have questions so ask away :)    I need to do some more comprehensive testing and characterization before it is ready for release.  You can find some simple examples in the test hierarchy as well as in the characterization repository (well, those are a bit more complex :) :) ).
I am actually quite pleased with its accuracy.
You can find the configuration file for running the characterization tests here.
Have fun!

@peferron  @gianm
check out the updated docs

@peferron @gianm
This sketch has now been released as part of sketches-core 0.13.2 and is fully documented on the website with example and error characterization.

Thanks @leerho. I'm in vacation atm but should be able to characterize it against a few of our datasets after I'm back in early May.


Please invite me to the Recode-Hive GitHub Community Organization

Name
Rahul Kumar
Discord Username (if applicable)
No response
Additional Context
Context
I’m excited to join the Recode-Hive GitHub Community because it provides an excellent platform to collaborate with like-minded developers, learn best practices, and contribute to meaningful open-source projects. I appreciate the community's focus on knowledge-sharing and continuous learning, which aligns with my goal of advancing my skills and working alongside passionate peers. Joining Recode-Hive will allow me to actively participate in open-source initiatives and improve my technical abilities while giving back to the community.
Profile Details
Github: https://github.com/rahulgithub-web
LinkedIn: https://www.linkedin.com/in/rk-barnwal23/

Invitation sent to join the GitHub Organisation. Welcome to the community 🎉
Don't forget after accepting to make it public so it appears on your GitHub profile for everyone else to see. You can do this by finding your name in the GitHub organisation list and change the dropdown to public https://github.com/orgs/recodehive/people


changes wording and elif statement

Hi @dodoarg
Changed wording of the errors and added some punctuation. This has nothing to do with your changes. They were there already.
I also changed the elif for an else, given that there are not multiple conditions, I felt it was more appropriate.
This is ready to go.
Thank you for the changes!

ok let me just make sure that the tests pass


Remove unused GenericScopedLock and GenericScopedTryLock.

Update documentation of GenericScopedUnlock.

@miguelportilla @scottschurr

[Current coverage][cc-pull] is 68.65%

Merging [#1693][cc-pull] into [develop][cc-base-branch] will decrease coverage by -<.01%

4 files (not in diff) in src/ripple were modified. more

Misses +4
Hits -4

@@            develop      #1693   diff @@
==========================================
  Files           866        866
  Lines         61435      61435
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits          42178      42174     -4
- Misses        19257      19261     +4
  Partials          0          0

Powered by Codecov. Last updated by [36ac112...dc170fd][cc-compare]
[cc-base-branch]: https://codecov.io/gh/ripple/rippled/branch/develop?src=pr
[cc-compare]: https://codecov.io/gh/ripple/rippled/compare/36ac1124f4b4b96a50b1f596d4acb7880803e870...dc170fd04db95794d7a81f080efbdfbf12dbab58
[cc-pull]: https://codecov.io/gh/ripple/rippled/pull/1693?src=pr

:+1: Nice to remove unused (and untested) code.

:+1:

Took care of the vertical space Scott mentioned.

merged.


Adds dotfiles to git

All dotfiles were gitignore'd but we need some of them for configuration.

Current dependencies on/for this PR:

main

PR #5

PR #6

PR #7

PR #8   👈

This comment was auto-generated by Graphite.

Merge Activity

Sep 24, 1:44 PM: @tylergaw started a stack merge that includes this pull request via Graphite.


Add Jdbc Source

Using same config options as XD version: --query, --split, --update, --maxRowsPerPoll, --fixedDelay, --maxMessages

Resolves #86

The #146 has been deprecated.
This stuff can be reworked to the @PollableSource or just @Autowired PollerMetadata defaultPoller, if that.

A bit confused how I would replace this:
	@Bean
	public PollerMetadata poller() {
		return Pollers.fixedDelay(properties.getFixedDelay(), TimeUnit.SECONDS).
				maxMessagesPerPoll(properties.getMaxMessages()).
				transactional(transactionManager).get();
	}

See TriggerConfiguration, there is a bean like:
@Bean
public PollerMetadata defaultPoller(Trigger trigger) {

You should use it like:
@Autowired
@Qualifier("defaultPoller")
private PollerMetadata poller;

Since SCS already has its onw PollerMetadata.DEFAULT_POLLER, we can't just use @Autowired for that case.
Also see the FileSource exactly for the sample which you are looking for.
And yes: you should remove that your poller() altogether :smile:

had to add @Import(TriggerConfiguration.class) and works better now. Still confused about the transactional part - not needed?

Indeed. Let's think if we really need it here.
We read data from DB, may update it and send to the output. May do that for each record with split.
That's all: our work here is done. The data is send to the Binder over the network.
No anything else which may impact the process!
It is just done here on the output channel.

SO, are you saying that we don't need transactional? What would happen if one of the update statements failed?

What would happen if one of the update statements failed?

We have there (JdbcPollingChannelAdapter) only one of them according this (JDBC Source module) our settings:
private Object poll() {
	List<?> payload = doPoll(this.sqlQueryParameterSource);
	if (payload.size() < 1) {
		payload = null;
	}
	if (payload != null && updateSql != null) {
		if (this.updatePerRow) {
			for (Object row : payload) {
				executeUpdateQuery(row);
			}
		}
		else {
			executeUpdateQuery(payload);
		}
	}
	return payload;
}

Where transaction, if that, is applied exactly around this poll(). Over the parent call stack, of course.
See AbstractPollingEndpoint#createPoller()  and AbstractPollingEndpoint.Poller class:
while (initialized && (maxMessagesPerPoll <= 0 || count < maxMessagesPerPoll)) {
	try {
        	if (!pollingTask.call()) {
				break;
		}
	....
}

That pollingTask.call() is transaction in our case.
In other words the tx (or any poller's advice) is applied to the single MessageSource invocation only.
So, having that all I really don't see reason to have the TX for single SELECT/UPDATE. That works atomically on the RDBMS. You can reach better isolation with the SELECT .... FOR UPDATE, I think...

Rebased switched to use defaultPoller from TriggerConfiguration class

In addition the sources.adoc article must be there.
I'll show you polishing on the matter.

Here you are the polishing for your confirmation: https://github.com/artembilan/spring-cloud-stream-modules/commit/1fbb15bc17f2b1e4697ceecd0ab92ce7af49c28d

@trisberg , does your "LGTM" mean that I can merge ?

Yes, go for it :)

Merged as https://github.com/spring-cloud/spring-cloud-stream-modules/commit/48f8c2df4c9a8a05cab60c7170622ded98997823

Also see https://github.com/spring-cloud/spring-cloud-dataflow/commit/e2c10359e769c7a03626ca33449ecd2e04b1e122


eks package's createNodeGroup function would benefit from some advanced examples

I am struggling to use createNodeGroup with extraNodeSecurityGroups parameter. the documentation says the value of this parameter needs to be SecurityGroup[]. I tried passing an array of strings of security group ids and also passing the output of ec2
.getSecurityGroup but it is simply not assigning those security groups to worker nodes.
My code that is not working looks like this
      cluster1.createNodeGroup("abc", {
      instanceType: "t3.medium",
      autoScalingGroupTags: {
        ["k8s.io/cluster-autoscaler/enabled"]: "true",
        [asgtag]: "owned",
      },
      spotPrice: "0.6",
      nodeAssociatePublicIpAddress: false,
      nodeSubnetIds: [privatesnetids[0][0][0], privatesnetids[0][0][1]],
      desiredCapacity: 1,
      amiId: "ami-00d7f1aeda5144f18",
      minSize:1,
      nodeUserDataOverride: userdata,
      extraNodeSecurityGroups: [
        aws.ec2
          .getSecurityGroup({
            id: "sg-78673456687673",
          })
          .then((selected) => {
            selected
          }),
      ],
      maxSize: 4,
.....

Where am i getting it wrong ? any guidance is much appreciated Thank you

further debugging the issue . I noticed that if I add extraNodeSecurityGroups after a node group is created . It throws a panic with the following message
panic: fatal: An assertion has failed: Expected diff to not require deletion or replacement during Update of urn:pulumi:dev::eks18::eks:index:Cluster$eks:index:NodeGroup$aws:ec2/launchConfiguration:LaunchConfiguration::apps-spot4-nodeLaunchConfiguration

Looks like updating a launchconfiguration with new securitygroupids seem to fail when done an existing node group. If it is a new nodegroup it simply ignores extraNodeSecurityGroups.
FYI pulumi version I am on is v2.14.0

Digging deeper It seems pulumi is not aware that a launch configuration cannot be edited/updated. As per AWS API only a new launch configuration can be created and attached to existing ASG. AWS doesn't allow edit of launch configurations.
Terraform does it as documented below.
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_configuration
Pulumi tries to update in place a node launch configuration which results in this error. Starting to think this is a bug.

@bit-cloner Sorry for the issues here.  The error you are seeing Expected diff to not require deletion or replacement is always a bug in Pulumi, so this is definitely something we need to look into - and is not expected behaviour.
I don't 100% follow the repro case here.  It sounds like you hit that panic after hitting some other issue - but what exactly was the original issue you hit?  Any chance you have a repro for each of the two issues?

Looks like this may be ultimately related to https://github.com/pulumi/pulumi-terraform-bridge/issues/296.

Possibly. But can't confirm. What I was trying to do is add "extraNodeSecurityGroups" to an existing node group.
Panic error happens when adding these to existing resource.
Ultimately I Want to be able to create a node group with extra security groups.
I will raise a detailed repro tomorrow
Thank you

Closing as a duplicate of #1242.


Death pile timer resets when hopping worlds

Hey, can you please provide as much detail as you can around the circumstances of your death, location, how long the client was open, what you did after the death, etc please? This issue keeps coming up every so often but it only happens some times, so it's really hard to figure out why it's happening.

Took a clip of it happening, happens every time I hop so didn't think to include much info
I pick nettles to die in edge
here's a clip of it happening: https://streamable.com/d4pjeg
Expires in 39 minutes is correct, but upon hopping it goes to 53 years ago, to 60 minutes

This happens because there is no reliable way for the plugin to get your account's played time, the current workaround is to ensure that the "Character summary" (brown star) tab is active in the quest interface at least once per session before you die and when checking how long is left.
I'll be changing the behaviour of deathpiles so that there is a choice in the settings between this current behaviour (accurate timers, but requires the brown star tab to be active at least once per login/hop) or static timers (60 minutes from point of death, only ticks down while logged in, on runelite, with the plugin enabled - meaning that if you log in for 5 minutes on mobile, the timer will be wrong)


login crashing

http://pastebin.com/p5405D4a

Details of what occurred during the crash. Any reason to believe its your inventory or your chunk? So far chunk backups from the past 3 days are not working.
You should always provide as many details as you can when leaving a crash report.

Sorry, I was mining last time I was on. Nothing really advanced in my base to cause a problem. It just dumps me out when I get to the point where I am entering the world.

Cuppy, If you want to clear everything I have on me, to see if that works, feel free. Nothing that can't be remade.

player data cleared

Now I am getting the errors "Disconnected from server" and "A fatal error has occured, this connection is terminated" After this happened the first 4 tries or so, I selected reinstall before logging on. I am still getting the messages.


Angular renderer for dynamic components

It would be fantastic if it was possible to render dynamic Angular components into the editor. Perhaps an Angular clone of the official VueRenderer or ReactRenderer.

@dkrusenstrahle I am able to make it work.
https://user-images.githubusercontent.com/20282546/116850859-01d2a100-ac0f-11eb-997c-562caf4a0041.mov
I will publish this today or tomorrow after some cleanup and added test cases.

@sibiraj-s Wow, great work! Really looking forward to try it out.

@sibiraj-s I can see that you have released the renderer super cool! However I cannot access it even if I reinstall or update ngx-tiptap. Any idea why? Is there a delay?

@sibiraj-s I extracted the file for now until the package is updated (worked perfectly). One thing comes into mind. I think the editor instance should be accessible within the dynamic component so that it can check if the editor is editable / activated or disabled in order to display different content for the user. Perhaps should the node instance it self also be accessible? What do you think?

Fixed in v1.2.0.  All these props are now made available

One thing comes into mind. I think the editor instance should be accessible within the dynamic component so that it can check if the editor is editable / activated or disabled in order to display different content for the user. Perhaps should the node instance it self also be accessible? What do you think?

These things I didn't add at first. I was slowly making changes and progressively pushing it to master. All these should be available now. I pushed all changes to master to fasten up the development, hence the confusion.

Perfect! I cannot get the Angular Node view redererer to work with 1.2.0. I get this error. I have reinstalled the package and restarted the server.
Module '"ngx-tiptap"' has no exported member 'AngularNodeViewRenderer'

Let me check.

Nevermind got it to work now. Saw there was some changes for updatesAttributes I had to fix. I reinstalled a few times and now it works.

Could be cache. Sometimes it takes small time to reflect after publishing the package.


Update Go-Git to take advantage of LargeObjectThreshold

Following the merging of https://github.com/go-git/go-git/pull/330 we can now
add a setting to avoid go-git reading and caching large objects.
Signed-off-by: Andrew Thornton<EMAIL_ADDRESS>

making this 1.15 unless there are good arguments against.

@zeripath can you rm gitea.db-pre-test

sorry


Looking into using SD serial to wifi module to connect Wirelessly with the bot (AKA 306 IOT device)

Hey Josh wanted us to looking into this as a possible replacement for the NICs as method of wireless debugging and connecting to the bot. I will see if I can figure this out in future session when I bring my module and see if I can connect to the bone.

See issue  #434


Whether the simsimd library can be changed to optional configuration

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
simsimd cannot be installed properly on some older systems, and simsimd is not a necessary dependency in my opinion.
Describe the solution you'd like
A clear and concise description of what you want to happen.
After testing, I found that simsimd requires libc version 2.28+, and the centos version I use is only 2.17, and upgrading libc is too risky.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
After looking at the langchain-weaviate code, I see that the vector computation is only accelerated in _math.py, which appears on line 29.
Z = 1 - np.array(simsimd.cdist(X, Y, metric="cosine"))
In fact, this can be changed to an optional configuration, rather than a necessary configuration, so that the package is not available because simsimd is not installed.
For example:
try:
    import simsimd  # type: ignore
except ImportError:
    import numpy as simsimd

Additional context
Add any other context or screenshots about the feature request here.

I like this idea, just also add a logging message to inform user that simsimd lib is not found and will drop numpy.
leaving this open for community contribution.


Meta issue: add code blocks to the relevant sections of the new-issue template to facilitate formatting

Not sure where to post this, so I'm doing it here.
Below is a modified version of the new-issue template (use the markdown source) that adds fenced code blocks / output blocks  to the Steps to reproduce, Expected behavior and Actual behavior sections, to facilitate proper formatting.

Steps to reproduce

Expected behavior

Actual behavior

Environment data

> $PSVersionTable

You can submit a PR on the issue template - it's here

Thanks, @lzybkr - find the PR at #3953


Implement @orbit/identity-map

interface RecordModel {
  id: string
}

interface ModelFactory {
  create(identity: RecordIdentity): RecordModel;
  disconnect(identity: RecordIdentity): void;
  notifyPropertyChange(identity: RecordIdentity, property: string): void;
}

interface IdentityMapSettings {
  modelFactory: ModelFactory;
}

interface IdenityMap {
  constructor(settings: IdentityMapSettings);

  lookup(identity: RecordIdentity): RecordModel;

  lookupMany(identities: RecordIdentity[]): RecordModel[];

  lookupQueryResult(query: Query, result: Record | Record[] | null): RecordModel | RecordModel[] | null;

  has(identity: RecordIdentity): boolean;

  evict(identity: RecordIdentity): void;

  notifyPropertyChange(identity: RecordIdentity, property: string): void;

  deactivate(): void;

  forEach(callback: (record: RecordModel) => void): void;
}

Hey dan. I have done more experimenting in my own store implementation. I have quite change my mind in comparaison to the work done here. My current thinking is that IdentityMap should be just this – a Map so I guess we are thinking in the similar direction. My current implementation looks like this:
type IdentityMap = Map<RecordIdentity, RecordModel>;
type IdentityMapFactory = {
  get(source: MemorySource): IdentityMap;
};

Where RecordModel is an abstract base interface for a Model class.
You can see the full implementation here :
https://github.com/tchak/ember-orbit-store/blob/master/addon/identity-map.ts
I like the idea to add a IdentitySerializer to the mix. I will try to add it to my current implementation and will report how it went.

@dgeb I pushed un update with what I have so far

I think to make it useful with forking we will need an extra piece IdentityMapFactory. My current implementation looks like this. I am more then happy to add it in a later PR.
export default class IdentityMapFactory<Identity, Model> {
  protected _serializer: IdentitySerializer<Identity>;

  constructor(settings: IdentityMapSettings<Identity>) {
    this._serializer = settings.serializer;
  }

  get<Source extends object>(source: Source) {
    let identityMap = identityMapBySource.get(source);
    if (!identityMap) {
      identityMap = new IdentityMap<Identity, Model>({ serializer: this._serializer });
      identityMapBySource.set(source, identityMap);
    }
    return identityMap;
  }
}

const identityMapBySource = new WeakMap();

@tchak This is looking quite good to me. I'm not fulling understanding IdentityMapFactory at the moment, so let's leave it for a future discussion / PR.

I want to add more tests but will wait for your confirmation that you are happy with the design

I had to adapt the types a bit to match Map interface. But it should be all good and tested now :)

This is looking even better now. One last level of generalization could be achieved by removing the dependencies from other orbit packages, and perhaps moving the identity serializer defined in record-identity-serializer.ts to be used for tests only. What do you think @tchak?
I'm planning in the near future to introduce lid as a unique identifier for records (unique across types), which will simplify questions of identity even further.


Setting up a Push server on existing PHP code

Hi,
I'm new to the Ratchet framework and facing some issue understanding how the architecture is designed. I have set up a basic Ratchet server and webSocket connection is being established from an Android client. Now, I have another PHP server that takes care of existing functionalities. I want to integrate both of these. So, the code base can be same or I would have to setup the Ratchet codebase as a separate application.
Another issue is that how would I setup push messages from existing server. Would I have to run another PHP script on the Ratchet server or this would be sufficient?
Sorry if these questions sound naive, but pls help me..
Thanks,

A WebSocket connection stays open between a client (web browser) and the server, unlike traditional web application that establish a fresh connection on each request.
In a traditional website setup, the following happens:

A browser sends a request to a server.
The server and browser confirm this request (handshake).
The server processes the request, and then sends the response.
The connection is closed.

The web is naturally a stateless environment, which is why we add client-side cookies, and server-side sessions to maintain (remember) state.
WebSockets maintains a connection between the server and connected clients. Basically, with a connection established, the server can send messages to all connected clients at any time. The request process can begin at the server.
Here's how it looks:

The client (browser) request a WebSocket connection with the server.
The server receives the request and adds the client to a list.
The connection remains open, with the server aware of all connected clients.
Messages can be sent back-and-forth between client and server.
The client or server can 'close' the connection at any time.

In order to involve a second server, you'd likely need some kind of server-to-server communication channel, and then the Ratchet server sends any notifications that it becomes aware of. That's all I can suggest without knowing more about your needs and setup.
Maybe give us a block-diagram showing your desired setup?

Closed for inactivity.
@him2016 This majority scenario is the example pusher project.


Pattern Matching with type()

Environment data

Language Server version:  2023.7.31 (pyright b917b7a5)
OS and version: WSL2 Debian
Python version (& distribution if applicable, e.g. Anaconda): 3.11

Code Snippet
y = str

match y:
    case type(): # <-- This line complains - Pattern will never be matched for subject type "y"
        print(8)
    case _:
        pass

Thanks for the issue. Took me a bit to see the error. Requires turning on reportUnnecessaryComparison.

Seems like a core type checking issue. Transferring to Pyright.


Adding simulator editor events

Adds simulator events for the editor including the usual start, stop, resume, exception, and a new one for when the top level code has finished executing.

As discussed, honour the allowParentController flag when sending these messages


added image sharing on android and ios

Hi, have someone tested this pull request? If someone confirms it's working I can merge and update on haxelib

I tested it for both Android and iOS, it worked for me.


make provider.MissingValueException part of the published api

Expected Behavior
Make org.gradle.api.internal.provider.MissingValueException part of the published API.
Current Behavior (optional)
No response
Context
in order to test a ValueSource correctly I might also need to know when it's returning null
  @Test
  void obtainWithHeadBranchSetNoRemotes() {
    var hbvsp = project
      .getProviders()
      .of(HeadBranchValueSource.class, c -> {
        c.getParameters().getHeadBranch().set("other");
        c.getParameters().getProjectDir().set(project.getProjectDir());
      });

    assertThatExceptionOfType(MissingValueException.class).isThrownBy(() -> hbvsp.get());
  }

current version 8.10.2

This issue needs a decision from the team responsible for that area. They have been informed. Response time may vary.

@gradle/bt-configuration What should be the flow on the user side? Do we expect users to catch those exceptions or not?

We need more information to understand the request. Could you please provide more context about what you want to achieve?
If we receive a clear use case description, we may be able to suggest a workaround.
This issue will be closed after 7 days unless you provide more information.

The proposed use case seem to be served by Provider.isPresent or .getOrNull, and the exception type is already specified as IllegalStateException. Maybe there is another use case where a more precise type is required?

My doing, wanting this might have been due to one of the many other things that has surprised me that doesn't work. I'm not certain what the state of my code was when I created this that I thought this was a good idea.
I might have been trying to inject another value source and expecting it to throw this as I didn't set the state. If I return null in the ValueSource and I want that value returned and cached from a ValueSource or if a Property is set to null. I guess Provider is just a lazier Optional (dear java, why isn't Optional lazier like Stream, also why couldn't it have been my 3 states).
I'm not certain I've ever really agreed with Optional throwing an error on get(). it's generally a good practice to catch the most specific exception possible if you need to.
Btw, did anyone realize the name Provider conflicts with a class in "javax.inject".
I'm closing since I don't recall, unfortunately. It might also have been my brain "oh this throws... let's catch it in our test".

This is the intended behavior, and there’s no plan to change it. This issue will be closed.

So far I don't see a compelling use case to catch the exception coming from an unset provider specifically. Just like with arrays or Optional, the consumer is expected to check for the presence of the value before obtaining it, or use fallback-taking variants, like getOrElse.


Word bases suggestions stopped working properly being taken over by snippet suggestions

Issue Type: Bug
Description
I am editing TCL files. TCL plugin (tcl|bit|0.4.0) wasn't updated recently at all. For this specific setup, I also have a tcl.json - set of ~10k snippets from 3rd party library. Everything worked great since months if not year[s].
Up to recently everything worked as expected:

I wrote some partial name of variable, hit CTRL-SPACE and I got either a word based completion or a snippet completion, depending which pattern match was better

Then, the only thing I am aware, I revisited my tcl.json snippet file, added one more. After VS Code reloaded snippets, I no longer get a proper word based suggestions, unless NONE of the snippets is matched (which is hard to do with some kind of fuzzy matching). It looks almost like VS Code was using some cached version (working properly), then rebuilt something.

I type one letter - my completion consists only snippets

I type further, being very close to the name of variable - still non of the word based suggestion appears, only list of snippets suggestions gets narrowed down

if I type a combination that none of the snippet matches - then word suggestion somehow appears

Hitting ESC, CTRL-SPACE helps depending on a current input word and how it matches against snippets.
If I turn off snippet matching ("editor.snippetSuggestions": "none"), word based suggestion works fine. Setting it to "bottom" doesn't change anything (still doesn't work and I only get snippets).
It's like hitting some limiter that entirely turns off word based suggestions.
Steps to reproduce

install tcl|bit|0.4.0 https://marketplace.visualstudio.com/items?itemName=bitwisecook.tcl
CTRL-SHIFT-P, "Configuer User Snippets", pick language TCL, paste these snippets:{
    "my something": {
        "prefix": "my-something",
        "body": "my::something ${1}\n${0}"
    },
    "my function": {
        "prefix": "my-function",
        "body": "my::function ${1} ${2}\n${0}"
    }
}

open folder in VS Code, create somefile.tcl, paste in this content:namespace eval firstProgram {
   proc hello { {myVar world} } {
       # try to print myVar (command in TCL is `puts`, value of variable is
       # reached with $)
       # after $ character press ctrl-space

       puts $
   }
}

I can't workout the logic I see from editor's behavior.

If I input puts $m<ctrl-space>, I get auto suggestion triggered with snippets (sometimes triggers without ctrl-space); I should see words and snippets:

If I input puts $<ctrl-space>, I see list of words:

a. If I continue with letter I get myVar caught, I can finish the filling:

Barrier which changes snippets to words
a. puts $m as in 1.
b. puts $my similar:

c. puts $myV:

For this tests I remove suggestion related settings, so it used defaults.
VS Code version: Code 1.62.0 (b3318bc0524af3d74034b8bb8a64df0ccf35549a, 2021-11-03T15:23:01.379Z)
OS version: Windows_NT x64 10.0.18363
Restricted Mode: No
Remote OS version: Linux x64 3.10.0-1127.10.1.el7.x86_64

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz (8 x 2112)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledskia_renderer: enabled_onvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
15.88GB (8.07GB free)

Process Argv
--crash-reporter-id 5677dc62-2d7a-4388-bb68-4cdafcb6b515

Screen Reader
no

VM
0%

Item
Value

Remote
SSH: xxx

OS
Linux x64 3.10.0-1127.10.1.el7.x86_64

CPUs
Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz (80 x 2899)

Memory (System)
503.41GB (12.58GB free)

VM
0%

Extensions (32)

Extension
Author (truncated)
Version

bracket-pair-colorizer
Coe
1.0.61

vsc-material-theme
Equ
33.2.0

vsc-material-theme-icons
equ
1.2.2

jupyter-keymap
ms-
1.0.0

remote-ssh
ms-
0.66.0

remote-ssh-edit
ms-
0.66.0

remote-wsl
ms-
0.58.5

material-icon-theme
PKi
4.10.0

jinjahtml
sam
0.16.0

code-settings-sync
Sha
3.4.3

fish-vscode
sky
0.2.1

vscode-icons
vsc
11.7.0

vscode-todo-highlight
way
1.0.5

tcl
bit
0.4.0

prettier-vscode
esb
9.0.0

jenkins-pipeline-linter-connector
jan
1.2.0

rainbow-csv
mec
1.9.1

perforce
mjc
4.15.2

python
ms-
2021.11.1422169775

vscode-pylance
ms-
2021.11.0

jupyter-keymap
ms-
1.0.0

jupyter-renderers
ms-
1.0.3

vsliveshare
ms-
1.0.5065

autodocstring
njp
0.5.4

vscode-commons
red
0.0.6

vscode-yaml
red
1.1.0

vscode-sort-json
ric
1.20.0

LiveServer
rit
5.6.1

code-spell-checker
str
2.0.12

vscode-markdown-paste-image
tel
0.13.8

simple-rst
tro
1.5.0

markdown-all-in-one
yzh
3.4.0

(2 theme extensions excluded)

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythontb:30283811
pythonvspyt551:30345470
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscorecescf:30384386
pythondataviewer:30285071
vscod805:30301674
pythonvspyt200:30340761
binariesv615:30325510
vsccppwtct:30382698
bridge0708:30335490
pygetstartedt3:30385195
dockerwalkthru:30377721
bridge0723:30353136
pythonrunftest32:30373476
pythonf5test824:30373475
javagetstartedt:30391933
pythonvspyt187:30373474
vsqsis300:30386381
vsaa593:30376534
vssld246:30386377

Can you please share a concrete sample where words should be showing but aren't. What snippet/snippets are taking over in that case? What language is that?

TCL, as mentioned.
Not sure if I can share all snippets, sorry, bit I will try to prepare synthetic test case. Amount of snippets can be important though.
Sorry I didn't do it in the first post, but I hoped something obvious would pop out. That problem just sounds eccentric.

@jrieken, initial issue description updated.

I see this when trigger suggest via ctrl+space right after the $. Are you saying you don't see that?

That's exactly how it should work, yup.
Moved my settings.json, here it what I see:

It's like, whatever was combining the suggestions, got broken.

It's like, whatever was combining the suggestions, got broken.

Well, there is word based suggestions but no snippet suggestions. There is no snippets because you aren't in-between or inside words but after the $. In that case there is no snippets. If you trigger (!) suggest after $m you get both - word based suggestions and snippets

You're right, sorry, missed that you typed m. But you have an example how it looks at my side in first message. With my settings.json moved away nothing changed:

https://user-images.githubusercontent.com/1786383/140521805-b72b47a9-d2aa-4211-ace8-82fa5e34874b.mp4

ctrl-space after $ only triggers word completion, as you described
adding m narrows down choice to one variable
escaping, retriggering suggestions with $m leaves me with snippets only

Can start VS Code like so code --user-data-dir ~/Desktop/fresh and try again? With that command you have a truly fresh workspace and zero chance of conflicting settings. Also, make sure you have no settings in your workspace (.vscode-folder)

Tried --user-data-dir, didn't help. No .vscode folder as well.
--disable-extensions makes TCL non-functional. Is there a way to just enable one extension?

One more try with --disable-extensions, I moved snippets to a global configuration and it seems to work just fine:

@jrieken, in a test you made here: https://github.com/microsoft/vscode/issues/136500#issuecomment-961888978
Did you try it more than once? ESC the suggestion, then try again? Delete m, then try again re-typing?
It seems that in my production environment, after reload of a window, the first time I type puts $m, popup is correct:

Yet the very next usage drops a word suggestion:

Did you try it more than once? ESC the suggestion, then try again? Delete m, then try again re-typing?

Yes, works repeatedly for me.

I removed Code folder from AppData/Roaming, as well as .vscode one from C:/Users/.
I am guessing the latter may have help here, as the first one we covered with additional switch previously. Maybe something else was still pulled from global are though...
.vscode one keeps extensions folders, maybe some of them contained some broken data... I still relate this issue with a moment in which VS Code recomputed big amount of snippets for particular language.

After some time bug returned, word based completion still do not work properly for me.

@jrieken, what platform did you use for testing? From your previous help I am guessing it was Linux of some sort.
I am having this issue on Windows, maybe here we have difference.

I tried for an hour to replicate the issue on another windows installation, personal one. No luck, until I tried to install plugins one by one to even it out with my initial setup.
It's code-spell-checker.
Code spell checker not installed:

Code spell checker installed (just in case try triggering suggestions couple of times):


BN should be used after ReLU

https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/convolutional_neural_network/main.py#L41-L42
nn.ReLU(),
nn.BatchNorm2d(...),

BN should be used after ReLU. Features might be truncated by non-linearity like ReLU, so BN is used to normalize the distribution of features.

I have been also suffered from this issue.
When ReLU is used after BN, Loss is not converged at all.
nn.BatchNorm2d(16),
nn.ReLU(),

...
Epoch [0/10], Step: [200/600], Loss: [0.39470914006233215]
Epoch [0/10], Step: [400/600], Loss: [0.23821884393692017]
Epoch [0/10], Step: [600/600], Loss: [0.1606658548116684]
...
Epoch [9/10], Step: [200/600], Loss: [0.010754228569567204]
Epoch [9/10], Step: [400/600], Loss: [0.047237977385520935]
Epoch [9/10], Step: [600/600], Loss: [0.06365286558866501]
Test Accuracy: [9.78]


Black screen with audio - Remux movies

Hi moneytoo. I am using Just player as my number one video player, but I have problems playing "remux 4k movies". Every time I try to play it, my screen goes black, but it plays sound. I tested other video players, such as default exo player, vlc, mxplayer etc. and I don't have that problem. Is there any way to fix this? I really like your player, so if you have any idea how to solve this issue I would be thankful. Enjoy your day !

I have the same problem

Just Player uses ExoPlayer so it should generally play everything ExoPlayer plays. If you have issue with particular video file, please share a sample of such file.

The.Northman.2022.2160p.BluRay.REMUX.HEVC.DTS-HD.MA.TrueHD.7.1.Atmos-FGT
Everything.Everywhere.All.at.Once.2022.2160p.BluRay.REMUX.HEVC.DTS-HD.MA.TrueHD.7.1.Atmos-FGT
Minions.The.Rise.of.Gru.2022.2160p.BluRay.REMUX.HEVC.DTS-HD.MA.TrueHD.7.1.Atmos-FGT

Minions.The.Rise.of.Gru.2022.2160p.BluRay.REMUX.HEVC.DTS-HD.MA.TrueHD.7.1.Atmos-FGT

Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5.1@High
HDR format                               : Dolby Vision, Version 1.0, dvhe.07.06, BL+EL+RPU, Blu-ray compatible / SMPTE ST 2086, HDR10 compatible
Codec ID                                 : V_MPEGH/ISO/HEVC

This is a Dolby Vision file so with the latest stable ExoPlayer it plays only on devices supporting DV as it doesn't fallback to HDR. #166, https://github.com/google/ExoPlayer/issues/9794

Release v0.125 includes ExoPlayer r2.18.2 which has some DV related improvements. Does it work any better for you?

Duplicate of #270. The one file I downloaded was DV profile 7. One of the upcomings update should include an option to fallback to HDR.


cannot build with swift

$ swift build
error: missing LinuxMain.swift file in the Tests directory

I got something like this when building the project. My swift version is:
$ swift --version
Swift version 5.0.2 (swift-5.0.2-RELEASE)
Target: x86_64-unknown-linux-gnu

Could you please help to fix this issue? thanks a lot!

This issue affected fuzzilli some time ago; you should be able to touch the file in the repo root and successfully build. That being said, a more up-to-date version of this work is to be found here: https://github.com/googleprojectzero/fuzzilli/pull/378

Hi thank you, I have solved this problem and successfully compile the code.
While when running it, I got such an error:
$ swift run FuzzilliCli --profile=jerryscript --inspect=history --storagePath="./output" --minimizationLimit=1.0 /tmp/jerryscript-8ba0d1b6ee/build/bin/jerry
Building for debugging...
Build complete! (0.20s)
[REPRL] Script execution failed: Child in weird state after execution. Retrying in 1 second...
[REPRL] Script execution failed again: Child in weird state after execution. Giving up
[Coverage] Initialized, 14544 edges
[JavaScriptEnvironment] Initialized static JS environment model
[JavaScriptEnvironment] Have 50 available builtins: ["Date", "Reflect", "undefined", "resourceName", "isNaN", "gc", "BigInt", "ArrayBuffer", "Int16Array", "WeakMap", "Boolean", "Number", "WeakSet", "Array", "Uint8ClampedArray", "Float32Array", "RangeError", "Object", "Infinity", "print", "Float64Array", "Map", "Int8Array", "NaN", "parseInt", "EvalError", "URIError", "DataView", "parseFloat", "isFinite", "String", "TypeError", "placeholder", "Math", "Uint8Array", "Promise", "Set", "SyntaxError", "Uint16Array", "Function", "Error", "RegExp", "Symbol", "AggregateError", "Int32Array", "Uint32Array", "Proxy", "eval", "JSON", "ReferenceError"]
[JavaScriptEnvironment] Have 212 available method names: ["stringify", "defineProperties", "m", "getMinutes", "indexOf", "apply", "log", "floor", "setFloat64", "isArray", "abs", "pop", "getMilliseconds", "getUTCSeconds", "keys", "push", "padStart", "cosh", "toUpperCase", "resolve", "getTime", "acos", "getOwnPropertySymbols", "setPrototypeOf", "sinh", "reduceRight", "shift", "getUTCDay", "test", "isFinite", "setHours", "setUTCSeconds", "getUTCHours", "getFloat32", "min", "getFullYear", "catch", "bind", "reverse", "raw", "expm1", "n", "replaceAll", "getUint16", "construct", "getInt32", "call", "asIntN", "is", "cos", "has", "cbrt", "replace", "some", "startsWith", "fround", "set", "parse", "entries", "toString", "flatMap", "allSettled", "imul", "then", "log10", "setMilliseconds", "isView", "o", "find", "getInt16", "race", "getUTCMinutes", "random", "exp", "setInt32", "round", "flat", "getDay", "ownKeys", "endsWith", "getUint8", "getMonth", "search", "trimStart", "setUTCMinutes", "filter", "setInt16", "toISOString", "getUTCMonth", "map", "match", "tan", "findIndex", "substring", "UTC", "get", "getUTCMilliseconds", "trim", "resize", "trunc", "asin", "toTimeString", "setUTCFullYear", "isInteger", "setUint32", "getFloat64", "getOwnPropertyNames", "includes", "toLocaleString", "getInt8", "lastIndexOf", "getOwnPropertyDescriptors", "padEnd", "setUTCMonth", "forEach", "setYear", "unshift", "seal", "defineProperty", "getUTCDate", "join", "acosh", "charAt", "splice", "isFrozen", "trimLeft", "toJSON", "fromCodePoint", "charCodeAt", "pow", "finally", "setMonth", "getOwnPropertyDescriptor", "toLowerCase", "setSeconds", "matchAll", "assign", "log2", "setUTCMilliseconds", "tanh", "hypot", "setUTCDate", "localeCompare", "isExtensible", "delete", "trimRight", "reject", "subarray", "keyFor", "all", "setDate", "deleteProperty", "p", "toGMTString", "asinh", "split", "isSafeInteger", "asUintN", "max", "repeat", "slice", "setInt8", "for", "getHours", "setMinutes", "getTimezoneOffset", "setFloat32", "values", "freeze", "compile", "sin", "create", "setUint16", "every", "ceil", "setUTCHours", "fromCharCode", "from", "atan", "concat", "fromEntries", "isSealed", "trimEnd", "sign", "getUTCFullYear", "setUint8", "codePointAt", "clz32", "getSeconds", "fill", "now", "atan2", "getYear", "of", "setTime", "copyWithin", "preventExtensions", "isNaN", "clear", "reduce", "getDate", "sort", "toUTCString", "exec", "getPrototypeOf", "toDateString", "add", "sqrt", "setFullYear", "atanh", "getUint32", "log1p"]
[JavaScriptEnvironment] Have 52 property names that are available for read access: ["unscopables", "global", "size", "message", "species", "caller", "b", "hasInstance", "prototype", "POSITIVE_INFINITY", "byteOffset", "EPSILON", "constructor", "buffer", "replace", "PI", "dotAll", "__proto__", "source", "cause", "MIN_VALUE", "e", "c", "flags", "MAX_VALUE", "toPrimitive", "E", "d", "unicode", "multiline", "matchAll", "ignoreCase", "search", "MAX_SAFE_INTEGER", "arguments", "NEGATIVE_INFINITY", "byteLength", "description", "NaN", "MIN_SAFE_INTEGER", "split", "valueOf", "toStringTag", "iterator", "name", "toString", "length", "asyncIterator", "a", "sticky", "isConcatSpreadable", "match"]
[JavaScriptEnvironment] Have 10 property names that are available for write access: ["a", "toString", "constructor", "c", "valueOf", "length", "d", "__proto__", "e", "b"]
[JavaScriptEnvironment] Have 5 custom property names: ["c", "a", "b", "e", "d"]
[JavaScriptEnvironment] Have 4 custom method names: ["p", "n", "o", "m"]
[Fuzzer] Initialized
[REPRL] Script execution failed: Child in weird state after execution. Retrying in 1 second...
[REPRL] Script execution failed again: Child in weird state after execution. Giving up
[Fuzzer] Cannot execute programs (exit code must be zero when no exception was thrown). Are the command line flags valid?
[Fuzzer] Shutting down due to fatal error

++++++++++ Fuzzer Finished ++++++++++

Fuzzer Statistics
-----------------
Fuzzer phase:                 Fuzzing (with MutationEngine)
Uptime:                       0d 0h 0m 3s
Total Samples:                0
Interesting Samples Found:    0
Last Interesting Sample:      0d 0h 0m 3s
Valid Samples Found:          0
Corpus Size:                  0
Correctness Rate:             -nan% (-nan%)
Timeout Rate:                 -nan% (-nan%)
Crashes Found:                0
Differentials Found:          0
Timeouts Hit:                 0
Coverage:                     0.00%
Avg. program size:            -nan
Avg. corpus program size:     -nan
Connected workers:            0
Execs / Second:               0.00
Fuzzer Overhead:              100.00%
Total Execs:                  1
Differential Tests:           0

The following error message shows up twice:
[REPRL] Script execution failed: Child in weird state after execution. Retrying in 1 second...
[REPRL] Script execution failed again: Child in weird state after execution. Giving up
Do you know why I got this error or how to bypass it? Thank you!

Today I further inspect the code by printing out the stderr and stdout of the failed script execution, I found the stderr being empty and stdout being:

[COV] edge counters initialized. Shared memory: shm_id_867741_0 with 767659 edges
Exception: ReferenceError: Can't find variable: fuzzilli_hash
main@[REPRL]:2:28
global code@[REPRL]:7:5

So it seems the bug is related to fuzzilli_hash variable. And I also found your PR to Fuzzilli repo, does that mean differential testing has been fully integrated? If so I can try differential testing on Fuzzilli, since I have already successfully run Fuzzilli on crash finding.

Hi thank you, I have solved this problem and successfully compile the code. While when running it, I got such an error:
$ swift run FuzzilliCli --profile=jerryscript --inspect=history --storagePath="./output" --minimizationLimit=1.0 /tmp/jerryscript-8ba0d1b6ee/build/bin/jerry
Building for debugging...
Build complete! (0.20s)
[REPRL] Script execution failed: Child in weird state after execution. Retrying in 1 second...
[REPRL] Script execution failed again: Child in weird state after execution. Giving up
[Coverage] Initialized, 14544 edges
[JavaScriptEnvironment] Initialized static JS environment model
[JavaScriptEnvironment] Have 50 available builtins: ["Date", "Reflect", "undefined", "resourceName", "isNaN", "gc", "BigInt", "ArrayBuffer", "Int16Array", "WeakMap", "Boolean", "Number", "WeakSet", "Array", "Uint8ClampedArray", "Float32Array", "RangeError", "Object", "Infinity", "print", "Float64Array", "Map", "Int8Array", "NaN", "parseInt", "EvalError", "URIError", "DataView", "parseFloat", "isFinite", "String", "TypeError", "placeholder", "Math", "Uint8Array", "Promise", "Set", "SyntaxError", "Uint16Array", "Function", "Error", "RegExp", "Symbol", "AggregateError", "Int32Array", "Uint32Array", "Proxy", "eval", "JSON", "ReferenceError"]
[JavaScriptEnvironment] Have 212 available method names: ["stringify", "defineProperties", "m", "getMinutes", "indexOf", "apply", "log", "floor", "setFloat64", "isArray", "abs", "pop", "getMilliseconds", "getUTCSeconds", "keys", "push", "padStart", "cosh", "toUpperCase", "resolve", "getTime", "acos", "getOwnPropertySymbols", "setPrototypeOf", "sinh", "reduceRight", "shift", "getUTCDay", "test", "isFinite", "setHours", "setUTCSeconds", "getUTCHours", "getFloat32", "min", "getFullYear", "catch", "bind", "reverse", "raw", "expm1", "n", "replaceAll", "getUint16", "construct", "getInt32", "call", "asIntN", "is", "cos", "has", "cbrt", "replace", "some", "startsWith", "fround", "set", "parse", "entries", "toString", "flatMap", "allSettled", "imul", "then", "log10", "setMilliseconds", "isView", "o", "find", "getInt16", "race", "getUTCMinutes", "random", "exp", "setInt32", "round", "flat", "getDay", "ownKeys", "endsWith", "getUint8", "getMonth", "search", "trimStart", "setUTCMinutes", "filter", "setInt16", "toISOString", "getUTCMonth", "map", "match", "tan", "findIndex", "substring", "UTC", "get", "getUTCMilliseconds", "trim", "resize", "trunc", "asin", "toTimeString", "setUTCFullYear", "isInteger", "setUint32", "getFloat64", "getOwnPropertyNames", "includes", "toLocaleString", "getInt8", "lastIndexOf", "getOwnPropertyDescriptors", "padEnd", "setUTCMonth", "forEach", "setYear", "unshift", "seal", "defineProperty", "getUTCDate", "join", "acosh", "charAt", "splice", "isFrozen", "trimLeft", "toJSON", "fromCodePoint", "charCodeAt", "pow", "finally", "setMonth", "getOwnPropertyDescriptor", "toLowerCase", "setSeconds", "matchAll", "assign", "log2", "setUTCMilliseconds", "tanh", "hypot", "setUTCDate", "localeCompare", "isExtensible", "delete", "trimRight", "reject", "subarray", "keyFor", "all", "setDate", "deleteProperty", "p", "toGMTString", "asinh", "split", "isSafeInteger", "asUintN", "max", "repeat", "slice", "setInt8", "for", "getHours", "setMinutes", "getTimezoneOffset", "setFloat32", "values", "freeze", "compile", "sin", "create", "setUint16", "every", "ceil", "setUTCHours", "fromCharCode", "from", "atan", "concat", "fromEntries", "isSealed", "trimEnd", "sign", "getUTCFullYear", "setUint8", "codePointAt", "clz32", "getSeconds", "fill", "now", "atan2", "getYear", "of", "setTime", "copyWithin", "preventExtensions", "isNaN", "clear", "reduce", "getDate", "sort", "toUTCString", "exec", "getPrototypeOf", "toDateString", "add", "sqrt", "setFullYear", "atanh", "getUint32", "log1p"]
[JavaScriptEnvironment] Have 52 property names that are available for read access: ["unscopables", "global", "size", "message", "species", "caller", "b", "hasInstance", "prototype", "POSITIVE_INFINITY", "byteOffset", "EPSILON", "constructor", "buffer", "replace", "PI", "dotAll", "__proto__", "source", "cause", "MIN_VALUE", "e", "c", "flags", "MAX_VALUE", "toPrimitive", "E", "d", "unicode", "multiline", "matchAll", "ignoreCase", "search", "MAX_SAFE_INTEGER", "arguments", "NEGATIVE_INFINITY", "byteLength", "description", "NaN", "MIN_SAFE_INTEGER", "split", "valueOf", "toStringTag", "iterator", "name", "toString", "length", "asyncIterator", "a", "sticky", "isConcatSpreadable", "match"]
[JavaScriptEnvironment] Have 10 property names that are available for write access: ["a", "toString", "constructor", "c", "valueOf", "length", "d", "__proto__", "e", "b"]
[JavaScriptEnvironment] Have 5 custom property names: ["c", "a", "b", "e", "d"]
[JavaScriptEnvironment] Have 4 custom method names: ["p", "n", "o", "m"]
[Fuzzer] Initialized
[REPRL] Script execution failed: Child in weird state after execution. Retrying in 1 second...
[REPRL] Script execution failed again: Child in weird state after execution. Giving up
[Fuzzer] Cannot execute programs (exit code must be zero when no exception was thrown). Are the command line flags valid?
[Fuzzer] Shutting down due to fatal error

++++++++++ Fuzzer Finished ++++++++++

Fuzzer Statistics
-----------------
Fuzzer phase:                 Fuzzing (with MutationEngine)
Uptime:                       0d 0h 0m 3s
Total Samples:                0
Interesting Samples Found:    0
Last Interesting Sample:      0d 0h 0m 3s
Valid Samples Found:          0
Corpus Size:                  0
Correctness Rate:             -nan% (-nan%)
Timeout Rate:                 -nan% (-nan%)
Crashes Found:                0
Differentials Found:          0
Timeouts Hit:                 0
Coverage:                     0.00%
Avg. program size:            -nan
Avg. corpus program size:     -nan
Connected workers:            0
Execs / Second:               0.00
Fuzzer Overhead:              100.00%
Total Execs:                  1
Differential Tests:           0

The following error message shows up twice: [REPRL] Script execution failed: Child in weird state after execution. Retrying in 1 second... [REPRL] Script execution failed again: Child in weird state after execution. Giving up
Do you know why I got this error or how to bypass it? Thank you!

Today, I encountered the same issue while running, and I would like to know if this issue has been resolved. Could you please tell me?

The implementation deviates from the standard fuzzilli reprl interface by adding more parameters. This change is implemented for jsc, v8 and spidermonkey only. If you'd like to fuzz jerryscript you'll need to extend the respective bindings. Furthermore, the fuzzilli_hash function is not implemented on jerryscript, so that is something you'd need to add also.

The implementation deviates from the standard fuzzilli reprl interface by adding more parameters. This change is implemented for jsc, v8 and spidermonkey only. If you'd like to fuzz jerryscript you'll need to extend the respective bindings. Furthermore, the fuzzilli_hash function is not implemented on jerryscript, so that is something you'd need to add also.

Your advice has been incredibly helpful to me, and I will proceed to explore further based on your recommendations. Thank you for your reply!


Fix bugs

Pull Request Test Coverage Report for Build 77

5 of 5 (100.0%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.03%) to 88.825%

Totals

Change from base Build 75:
0.03%

Covered Lines:
1550

Relevant Lines:
1745

💛  - Coveralls