[PROPOSAL]: Data Skipping Indexes

Problem Statement
Add support for data skipping indexes.
Background and Motivation
Hyperspace has been supporting hash-partitioned covering indexes only. Covering indexes are good for some workloads, but we should have more index types to facilitate a broader range of use cases.
Proposed Solution
Refactor existing code by moving covering index specific code into a single implementation for a new Index API and implementing data skipping indexes for the API.
Alternatives
The proposed solution requires heavy refactoring, as there is a lot of code specific to covering indexes scattered throughout the codebase. Alternatively, we can make DataSkippingIndex without introducing a new API and write many if-elses depending on the index type. It might be quick to implement the required features for now in this way, but it will make the code hard to maintain in the long run.
Known/Potential Compatibility Issues
Regarding data compatibility, it is very likely that this will make the new version of Hyperspace incompatible with indexes created in the older versions. This is unfortunate but can be excused as Hyperspace is still in version zero. If a strong need arises, this can be mitigated by a migration utility that can be designed separately.
The API compatibility can be retained if we introduce new optional arguments to IndexConfig. However, includedColumns doesn't make sense for data skipping indexes, and newly added arguments won't for covering indexes. Therefore, it might be wise to break compatibility now and make things clean for the future.
Design
Data Skipping Index Basics
Data skipping indexes can filter out certain files that cannot have the desired values in indexed columns based on statistics or sketches on the values in files. Examples of statistics or sketches include min/max values and bloom filters.
Suppose that we have sample data like below. There is a relation df which has two columns, a and b, and consists of two files, /data/p0 and /data/p1:
/data/p0:

a
b

1
2

2
4

6
6

/data/p1:

a
b

5
10

10
10

A min-max data skipping index on a column a will look like this:

fileid
min
max

0
1
6

1
5
10

The file ID column is used to map files to unique identifiers. Identifiers are created based on the file path, size, and modification time.
Then, when a query like df.filter($"a" < 4) is given, we know that we can skip the /data/p1 file.
Note that the real implementation will contain file size and modified time columns for files, to distinguish files with the same name but different sizes/modified times to support deleted/added source files.
Storing Index Data
The index data can be stored as parquet files like it is done for covering indexes. The difference here is that we don't need bucketing. Also, the data size is a lot less because there is only a single row for each file.
To speed up range queries like the last example, it might be necessary that the data is cached in the main memory as sorted sequences ordered by min and max. This should be cheap to compute as the data size is expected to be small.
If there are too many files, we might need to consider structures like B-tree to access relevant files only. But this might not be a real issue; the index data should be orders of magnitude smaller than the source data.
Index Application
Data skipping indexes can be applied in three steps:

Extract constraints about key columns.
Filter out irrelevant files with the constraints and index data.
Rewrite the logical relations with the remaining files.

Unlike covering indexes, different data skipping indexes can be applied for the same relation multiple times.
Examples of constraints include a < 4, a = 10, or a in (1, 2, 3). Combined with statistics or sketches like min/max values and/or bloom filters, this information can be used to weed out irrelevant files.
Theoretically, applying data skipping indexes for joins is possible. Still, the implementation will be complex, and benefits are unclear over the traditional broadcast hash join, especially when the indexed columns have a low correlation with the partitioning columns. Therefore, we will focus on filters for now.
Hybrid scan is trivial and requires no special handling for data skipping indexes, as we only filter out irrelevant files and the remaining files are not touched.
The estimated cost/benefit of applying data skipping indexes - TODO
Index Maintenance
Given a dataframe (a single relation consisting of files), an index name, a column/columns to index, a statistics/sketch type (min/max, bloom filter, etc.), we can create a data skipping index by either scanning every row in the files and building the specified statistics or looking up existing statistics in the files.
Incremental refresh is possible, and in fact, is not much different from a full refresh. During refresh, we drop rows for deleted files from the index data and add rows for added files. If there are no deleted files, incremental refresh can store new rows in new files, not touching old files. Full refresh can then merge such files later.
Implementation
PRs

#443
#481
#461
#482
#483
#486

TODO

String type support enhancement

Implement sketch types specialized for string typed data
Support predicates useful for strings such as LIKE

BloomFilterSketch user experience enhancement

Allow users to omit the expected number of distinct values. It can be done with an additional query with approx_count_distinct. The current implementation doesn't allow setting different numbers of distinct values for each file, so we should determine which value we will use, such as mean, median, or n-th percentile of the distinct counts.
Allow users to specify the target index size, absolute or relative to the source data size, to determine the false positive probability automatically.

Supportability features

Provide the index effectiveness score. The rationale is that the data skipping index is not for all types of source data (layout). For example, if there is a low-cardinality column and its distribution is uncorrelated with its row position (i.e. data is randomly distributed across files), it is impossible to filter out files because every file has every possible value. If a high-cardinality column (extreme case - unique column) has a random distribution, then MinMaxSketch will be ineffective and only sketches like BloomFilterSketch will be useful. We can provide a score, for example, in the range of 0-100. 0 is for absolutely useless indexes and 100 is for fantastic ones. Here's an example: if for every file min/max values are very similar to the min/max values of the entire column, then the index will be ineffective, and its score will be close to 0.

Dynamic file pruning

Similar to dynamic partition pruning, we can implement dynamic file pruning with the data skipping index. To do this, we can copy the existing Spark's dynamic partition pruning optimization rule and modify it to do file pruning instead of partition pruning.

More sketch types
Check if Spark gives you the partition-pruned file lists when we apply data skipping indexes. If it doesn't, then we should do partition-pruning ourselves prior to filtering out files to speed up the process.

@clee704, @imback82, @sezruby, @rapoth: There is another OSS data skipping index called XSkipper which has been developed by IBM. Maybe we can just plug it in in Hyperspace.

Thanks @andrei-ionescu for the info. I just took a quick look at it and it doesn't seem straightforward to plug it in (@clee704 / @sezruby please correct me if I am wrong here), and since the project is fairly new, I am not sure if it's a good idea to have a dependency on it at this stage. Also, the index maintainence scenario seems different (no support for concurrent updates, etc.) But we will revisit when the project becomes mature.
This proposal is a scoped down version of #335 (and adding few more variations) to support filter queries first.

@andrei-ionescu I've skimmed overall XSkipper codebase and I don't think we could easily plug it in. We are planning to support similar features - BF, (min/max), partition elimination index (similar to valueList). FYI @clee704 is working on this data skipping feature now.

Implementation added as #443.

This a good direction to work on. Skipping indexes are very helpful in the context of very big datasets: terabytes of data spanning across millions of files.
The start of the proposal is good but it needs more information and we need to have at least some parial answers to the following questions:

Will the index be bucketed?
What's the order inside the index? Is it ordered by the file names or by the min/max values? Having the index ordered and bucketed by the min/max values will reduce the query time. Having the index ordered by file names will help at update time. Can we have both?
If the index contains a lot of min/max overlaps how it will be decided wether to use the index or not?
How is data restatement handled (ie: replace /data/p1 with a new /data/p1 with different values that will require re-compute and update the min/max values in the index)?
Will there be a different API for this type of index?
Updating the index in the case of 10x millions of files? What's the process to add 1 files with 100 million rows with values in-between the already present index? Will it require a full rebuild?
What the targeted SLA for consistency? I think Hyperspace offers consistency when the hybrid scan is enabled, but otherwise is there an SLA for getting to consistent state? I'm bringing this because if the index gets bigger and bigger the process of keeping it up to date may become slower and slower thus not being consistent with the data itself.
Do we mark also the null values? How are the null values handled because they are neither min nor max?

Some more information would be very helpful on how the plans will look. Can you provide some snippets of such plans altered to make use of skip indexes?

@clee704, @sezruby Could you go through this proposal again and add more information in regards to the raised concerns above? Thanks!

Will the index be bucketed?

No, because the index data is expected to be very small compared to the source data and a full scan of index data (or using a cached one) should be fine. For example, suppose we have 100 TB of data, stored in files, each file being 1 GB. It means there are 100,000 files and our index data has 100,000 rows. It might seem large, but actually it's not compared to the 100 TB of data.
If a full scan turns out to be too slow, then we might consider adding indexes to the index data. For example, we can index min/max sketches to speed up range queries.

What's the order inside the index? Is it ordered by the file names or by the min/max values? Having the index ordered and bucketed by the min/max values will reduce the query time. Having the index ordered by file names will help at update time. Can we have both?

There is no order, and having the index data ordered by file names won't help much because we don't support mutable files. All we support is deleted files and newly added files.
For incremental refreshes, we just ignore deleted files and add new index data files for newly appended files.
If deletion is a frequent operation, then we might consider partitioning the index data in the same way as the source data for a faster deletion of index data when an entire partition of source data is deleted, which seems to be a common operation in the field.
For ordering by min/max values, please refer to the first answer above.

If the index contains a lot of min/max overlaps how it will be decided wether to use the index or not?

We'll start with a naive approach which is just applying the index regardless of the effectiveness. Except for the index processing time, the query plan doesn't get worse and the index processing time would be dwarfed by the query processing time.
Later, we might add mechanisms to help users, such as notifying the expected effectiveness of an index based on the selected sketches.

How is data restatement handled (ie: replace /data/p1 with a new /data/p1 with different values that will require re-compute and update the min/max values in the index)?

In Hyperspace, a file is considered to be equal to another file only if their paths, sizes, and modified times are the same. The tuple (path, size, modified time) can identify files in cases like the one you mentioned. I'll update the examples.

Will there be a different API for this type of index?

For users, the only difference is that they provide a different type of IndexConfig to Hyperspace.createIndex. Users can use DataSkippingIndexConfig to create data skipping indexes. Other than that the API is the same, i.e. methods like refreshIndex or optimizeIndex can be done with the same interface.

Updating the index in the case of 10x millions of files? What's the process to add 1 files with 100 million rows with values in-between the already present index? Will it require a full rebuild?

Adding new files is easy. New index data will be stored in new parquet files.
Spark will be unusably slow with 10x millions of files to begin with, even without Hyperspace. In the first version of data skipping indexes, we will focus on a smaller number of files and will consider improving the performance for a large number of files.

What the targeted SLA for consistency? I think Hyperspace offers consistency when the hybrid scan is enabled, but otherwise is there an SLA for getting to consistent state? I'm bringing this because if the index gets bigger and bigger the process of keeping it up to date may become slower and slower thus not being consistent with the data itself.

What does SLA stand for? Anyway, incremental refresh can add files and optimize can reduce the number of files. This is more or less the same as covering indexes.

Do we mark also the null values? How are the null values handled because they are neither min nor max?

For MinMaxSketch, we will ignore null values and just consider non-null values only. It means filters like a > 10 can be accelerated while filters like a > 10 or a is null cannot. We can add something like NullSketch if filters allowing null values are common.

Some more information would be very helpful on how the plans will look. Can you provide some snippets of such plans altered to make use of skip indexes?

A basic example is given in the proposal. Please refer to the section "Data Skipping Index Basics". The transformed plan will be the same as the original plan, except that the relation node in the transformed plan will have fewer files than the original one.

How is data restatement handled (ie: replace /data/p1 with a new /data/p1 with different values that will require re-compute and update the min/max values in the index)?

In Hyperspace, a file is considered to be equal to another file only if their paths, sizes, and modified times are the same. The tuple (path, size, modified time) can identify files in cases like the one you mentioned. I'll update the examples.

Can you link me to the updated example? I couldn't find them. Thanks

Updating the index in the case of 10x millions of files? What's the process to add 1 files with 100 million rows with values in-between the already present index? Will it require a full rebuild?

Adding new files is easy. New index data will be stored in new parquet files.

So, it will add a new skipping index entry in the index with /data/p999999, min: -999999, max: 999999 overlapping with the majority of the already existing entries. No rebuild.

Spark will be unusably slow with 10x millions of files to begin with, even without Hyperspace. In the first version of data skipping indexes, we will focus on a smaller number of files and will consider improving the performance for a large number of files.

Depending on the Spark cluster size, 10 millions of files can be easily processed in parallel. Given the biggest Spark cluster size can hold up 8000 nodes, and that max cores in a VM is about 32, the total number of files to be processed in parallel is 256000. For 1 million, 4 times this process is easily doable, and even 40 times the process is easily achievable.
Now, Hyperspace is the key. If the data skipping is well defined and implemented, then instead of using such a big cluster, it may reduce to a small cluster to achieve the same result.
We do have many of such datasets, PetaBytes of data with millions of files. 1PB / 1 000 000 = 1Gb, means 1 million of 1Gb files means 1Pb of data. This is a use case that we see more and more.
Data Skipping index working for small datasets is a helper but having it work for very big datasets will be a game changer.
So, please think big and try to accommodate the heavy scenarios.

What the targeted SLA for consistency? I think Hyperspace offers consistency when the hybrid scan is enabled, but otherwise is there an SLA for getting to consistent state? I'm bringing this because if the index gets bigger and bigger the process of keeping it up to date may become slower and slower thus not being consistent with the data itself.

What does SLA stand for?

SLA = Service Level Agreement.
My question is what is the time from a dataset being changed and the index being brought up to date with the new data? Or, what's the time for a incremental refresh in the case of Data Skipping Index? Is this update time keeping constant over the time while the data grows?
I'm bringing these questions up because if my data update/append takes 10s and the data skipping index update takes 10min then the index will be unusable in the fast changing datasets use cases.

Some more information would be very helpful on how the plans will look. Can you provide some snippets of such plans altered to make use of skip indexes?

A basic example is given in the proposal. Please refer to the section "Data Skipping Index Basics". The transformed plan will be the same as the original plan, except that the relation node in the transformed plan will have fewer files than the original one.

I did read the example and make sense. I'm more interested in the Spark Plan itself. Like how the .queryExecution.optimizedPlan and .queryExecution.sparkPlan looks at the query time.
@rapoth, @imback82, @sezruby: I'm CC'ing you as the discussion is in part leaning on the product definition side.

How is data restatement handled (ie: replace /data/p1 with a new /data/p1 with different values that will require re-compute and update the min/max values in the index)?

In Hyperspace, a file is considered to be equal to another file only if their paths, sizes, and modified times are the same. The tuple (path, size, modified time) can identify files in cases like the one you mentioned. I'll update the examples.

Can you link me to the updated example? I couldn't find them. Thanks

It is in the PR description: https://github.com/microsoft/hyperspace/pull/461. I've also updated the issue description.

Updating the index in the case of 10x millions of files? What's the process to add 1 files with 100 million rows with values in-between the already present index? Will it require a full rebuild?

Adding new files is easy. New index data will be stored in new parquet files.

So, it will add a new skipping index entry in the index with /data/p999999, min: -999999, max: 999999 overlapping with the majority of the already existing entries. No rebuild.

That's right. Incremental refresh will only add index data files unless there are deleted source data files. If there are deleted source data files, the index data is rewritten without entries for those deleted files, but this should be a cheap operation compare to reading all source data files again.

Spark will be unusably slow with 10x millions of files to begin with, even without Hyperspace. In the first version of data skipping indexes, we will focus on a smaller number of files and will consider improving the performance for a large number of files.

Depending on the Spark cluster size, 10 millions of files can be easily processed in parallel. Given the biggest Spark cluster size can hold up 8000 nodes, and that max cores in a VM is about 32, the total number of files to be processed in parallel is 256000. For 1 million, 4 times this process is easily doable, and even 40 times the process is easily achievable.
Now, Hyperspace is the key. If the data skipping is well defined and implemented, then instead of using such a big cluster, it may reduce to a small cluster to achieve the same result.
We do have many of such datasets, PetaBytes of data with millions of files. 1PB / 1 000 000 = 1Gb, means 1 million of 1Gb files means 1Pb of data. This is a use case that we see more and more.
Data Skipping index working for small datasets is a helper but having it work for very big datasets will be a game changer.
So, please think big and try to accommodate the heavy scenarios.

Data skipping index per se will be scalable with >1 million files. The problem here is that the way we process files in Hyperspace is not optimized for it. We'll keep improving things in this regard.

What the targeted SLA for consistency? I think Hyperspace offers consistency when the hybrid scan is enabled, but otherwise is there an SLA for getting to consistent state? I'm bringing this because if the index gets bigger and bigger the process of keeping it up to date may become slower and slower thus not being consistent with the data itself.

What does SLA stand for?

SLA = Service Level Agreement.
My question is what is the time from a dataset being changed and the index being brought up to date with the new data? Or, what's the time for a incremental refresh in the case of Data Skipping Index? Is this update time keeping constant over the time while the data grows?
I'm bringing these questions up because if my data update/append takes 10s and the data skipping index update takes 10min then the index will be unusable in the fast changing datasets use cases.

Incremental refresh should be fast, because it only looks at new data and appends new index data files without accessing previous source data or index data.

Some more information would be very helpful on how the plans will look. Can you provide some snippets of such plans altered to make use of skip indexes?

A basic example is given in the proposal. Please refer to the section "Data Skipping Index Basics". The transformed plan will be the same as the original plan, except that the relation node in the transformed plan will have fewer files than the original one.

I did read the example and make sense. I'm more interested in the Spark Plan itself. Like how the .queryExecution.optimizedPlan and .queryExecution.sparkPlan looks at the query time.
@rapoth, @imback82, @sezruby: I'm CC'ing you as the discussion is in part leaning on the product definition side.

At the query time, the plan won't be much different from the original plan. The only difference would be that file scan nodes in the optimized plan will access fewer files.


Typo in Chapter 3 - EAFP vs LBYL

LBYL = “Look before you leep”
Change "leep" to "leap".

Thank you @wstomv !!


Workflows: Update images, GitHub Actions, Kubernetes versions

One question is how many Kubernetes releases with EOL state we even want to support...

The required checks in this repositories branch protection settings must be adjusted with this PR.


invalid byte sequence in UTF-8

I installed fus using sudo gem install fus and tried running it in the root of my project but got the following error message:
$ fus find
/Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/lib/fus/swift_class.rb:26:in `match': invalid byte sequence in UTF-8 (ArgumentError)
	from /Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/lib/fus/swift_class.rb:26:in `match'
	from /Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/lib/fus/swift_class.rb:26:in `used_in_obj_c?'
	from /Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/lib/fus/finder.rb:69:in `block in used_in_obj_c?'
	from /Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/lib/fus/finder.rb:69:in `any?'
	from /Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/lib/fus/finder.rb:69:in `used_in_obj_c?'
	from /Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/lib/fus/finder.rb:48:in `block in unused_classes'
	from /Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/lib/fus/finder.rb:44:in `reject'
	from /Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/lib/fus/finder.rb:44:in `unused_classes'
	from /Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/lib/fus/finder.rb:21:in `unused_classnames'
	from /Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/bin/fus:14:in `find'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	from /Library/Ruby/Gems/2.3.0/gems/fus-0.0.3/bin/fus:34:in `<top (required)>'
	from /usr/local/bin/fus:22:in `load'
	from /usr/local/bin/fus:22:in `<main>'

Weird. You must have some character that the regex function doesn't like in one of your source files. I'd need to have more info about the specifics to fix

I have the same issue.

@oleghnidets if there is any way you can provide a test file that causes this failure that would be helpful.

While debugging, I noticed fus goes very deeply into the Pods folder analyzing if swift class is used in ObjC classes and failed on this file:

.../Pods/Postal/dependencies/build/macos/include/libetpan/clist.h

This file online: https://github.com/dinhviethoa/libetpan/blob/master/src/data-types/clist.h


Evaluate Marshmallow for data schema creation/validation

Docs: http://marshmallow.readthedocs.org/en/latest/why.html

Fixed by #10


Add license

Pick a license for the project and note in the footer.

https://github.com/devinbrady/openanc/commit/40630694421eff09f786f4ba91a51c6dd1b64244


The font size of the new project pop-up window is too large

It seems to be a problem of style priority. I fix this issue by disabling devFriendlySplitChunks feature and using splitChunks.


When switching to adjacent desktop the current window also moves to that desktop

When I use Left Ctrl + Left Win + (left or right arrow) to switch to the adjacent desktop (standard Windows functionality), sometimes the current window that I'm using moves to that desktop as well. If I click on the toolbar before trying to switch to the adjacent desktop, the window doesn't move. If I close SylphyHornPlus or stop and restart it, the problem goes away. It does eventually come back again.
The Move active window to adjacent desktop under Shortcut key 2 was set to Left Ctrl + Left Win + Left Alt + (left or right arrow) was set. Not sure if I did that or if it was set by default. It seems to sometimes override the Adjacent Desktop shortcut keys provided by Windows.
This doesn't happen all the time, but frequently enough that it is frustrating.
I'm running 4.0.0 beta.9 on Windows 10 Version 22H2 (OS Build 19045.2728).
Love the features of this app, aside from the couple of issues I've experienced. I hope development continues. Thank you!

Didn't see this before I logged this issue, but it might be related to (or the same as) Issue #27.


Add "-debugger" option to wide. Automatically open/lock/debug provided path

Hi I have been tinkering with wonkey again.  We actually adopted it 5+ years ago and put huge amounts of resources building it out to be a suitable development platform. Anyways...
Heres a pull request for a small feature. You can launch wide with:
wide -debugger "file.wx"
It will open, lock and start debugging the file. It only uses the built in actions/features of wide. There could potnetially be better implementation. For example it could be interesting to add a "debugger" layout to wide that limits the UI for a single debug session. So hide a ton of the projects, toolbars, etc.
The use case? Well I have been using webstorm to do monkey dev. I have a textmate bundle and just setup some run configs. For example here is a run config to debug a .wx file using wide:
open -n -a /Users/skn3/Documents/dev/wonkey/bin/macos/wide.app --args -debugger "/Users/skn3/Documents/dev/projects/wonkey/helloworld/helloworld2.wx"
The -n switch allowing for multiple copies of wide to be open!
Obviously it would be cleaner to creat a custom debug parser for intellij, but thats for another day if ever!

Hi, thanks for this PR. I'll look into this as soon as possible.
I notice that the Linux build is failing : SDL linking problem


Feature/2966/back button fix

I don't know why this fixes it, but it does.
dockstore/dockstore#2966

If possible would be interesting to find out exactly why this fixes it, maybe it will help with other search issues.


CMake flag improvements

Just my curious, since they are global compile options(for all targets), why not use add_compile_options()(once) instead of target_compile_options()(multiple times)

Good question. Before these changes, the CMake file was modifying the options globally for all C and C++ code. I'm not completely sure whether that was intentional and/or necessary for C, but from what I have seen it does seem to be necessary for C++ due to some of the sub-libraries that use C++. If we think it's also necessary to globally change the C options then I can make those options global as well. If not, I think using target_compile_options() is cleaner (even with the duplication) and follows the existing style more.

To be clear, I don't have strong opinions on this, so just let me know if you feel I should abandon this change.

@bnason-nf Not very sure what is the improvement of this PR? My first impression is that it is a little complex and makes the script not easy to understand - it introduces WAMR_COMPILE_FLAGS, WAMR_C_COMPILE_FLAGS, WAMR_COMPILE_FLAGS_STRING, etc. which makes me very confusing😊

Thanks for the feedback. Since this PR seems unpopular, I am going to close it. I will re-submit the commit for hidden visibility in a new PR.


Save properties

So as of right now, you have to refresh the page to see the post was saved, and unsaving the post isn't in.


C# FCM not getting delivered

Hello,
I am trying to send Push Notification using c# console app but its not delivering to the device
However, the app is receiving the Notification sent from the Firebase consoles
Here is my Console App
code
namespace FirebaseCloudMessaging.Example
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var defaultApp = FirebaseApp.Create(new AppOptions()
            {
                Credential = GoogleCredential.FromFile(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "key.json")),
            });
            Console.WriteLine(defaultApp.Name); // "[DEFAULT]"
            var message = new Message()
            {
                Data = new Dictionary<string, string>()
                {
                    ["FirstName"] = "John",
                    ["LastName"] = "Doe"
                },
                Notification = new Notification
                {
                    Title = "Message Title 1",
                    Body = "Message Body 1"
                },
                Topic = "adminalert"
            };
            var messaging = FirebaseMessaging.GetMessaging(defaultApp);
            //var result = await messaging.SendAsync(message).ConfigureAwait(false);
            var result = await messaging.SendAsync(message).ConfigureAwait(false);
            Console.WriteLine(result); //projects/myapp/messages/2492588335721724324
        }
    }
}

Please report notification delivery issues directly at https://firebase.google.com/support. You can indicate your project ID and the message ID returned by the SDK when reporting the issue, and the FCM team can help you figure out what the problem is.


Get-NinjaOneActivites -after and -before not passed to GET request.

When running Get-NinjaOneActivities, the cmdlet transforms the DateTime object to Unix Epoch timestamp but it does not append it to the GET request made to the NinjaRMM API endpoint.
Example
PS > $Date

Monday, 8 May 2023 9:45:42 am

PS > Get-NinjaOneActivities  -before $Date -verbose
VERBOSE: Converted 05/08/2023 09:45:42 to Unix Epoch timestamp<PHONE_NUMBER>
VERBOSE: Building parameters for Get-NinjaOneActivities. Use '-Debug' with '-Verbose' to see parameter values as they are built.
VERBOSE: Retrieving all device activities.
VERBOSE: Building [HttpQSCollection] for New-NinjaOneGETRequest
VERBOSE: Making a GET request to https://oc.ninjarmm.com:443/v2/activities
VERBOSE: HTTP/1.1 GET with 0-byte payload
VERBOSE: received -byte response of content type application/json

Fixed in 1.9.0 - thanks :-)


Fix TEMPORARY syntax

Index Issue: pingcap/tidb#11486
These are bad (incompatible) SQL cases:
create temporary table t (a int);
create temporary table if not exists t (a int);
create temporary table t1 (like test.t);

Claim this subtask based on the comments below this issue.

I'll fix it.


Use Babels Compact Mode

See babel/babel#3475.
Should we set {compact: true; comments: false} in our Babel usage?
Seems it could improve the speed a bit.

Just tried this out and tests pass—is there a way to measure performance?

I'll take a stab at it :)

@jfmengels is assigned!

Are we good to close based on the PR results?

Yup


Ligo alert script modifications

@caseyjlaw I've made some changes to the ligo receiving script. Can you please review them?
To post to slack, token and channel should be specified: https://github.com/ovrocaltech/ovro-alert/blob/a97d490a4b0b64d5725fadf7983b36be6daafea3/ligo-alert/gcn-buffer_trigger.py#L17-L18
Now the special case when the role of an alert is "test" and its type is "EarlyWarning" is treated as it was observation:
https://github.com/ovrocaltech/ovro-alert/blob/a97d490a4b0b64d5725fadf7983b36be6daafea3/ligo-alert/gcn-buffer_trigger.py#L78-L83
It is also possible to save the map for the corresponding alert in case it's provided:
https://github.com/ovrocaltech/ovro-alert/blob/a97d490a4b0b64d5725fadf7983b36be6daafea3/ligo-alert/gcn-buffer_trigger.py#LL99C6-L99C6

If you push a new commit to your branch, it should update this pull request.

@caseyjlaw, updated based on your suggestions. Can you take a look?

Hmm, it looks like this uses python 3.9 and I'm working with 3.6 on the server major. What version did you test with?

I used 3.10.9. I've just tried to downgrade to 3.6 and it seems like ligo.skymap and healpy are incompatible with it.

No problem. I've created a python3.9 environment and the ligo client is working.


Fix enumeration of Logitech G5 (2007)

Before this patch, ratbagd would abort initialization of the device because
no profiles are returned.
Once that was handled more gracefully, ratbagd would fail because it doesn't
read the DpiList from its own config file, as no ProfileType is specified.
Specifying G500 as the closest match.

This is a proposed fix for https://github.com/libratbag/libratbag/issues/966
I added an explanation on the issue.

I am really sorry for the delay!


fix some colours... kitty

so i tried your thing in kitty and when i ran htop i can't see many text can change colour or anything that can increase the readability of text   i tried catppuccin mocha...
(theme name is taribble..)

Can you send a screenshot?

Can you send a screenshot?

like it was not your fault it was mine... i'm using transparent window...sorry u can close this issue

Can you send a screenshot?

ok sorry it was on foot
here is the screenshot

here is my foot.nix
  programs.foot = {
    catppuccin.enable = true;
    enable = true;
    settings = {
      key-bindings = {
        scrollback-up-line = "Control+Shift+k";
        scrollback-down-line = "Control+Shift+j";
      };
      main = {
        term = "foot";
        font = "JetBrainsMono Nerd Font:size=9.5";
        dpi-aware = "yes";
      };
      scrollback = {
        lines = "5000";
        multiplier = "2";
        indicator-format = "percentage";
        indicator-position = "fixed";
      };
      # colors = {
      #   background = "282828";
      #   foreground = "ebdbb2";
      #   regular0 = "282828";
      #   regular1 = "cc241d";
      #   regular2 = "98971a";
      #   regular3 = "d79921";
      #   regular4 = "458588";
      #   regular5 = "b16286";
      #   regular6 = "689d6a";
      #   regular7 = "a89984";
      #   bright0 = "928374";
      #   bright1 = "fb4934";
      #   bright2 = "b8bb26";
      #   bright3 = "fabd2f";
      #   bright4 = "83a598";
      #   bright5 = "d3869b";
      #   bright6 = "8ec07c";
      #   bright7 = "ebdbb2";
      # };
    };
  };
}```

this is an issue with the foot port, as we only apply this theme. i would recommend following https://github.com/catppuccin/foot/issues/12 instead


chore: create a pull request after library generation

In this PR:

Create a pull request after library generation to verify the pull request can pass all checks.

For what I mentioned today, I meant one-time, manual pull request creation based on the generation script.
(This PR to continuously create/update is more than that.)

For what I mentioned today, I meant one-time, manual pull request creation based on the generation script.
(This PR to continuously create/update is more than that.)

I will remove the PR creation part after confirming the PR is fine.


Jbaker/fix npm ref

Alaska Airlines Pull Request
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes: #397
Summary:
npm is made a static part of the CDN bundle url
Type of change:
Please delete options that are not relevant.

[ ] New capability
[ ] Revision of an existing capability
[ ] Infrastructure change (automation, etc.)
[x] Docs

Checklist:

[x] My update follows the CONTRIBUTING guidelines of this project
[x] I have performed a self-review of my own update

By submitting this Pull Request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Pull Requests will be evaluated by their quality of update and whether it is consistent with the goals and values of this project. Any submission is to be considered a conversation between the submitter and the maintainers of this project and may require changes to your submission.
Thank you for your submission!
-- Auro Design System Team

:tada: This PR is included in version 4.0.5 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Add nbstripout to pre-commit

Implements @zmoon's suggestion to use nbstripout to delete kernel info.
This builds successfully locally, but asks you to choose a kernel when opening a notebook:

This builds successfully locally, but asks you to choose a kernel when opening a notebook:

I find that kind of annoying and possibly confusing to newcomers. Wish there was a way to simply overwrite kernelname="python3"...
Is it the case that following instructions here https://github.com/xarray-contrib/xarray-tutorial/blob/main/CONTRIBUTING.md#create-a-python-environment, you'll still end up with the different kernel names if nb_conda_kernels is installed?
also noting that this PR addresses https://github.com/xarray-contrib/xarray-tutorial/issues/78

AH I just checked.

If I run from my base environment with nb-conda-kernels installed it asks to choose a kernel.
If I run from xarray-tutorial it just opens up the "default" kernel. I assume this will happen on binder too, but we can triple-check after merging.

Seems like we could recommend tutorial users open jupyter lab or jupyter notebook in the xarray-tutorial environment and then things will work well?

I assume this will happen on binder too, but we can triple-check after merging.

You can just launch binder pointing to this nbstripout branch: https://mybinder.org/v2/gh/xarray-contrib/xarray-tutorial/nbstripout?urlpath=lab

Sweet got the default kernel on binder.


Chart dependency conditions vs enabled

From the docs here:
dependencies: # A list of the chart requirements (optional)
  - name: The name of the chart (nginx)
    condition: (optional) A yaml path that resolves to a boolean, used for enabling/disabling charts (e.g. subchart1.enabled )
    enabled: (optional) Enabled bool determines if chart should be loaded

What is the difference between conditions and enabled in chart dependencies ?
I guess i understand conditions but what is the enabled purpose ?

@eddycharly from my opinion, the difference at the moment is unclear. As in, enabled doesn't actually provide any functionality (see #7908), and though I created a PR to enable enabled #8138, I'm not sure if it instead should be removed from the documentation to avoid further confusion

@jeff-knurek thanks !
I can't say i understand a lot better the purpose of enabled but i feel less alone :D


Autoscaling not happening as per scale-up policy

Report
Hi Team,
Facing this below issue:
Keda version: 2.7.1
EKS version: 1.22
Scaleup policies:
scaleup:
    policies:
      - periodseconds: 15
        type: Pods
        value: 2
      - periodseconds: 15
        type: Percent
        value: 10
    selectpolicy: Max
    stabilizationWindowSeconds: 60
 triggers:
    - metadata:
        value: "65"
      metricType: Utilization
      type: cpu

Issue:
When the current no of replicas is 2 and threshold is breaching cpu utilization of 65% scaling is being triggered. But scaling happens to bring up only 1 pod instead of 2 as per the scaleup policy. The above mentioned behaviour of scaleup is happening.
Expected Behavior
As per the policy defined, it should bring up 2 static pods as the threshold is breached. If the current repliacas reach more than 21 then 10% of it i.e 2.1=~3 pods should be scaled.
Actual Behavior
Only one pod is being scaled irrespective of static 2 pods defined.
Steps to Reproduce the Problem

Logs from KEDA operator
example

KEDA Version
2.7.1
Kubernetes Version
< 1.23
Platform
Amazon Web Services
Scaler Details
No response
Anything else?
No response

Can you elaborate on what this "scaleup" policy is? Because KEDA only handles horizontal scaling

Can you elaborate on what this "scaleup" policy is? Because KEDA only handles horizontal scaling

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#scaling-policies

That's what I figured, but sample above was not clear

This is totally out from KEDA scope because KEDA is passing those values to the HPA. You should open the issue in the upstream


Better documentation

Context
Can't really find a nice place to document the form feature, probably need to setup a doc site to explain better?
Also maybe somewhere that can present the changelog as well
Tools

Site framework: Vitepress / docusaurus / Gatsby
Demo: https://github.com/codesandbox/sandpack / https://docusaurus.io/docs/markdown-features/code-blocks#interactive-code-editor
Font: https://fonts.google.com/specimen/Lexend

Will document form feature yet since feel like it's not ready, and feel like state logic can be enhanced


img2vid can we use multi GPUs to speed up inference?

Inference takes about 30 minutes for img2vid. Wondering is there a way to leverage multiple GPUs to improve speed? I have 8x 100 GPUs
Currently running using diffusers pipeline
from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained("local/path/stable-video-diffusion-img2vid-xt-1-1")
pipeline(image)

same question

same question

use CUDA_VISIBLE_DEVICES=3,4 python xxx, just use gpu 3


⚠ sandbox-api.piste.gouv.fr healthcheck has degraded performance

In 01bccb3, sandbox-api.piste.gouv.fr healthcheck (https://sandbox-api.piste.gouv.fr/cassation/judilibre/v1.0/healthcheck) experienced degraded performance:

HTTP code: 200
Response time: 4750 ms

Resolved: sandbox-api.piste.gouv.fr healthcheck performance has improved in a7b49cc.


VideoJS on Firefox won't play videos recorded with VideoJS-Record on Chrome

Description
VideoJS on Firefox won't play videos recorded with VideoJS-Record on Chrome
Steps to reproduce
Record a video with VideoJS-record on Chrome and try to play it on VideoJS on FF
Results
Expected
Video should render correctly
Actual
VideoJS displays error
Error output
The media could not be loaded, either because the server or network failed or because the format is not supported.
Additional Information
I have tried using the mimetypes specified here: https://collab-project.github.io/videojs-record/#/options
No success
versions
videojs
7.8.2
browsers
Firefox 77.01
OSes
MacOS

Looks related to this: https://github.com/muaz-khan/RecordRTC/pull/657
But you didn't specify what config you're using, and since webm is the default, that fix in recordrtc might help here as well.

duplicate of #392

duplicate of #392


HDDS-3121. Fix TestSCMPipelineBytesWrittenMetrics.

What changes were proposed in this pull request?
Fix flaky test TestSCMPipelineBytesWrittenMetrics.
One thing I observed is to populate this metric, we are iterating entire containerset on datanode and matching with pipelineID and then sending this as part of PipelineReport. Not sure if this is okay, as container count increases, processing taking more time.
In this Jira only fixed test.
  private long calculatePipelineBytesWritten(HddsProtos.PipelineID pipelineID) {
    long bytesWritten = 0;
    Iterator<org.apache.hadoop.ozone.container.common.interfaces.Container<?>>
        containerIt = containerController.getContainers();
    while(containerIt.hasNext()) {
      ContainerData containerData = containerIt.next().getContainerData();
      if (containerData.getOriginPipelineId()
          .compareTo(pipelineID.getId()) == 0) {
        bytesWritten += containerData.getWriteBytes();
      }
    }
    return bytesWritten;
  }

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3121
How was this patch tested?
Fixed UT.

LGTM +1, pending CI


Submit for Search field with floating label + input-group icon, form validation

I did the following fiddle cause I feel it's missing in Daemonite
https://jsfiddle.net/djibe89/13tzf7L4/
I also did a fiddle for using form validation with your Daemonite, it works perfectly with Parsley
https://jsfiddle.net/djibe89/9yoow8z3/

fb8c33b fixed a couple of issues with having .floating-label or .textfield-boxes in .input-group.
Working on the icon part now, but my idea is to have icons come after form controls in HTML and then move them in front using order so we can change the colour of the icons without JavaScript.
May need to create a new class though (maybe .input-group-icon), since this isn't a default Bootstrap behaviour, I guess it would be ok then.

You rock. As usual

Thank you so much for this perfect release


SMPTConfig

Is there a better solution to solving the SMPT configuration for account activation email. Right now with everything setup and Less secure apps enabled on the other end, it throws this error confirm: Failed to send e-mail: EOF

Since there is no reconnect functionality I applied the patch in this PR to mailer. It establishes the TCP connection once it sends an email.
Maybe I (or someone else) can supply re-connection handling to gomail when the net/stmp NOOP lands in Go 1.10.


Data replication from TiDB to Elasticsearch

What problem does this PR solve?
What is changed and how it works?
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation

Release notes:

NONE

Merge the data replication from tidb to elasticsearch to main branch


Question about Group Sync

I've set up Group Sync according to the wiki. Role ID is set up and the StaffCP shows integration is enabled and the bot successfully linked, but this happens:

User123 joins our Discord for first time and has @everyone role.
User123 signs up on NamelessHosting site and gets "Member" post-validation group.
User123 returns to Discord and !verify is successful, but is not promoted to Member role.

Am I mistaken that User123 should be promoted to Member after verification (wiki says "roles are synced in both directions")? Or am I having to wait for #102 ? API rules:

Maybe the website does not do role sync with post validation roles, @Aberdeener ?

Confirmed. Web -> Discord works but the opposite is broken.

Wasn't the original issue about web>discord not working?

Issue is that once a post-validated NamelessUser completes !verify their Discord role is not updated to match their Nameless group. It's now my understanding this is supposed to occur at https://github.com/NamelessMC/Nameless-Link/blob/76ca5b7dd55cf9767d1d2e39313f232ee2fe9269/src/main/java/com/namelessmc/bot/listeners/DiscordRoleListener.java#L153
as part of the #sendRolesToWebsite method, which I guess makes the flow Discord > web > Discord ? Honestly I'm a bit confused as I've never created a bot nor dealt with this sort of integration.

https://github.com/NamelessMC/Nameless/issues/2187


Trembling on Chrome

Hello, thank you for this shared work. I would like to use Scrollify but I notice after some tests that on Chrome, when the content of a section is longer than the viewport, it causes problems.  Not always but very often and it seems that the more we use it the more problems of tremors appear. I am using Chrome Version 65.0.3325.181 (Official Build) (64-bit) on a Imac. We can see the problem on the demo page (https://projects.lukehaas.me/scrollify/#configuration). It's really on overflow sections. I tried to restart Chrome but the problem persists, including with just a single open window.

I have the exact same problem on Chrome  66.0.3359.139 (Official Build) (64-bit)

Unfortunately, I have the same trouble.
The problem is also observed when moving down from the long section to the normal one.

I have the same issue, I am waiting for the fix.

Waiting for a fix on this as well.
It doesn't happen when you navigate using the arrow keys, only when I scroll using the touch pad.
Example:
https://vitruvi.ca/pages/ad-landing-page

+1

+1

+1


ROX-23625: Replace assertion about page address in clipboard

Description
Problems
Intermittent failures of integration tests during and after PatternFly 5 upgrade in #10401
Exception Management - Approved Deferrals Table should be able to navigate to the Request Details page by clicking on the request name

Timed out retrying after 4000ms: Expected to find element: div[role="tooltip"], but never found it.

Exception Management Request Details Page "before each" hook for "should be able to deny a request if approval permissions are granted": "before each" hook

Timed out retrying after 4000ms: Expected to find element: div[role="tooltip"], but never found it.

Because this error occurred during a before each hook we are skipping the remaining tests in the current suite

Analysis
Assertions about Copied tooltip:

failure in tests: https://github.com/stackrox/stackrox/blob/master/ui/apps/platform/cypress/integration/vulnerabilities/exceptionManagement/ExceptionManagement.helpers.ts#L72
failure in tests: https://github.com/stackrox/stackrox/blob/master/ui/apps/platform/cypress/integration/vulnerabilities/exceptionManagement/ExceptionManagement.helpers.ts#L104
failure in beforeEach blocks: https://github.com/stackrox/stackrox/blob/master/ui/apps/platform/cypress/integration/vulnerabilities/exceptionManagement/requestDetails.test.ts#L58

The code in beforeEach block is same as deferAndVisitRequestDetails heler function.
Thank you David Vail for observation that tooltip might not appear in a minority of cases since PatternFly 5 upgrade.
Solution
Gleb Bahmutov describes an alternative to stub the clipboard.write method call:
https://glebbahmutov.com/blog/spy-on-clipboard-copy/

Replace assertion about Copied tooltip:

Use id from response as source of truth.
Replace actual write with stub that asserts expected content.

Replace beforeEach block, in which an intermittent failure affects not just one test, but all tests in file.

Call deferAndVisitRequestDetails function at the beginning of each test that needs it as initialization.

Changes

cypress/helpers/request.js

Make returns type more specific in JSDoc comment for waitForResponses function.
Add cy.wrap for [] so it is chainable, which solves special case of never[] type which causes unnecessary type problem for callers.

Move deferAndVisitRequestDetails function from beforeEach block into tests:

vulnerabilities/exceptionManagement/approveRequestFlow.test.ts
vulnerabilities/exceptionManagement/denyRequestFlow.test.ts
vulnerabilities/exceptionManagement/requestDetails.test.ts
vulnerabilities/exceptionManagement/updateRequestFlow.test.ts
Also add PATCH arg to fillAndSubmitExceptionForm function call.

vulnerabilities/exceptionManagement/ExceptionManagement.helpers.ts

Factor out assertClipboardWriteAndVisitRequestPage function, which given id argument from response, asserts clipboard write via stub.
Get id from response in deferAndVisitRequestDetails and markFalsePositiveAndVisitRequestDetails functions.

vulnerabilities/workloadCves/WorkloadCves.helpers.js

Wait for requests in cancelAllCveExceptions function.
Return interception from fillAndSubmitExceptionForm function.
Assert header in verifyExceptionConfirmationDetails function.

vulnerabilities/workloadCves/WorkloadCves.selectors.js

Encapsulate selectors used only in helper functions.

Residue
Refactor to reduce relative imports between sibling subfolders.

Add cypress/integration/vulnerabilities/vulnerabilities.helpers.js file.
Encapsulate selectors which tests do not need for assertions.

Checklist

[ ] Investigated and inspected CI test results
[x] Edited integration tests

Testing Performed
Manual testing

Visit /main/vulnerabilities/workload-cves and defer a vulnerability via row action

POST /v2/vulnerability-exceptions/deferral has response which includes id and name properties.

Click copy button and see role="tooltip" attribute of transient div element at end of document body.

Integration testing
Before yarn cypress-open in ui/apps/platform folder:
export CYPRESS_ROX_VULN_MGMT_WORKLOAD_CVES=true
export CYPRESS_ROX_VULN_MGMT_UNIFIED_CVE_DEFERRAL=true

yarn cypress-open in ui/apps/platform

vulnerabilities/exceptionManagement/approvedDeferralsTable.test.ts
vulnerabilities/exceptionManagement/approvedFalsePositivesTable.test.ts
vulnerabilities/exceptionManagement/approveRequestFlow.test.ts
vulnerabilities/exceptionManagement/cancelRequestFlow.test.ts
vulnerabilities/exceptionManagement/deniedRequestsTable.test.ts
vulnerabilities/exceptionManagement/denyRequestFlow.test.ts
vulnerabilities/exceptionManagement/general.test.ts 1
vulnerabilities/exceptionManagement/pendingRequestsTable.test.ts 5
vulnerabilities/exceptionManagement/requestDetails.test.ts
vulnerabilities/exceptionManagement/updateRequestFlow.test.ts
vulnerabilities/workloadCves/cveListExceptionFlow.test.js
vulnerabilities/workloadCves/deploymentSingle.test.js
vulnerabilities/workloadCves/imageCveListExceptionFlow.test.js
vulnerabilities/workloadCves/imageCveSingle.test.js
vulnerabilities/workloadCves/imageSingle.test.js
vulnerabilities/workloadCves/workloadCveOverviewPage.test.js

gke-ui-e2e-tests failure
Workload CVE overview page tests should satisfy initial page load defaults

Timed out retrying after 4000ms: Expected to find element: h1:contains("Workload CVEs"), but never found it.

Screenshot shows still waiting for generic requests.
Because DOM assertion waits for 4 seconds, but request interceptions wait for 10 seconds, test would become more reliable if helper function intercepts and waits getImageCVEList for GraphQL query:
https://github.com/stackrox/stackrox/blob/master/ui/apps/platform/cypress/integration/vulnerabilities/workloadCves/WorkloadCves.helpers.js#L23-L38

ocp-4-15-ui-e2e tests failures

Timed out retrying after 4000ms: Expected to find element: [data-testid="tooltip-title"], but never found it.

Risk Event Timeline for Deployment Event Details

shows the process activity event details for a process with no uid change
shows the process in baseline activity event details
shows the container restart event details
shows the container termination event details

Risk Event Timeline for Pod Event Details

shows the container restart event details

Investigating how to make these tests more reliable after PatternFly 5 upgrade


uninstallation of Lens from  ubuntu 20.04

on this website: https://docs.k8slens.dev/getting-started/install-lens/   why is there no uninstallation instructions because i can't uninstall lens

If you have installed lens with apt, try to use the following command:
sudo apt-get remove lens


Show a CC Search → Openverse notice

Fixes

Fixes #194 by @zackkrida
Description

This PR creates the NoticeBar component for showing notices and uses it to show the MigrationNotice component on the top of the page.
Technical details

The notice is shown on first render and disappears after you navigate to a different link. This means that the code for this is split across a plugin (plugins/migration-notice.js) and a middleware (middleware/middleware.js).
Screenshots
URL: http://localhost:8443/?referrer=creativecommons.org

URL: http://localhost:8443/search/?q=cat&referrer=creativecommons.org

Checklist

[x] My pull request has a descriptive title (not a vague title like Update index.md).
[x] My pull request targets the default branch of the repository (main or master).
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[ ] I added or updated tests for the changes I made (if applicable).
[x] I added or updated documentation (if applicable).
[x] I tried running the project locally and verified that there are no visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

Resolved conflicts with master

@dhruvkb LGTM!


[next] type arguments for views not working as expected

https://github.com/sh-dave/eskimo-issues
I put up a small repo in case there are more issues, so you can easily reproduce. :stuck_out_tongue:
Just recursive clone and drop the issue folder into KS.
https://github.com/sh-dave/eskimo-issues/tree/master/view-arguments
../s/S.hx:6: characters 18-26 : Type not found : c.C1.
If you comment out the line, you see that this works for haxe's Map class.
It does also work when i put each component into it's own file and import them.
Edit: Also doesn't work when i try to import the actual class(es) by name:
import c.C.C1;

Good catch!
Haven't tested these cases, but I'll have a go at getting this to work as soon as possible.

You can just do:
import c.C;

and you should be able to use all contained types! Let me know if it works on your end.

:+1:


feat(conf): Mount source and resource ConfigMaps as subPaths instead of volumes

Fixes #1998.
Release Note
feat(conf): Mount source and resource ConfigMaps as subPaths instead of volumes

@astefanutti mind opening an issue on camel-k-runtime ? don't know if it needs to be adapted

@lburgazzoli I've created https://github.com/apache/camel-k-runtime/issues/614.
CI passes for 2610caf3a4c3b15afb016d79417ba2f41c03f7d1. I'm now rolling other commits for the other kind of resources.


Update fzf-tmux

In my case I have defined  alias rm=~/bin/safe-rm which move it into trash, fzf-tmux generate temporary file which can safe be deleted immediate. so add hook FZF_CLEANUP_CMD which default is  /bin/rm

Notice \ prefix of \rm. It makes bash ignore any aliases, so I think you're confused somehow.

thinks @junegunn , I found the issue  that  there is an other rm cli  at  $PATH  before /bin/rm


Wepback Path issue on Windows

@chaelli's description copied from https://github.com/vigetlabs/gulp-starter/pull/257#issuecomment-202039278

since this update I get a 404 on the http://localhost:3000/javascripts/app.js - there's also no "javascripts" folder in the "public" folder.
Any idea what could be wrong? the paths seem to be correct...

Update:

ouch.. found the issue.. Windows again ;)
just add ".replace(//g, '/')" to the following line in webpack-multi-config.js to fix it:
var publicPath = path.join(config.tasks.js.dest, '/')

Good ol' Windows paths. I thought path.join should normalize that. Does this also work?
var publicPath = path.normalize(config.tasks.js.dest +  '/')

I forget why I needed that trailing slash... I'm sure I added it for a reason at some point. But I'll try it again without it.

nope, normalize doesn't work - I guess normalize replaces all / and \ with the appropriate character for the current OS. So for windows it will replace all with \ => but as this "path" is not used as "path" but as "url", it doesn't work.

ooooh. Sounds like I need url.resolve then: https://nodejs.org/api/url.html#url_url_resolve_from_to

Ah. Right, we're converting a windows path to a URL. Your suggestion sounds like it should be good enough. Will add.


fix(drawer): missing elevation shadow

Fixes the elevation box shadow not being applied to drawers.
Fixes #8386.

fixes #6528

Caretaker note: potential screenshot diffs

Halo,
Iam using multiple themes and the elevation gone
<mat-toolbar color="primary" class="mat-elevation-z6"></mat-toolbar>

and fixed with adding z-index
<mat-toolbar color="primary" class="mat-elevation-z6" style="z-index: 6"></mat-toolbar>

is this pr fix that issue?


Windows: Microsoft compilers don't support VLA

Compilers 14.0 and below don't support Variable Length arrays, so the argon2_nif.c won't compile without modification on Windows, notably the argon2_hash_nif() function, due to things like
char hash[hashlen * 2 + 1];
char encoded[encodedlen];

Got around it by doing:
char *hash = malloc(hashlen * 2 + 1);
char *encoded = malloc(encodedlen);

and of course freeing them before returning, including the end of the function.

What version of Visual Studio is compiler 14.0?

I should have a fix ready later today.

Version 1.2.2 contains a fix.

Tried 1.2.2, and it still doesn't build.  encodedLen and hashspace are still not constants, so the compiler sees them as something that has to be determined at runtime, so they are still in VLA territory.
Going the malloc/alloc route lets you still treat them as arrays in the code -- they just have to be freed instead of living on the stack.
Compiler 14.0 is Visual Studio 2015.
Here's the output:
Microsoft (R) Program Maintenance Utility Version 14.00.24210.0
Copyright (C) Microsoft Corporation.  All rights reserved.

	erl -eval "io:format(\"~s~n\", [lists:concat([\"ERTS_INCLUDE_PATH=\", code:root_dir(), \"/erts-\", erlang:system_info(version), \"/include\"])])" -s init stop -noshell > Makefile.auto.win
	nmake /                   /F Makefile.win priv\argon2_nif.dll

Microsoft (R) Program Maintenance Utility Version 14.00.24210.0
Copyright (C) Microsoft Corporation.  All rights reserved.

	if NOT EXIST "priv" mkdir "priv"
	cl /O2 /EHsc /I"argon2\include" /I"argon2\src" /I"c:/Program Files/erl9.0/erts-9.0/include" /LD /MD /Fepriv\argon2_nif.dll argon2\src\argon2.c argon2\src\core.c argon2\src\blake2\blake2b.c argon2\src\thread.c argon2\src\encoding.c argon2\src\ref.c c_src\argon2_nif.c
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

argon2.c
core.c
blake2b.c
thread.c
encoding.c
ref.c
argon2_nif.c
c_src\argon2_nif.c(55): warning C4133: 'function': incompatible types - from 'argon2_type *' to 'unsigned int *'
c_src\argon2_nif.c(61): error C2057: expected constant expression
c_src\argon2_nif.c(61): error C2466: cannot allocate an array of constant size 0
c_src\argon2_nif.c(61): error C2133: 'hash': unknown size
c_src\argon2_nif.c(62): error C2057: expected constant expression
c_src\argon2_nif.c(62): error C2466: cannot allocate an array of constant size 0
c_src\argon2_nif.c(62): error C2133: 'encoded': unknown size
c_src\argon2_nif.c(141): warning C4133: 'function': incompatible types - from 'argon2_type *' to 'unsigned int *'
c_src\argon2_nif.c(171): warning C4133: 'function': incompatible types - from 'argon2_type *' to 'unsigned int *'
Generating Code...
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\cl.EXE"' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\nmake.EXE"' : return code '0x2'
Stop.

Ok. I suppose we have to the malloc route.
I've just updated the master branch, removing the hashspace references (as that 'fix' didn't work). Do you want to send me a PR?

I'll send something ASAP

Sent pull request

Thanks for all your help.
A quick question related to the dirty scheduler support -- is it hashing passwords correctly? No runtime errors?

So far, the password hashing/verification has worked for me, but Im not currently doing any heavy load when verifying logins,


⚠ Bofrost.lu has degraded performance

In b8a85b0, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2004 ms

Resolved: Bofrost.lu performance has improved in cc5d73b after 16 minutes.


Support for server->client batch requests?

JSON-RPC version 2.0 defines batch requests, where a JSON-RPC client can send multiple requests (including notifications) in one transport message: https://www.jsonrpc.org/specification#batch
My LSP server sends batch requests, but vscode-lspclient seems to ignore these batch requests. From my interpretation of the LSP specification and the JSON-RPC specification, vscode-lspclient's behavior is non-conforming.
Is vscode-lspclient supposed to be conforming to the JSON-RPC version 2.0 specification? Or does vscode-lspclient only support JSON-RPC version 1.0 (which does not have batch requests)?

@strager there is an item in the protocol itself on how batch requests should be supported. See https://github.com/microsoft/language-server-protocol/issues/988
When we did a initial investigation we noticed that quite some json rpc libraries don't have support for batch requests. This is why we stayed away from it in the LSP protocol and hence I never implemented it in the vscode json-rpc library.
I do agree that request batching is very useful and I would like to find a solution for it for LSP.

And to answer your questions: the library is currently 2.0 - batch requests. Adding it on the library level is fine for me, but we would still need to decide how to surface this on the protocol level to not break things


Added support for creating/using user's own storage account

Added ARM utils for template deployment and better performance on list/show/delete commands. Improved output formatting.

/cc @SajayAntony @sivagms @DavidObando

:shipit:


[WIP] EMAILPLT-193: Initial Travis CI configuration

@apetro could you switch on Travis CI for this Repo?

Example Build Output: https://travis-ci.org/ChristianMurphy/email-preview/builds/199411164


fix the return of duplicate plants

Had to change the fetchType to LAZY on the non-owning side of the relations, so that the returned lists get properly populated and no duplicates get returned.
The issue of duplicate plants occured both in the plantsOwned and the plantsCaredFor list
I already had this issue with the plantsContained list for spaces and changing the fetchtype on one side resolved the issue

All tests pass
Did we not enable eager loading because of some behaviour that ended up not showing child elements :thinking:
IMO: We can merge but need to keep this in mind. @M-Sigg


Added brakeman, bundler-audit, and gemsurance

Added brakeman, bundler-audit, and gemsurance

I find the brakeman and bundler-audit gems to be valuable for flagging possible security vulnerabilities.  I find gemsurance to be essential for flagging the gems that need to be updated because they're outdated or have security vulnerabilities.

These gems are indeed great. However, adding them to the Gemfile does have a drawback - a gem version can easily become outdated and in my experience, it takes a while for developers to update them. That means that the security advices that were added to the gem are not applied.
I found that it's best to make a simple script that install the latest brakeman/bundler-audit every time and run checks. Like this:
#!/usr/bin/env bash

gem install brakeman --no-ri --no-rdoc
brakeman -z --table-width 300


Shutting down after listenHTTP

listenTCP and listenUDP allow for e.g. shutting down once started.
However listenHTTP does not return these connections, instead it just discards them.
Use case:
Linux daemon + reloading of config. To allow for changing of ports/ip's.

Yes, in the HTTP server the contexts are stored in g_contexts and have nothing to allow removing one.
The listenTCP return type TCPListener has a stop method but it's unused so far. I think we would need an HTTPContexts object to manage the listeners but I can't think of any use cases for changing the ip/port during runtime.

I wasn't meaning changing the ip/port on an already listening interface.
I was meaning the use case of e.g. a web server like Apache httpd.
So stop them all and create new based upon config.

I wasn't meaning changing the ip/port on an already listening interface.
I was meaning the use case of e.g. a web server like Apache httpd.
So stop them all and create new based upon config.

Yes I've wanted to have that for a while now, I know what you mean. The only issue I see is the necessity to have r/w locks or maybe an atomic fence on every request because g_contexts now became a mutable shared array that can change at any time.

Would it be possible to refactor the API out, so that the control of the connection is by the end user?
e.g. they create the connection, on receive of data forward it to vibe.http.server function (possibly with some context involved) and with a hook get called with the request?

Would it be possible to refactor the API out, so that the control of the connection is by the end user?
e.g. they create the connection, on receive of data forward it to vibe.http.server function (possibly with some context involved) and with a hook get called with the request?

Yes I've also imagined something like that. It would only require to pass the connectTCP and listenTCP as delegates to requestHTTP and listenHTTP respectively. This way the HTTP listener actually sends its void delegate(TCPConnection) handler over so that it can be used internally with your own streams.
I think the HTTPServerSettings would be a nice place to have this delegate, I was motivated mostly by a unit testing interface but your use case seems genuine too, not sure what Sönke would say about it

Yeah, that works. All I need is some way to stop them. It's a blocker for daemon support.
There is no way I could release the web server, even if I get e.g. PHP support if the damn demonisation doesn't work.
Something that should be so simple... Yeah, wouldn't look good.

Yeah, that works. All I need is some way to stop them. It's a blocker for daemon support.

If you handle the listenTCP through a delegate you acquire the means to stop the listeners. The declaration is:
TCPListener listenTCP(ushort port, void delegate(TCPConnection conn) conn_callback, string address, TCPListenOptions options);
The TCPListener is defined as follows:
interface TCPListener {
	/// Stops listening and closes the socket.
	void stopListening();
}

So this should allow some more control over the TCP layer I guess

Thanks, I was thinking about quick workarounds to allow me to change the port/restart the server at runtime, before this feature came up =)


v2.7.5 isn't available in cocoapods repo

My private pod depends on your library: s.dependency 'UIDeviceComplete', '~> 2.7'
When I install pods, only v2.7.3 is fetched:
❯ bundle exec pod install --repo-update
Updating local specs repositories
Analyzing dependencies
Downloading dependencies
Installing UIDeviceComplete (2.7.3)
...

Temporary workaround is to point to the repo directly in Podfile:
pod 'UIDeviceComplete', :git => 'https://github.com/Nirma/UIDeviceComplete.git', :commit => '7fa25cf6c648a0c5fa4bb8c182fb91c7ded3e861'

I am able to query from cocoa pods and confirming that v2.7.5 is available. Maybe try pod update?

thanks, now it's ok, strange that it gave me wrong version initially. Sorry for the noise.


Make sure I can unpack aliases

This is for testing alias unpacking

[APPROVALNOTIFIER] This PR is NOT APPROVED
Needs approval from an approver in each of these OWNERS Files:

pkg/OWNERS

We suggest the following people:
cc @smarterclayton
You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

sorry for the spam clayton...please ignore

[APPROVALNOTIFIER] This PR is NOT APPROVED
Needs approval from an approver in each of these OWNERS Files:

pkg/kubectl/OWNERS

We suggest the following people:
cc sig-cli-maintainers
You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is NOT APPROVED
Needs approval from an approver in each of these OWNERS Files:

pkg/kubectl/OWNERS

We suggest the following people:
cc sig-cli-maintainers
You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is NOT APPROVED
Needs approval from an approver in each of these OWNERS Files:

pkg/kubectl/OWNERS

We suggest the following people:
cc sig-cli-maintainers
You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is NOT APPROVED
Needs approval from an approver in each of these OWNERS Files:

pkg/kubectl/OWNERS

We suggest the following people:
cc sig-cli-maintainers
You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is NOT APPROVED
Needs approval from an approver in each of these OWNERS Files:

pkg/kubectl/OWNERS

We suggest the following people:
cc adohe
You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
The following people have approved this PR: grodrigues3
Needs approval from an approver in each of these OWNERS Files:

pkg/kubectl/OWNERS [grodrigues3]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment


Disable bm_trickle from running under TSAN

Resolves #13035
Feel free to nix this, but I think it would be safe to disable under TSAN. Some scenarios take 30 min to run. My intuition is that running under all the sans sans tsan plus running bm_trickle will give a good enough testing signal.

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

Friendly ping. This PR is important to limit test job duration spikes.

Just disables a test. Passes sanity, so merging early


Asset: make ThirdParty properties public, remove methods

To make the Asset class a full DTO, change the methods to properties. This will help simplify the C# Foundation layer & memory management wrapping.

@MAG-AlessioRegalbuto @mag-michelleshyr FYI

I'm not sure the originating cause of (what I think is) the tests build error:
 C:\teamcity\uk-hw-win-01-01\work\d578f8b65d37e27d\ThirdParty\googletest\include\gtest\gtest.h(1746,6): error C2678: binary '==': no operator found which takes a left-hand operand of type 'const T1' (or there is no acceptable conversion) [C:\teamcity\uk-hw-win-01-01\work\d578f8b65d37e27d\MultiplayerTestRunner\MultiplayerTestRunner.vcxproj]
15:37:38             with
15:37:38             [
15:37:38                 T1=std::pair<std::string,std::string>
15:37:38             ]

Aye, this isn't you, it's a side effect of another change and a mismatched CI config, we're working on getting it resolved.

@mag-richardzirbes I took the liberty of rerunning your test jobs after we'd commited a fix, sorry for the bother.


Feature request: load/cache several models in RAM

Extracted from this conversation.
When you change a model (checkpoint) it must be loaded from the drive every time. If there was capability to pre-load multiple models, it'd be possible to switch between them much faster reducing delay between generations.
UX suggestions:

in the model selector list add an indicator next to each item to quickly see if it's in RAM or must be loaded
add a button/link to call a list of all models where you can load/unload one by one or in bulk

NB: I haven't used Swarm yet, so the above is not related to the current or planned UI.

Oh yeah. been playing with a1111 then comfy and now i'm here, yes, gen 4 nvme's load models decently fast and I am sure that gen 5 nvme's are even better. But you know what's better than that? Yeah the 128GB of DDR4 I have in my system 🗡
Two thumbs up 👍👍 for this request.

There's a ComfyUI PR today aimed at this https://github.com/comfyanonymous/ComfyUI/pull/3545


request.get_json() returns (Ellipsis, Ellipsis)

I'm trying to follow this article on building a simple blockchain. But right now I'm stuck at step 3: "Interacting with our Blockchain"
I tried accessing the http://localhost:5000/transactions/new using cURL with the following commands:
curl -X POST -H "Content-Type: application/json" -d '{"sender": "d4ee26eee15148ee92c6cd394edd974e", "recipient": "a4ee26eee15148ee92c6cd394edd974e", "amount": 5}' "http://localhost:5000/transactions/new"
curl --header "Content-Type: application/json"   --request POST --data '{"sender": "f3ba8", "recipient": "b5d29", "amount": 5}' http://localhost:5000/transactions/new
Both return "Missing values" and if i print out the values variable it says "(Ellipsis, Ellipsis)".
I already tried to use postman for the request, but still got the same error. I also tried to copy & paste the finished code from this repo to check if I made a mistake while following the steps in the article.
It seams to as if there is something wrong with my cURL command but i can't figure out what.

I'm having the same problem! Any updates? did you manage to solve it?

I was able to solve the problem.
the problem is in the Flask version of the original article, try to reinstall it and it should work!

I had the exact same issue, the updating version of Flask did it for me too. Thank for the fix.

I had the exact same issue.  Thank you! I had flask 0.12.2 and updated to the latest 2.2.2 and even the deprecated request.json worked like a charm!


Use ipykernel Patched for Python 3

Still Needs:

[x] https://github.com/ipython/ipykernel/pull/243 (included logic to patch rather than wait for PR)

I realized that I may need to add more to this.  There are probably some dependencies in here that will need to be changed.  Can I get back at you later?

@jpolchlo Would you mind taking another look at this?  I would like to merge this as-is and perhaps make a follow up PR if/when ipython/ipykernel#243 is merged

Yeah, I think we grew all of these scripts back when the build was more complicated (toree and friends).  There is very little going on now, so it seems better to just put the entire build in one easy-to-see place.


'scroll-behavior: smooth' causes issues with full page screenshots

One site I have recently been testing uses the following CSS property: html { scroll-behavior: auto; }. This causes the page to smoothly scroll to an anchor, or the top of the page. However, when a full page screenshot is being captured, the page is still smoothly scrolling up, causing issues.
The screenshot captured repeatedly shows that is displayed at the viewport height.
I have solved this on my particular project buy using the following: cy.get('html').invoke('css', 'scroll-behavior', 'auto'). However, this may not work for all sites.
The plugin should probably do one of the following:

Ensure that the page is not scrolling when a full page screenshot is being captured
Find a way to jump to the top of the page instantly

Thanks team,
Edward.

Hi. This issue is more related to Cypress itself and how cypress.screenshot command takes screenshots. Like you said, whatever we are trying to do here is just a hacky workaround, and we don't feel comfortable proceeding with implementing this particular feature.
However, we will update our docs to give other people a heads up about this issue. Thank you for your suggestion. 🙌


Update to twilight 0.14

Thanks for updating twilight's version! This PR looks good to me, closing and merging. I'll change the version and upload to crates.io


Debug always logs the same amount of memory used

What steps will reproduce the problem?
Clean install of yiisoft/yii-demo or yiisoft/app-api and enable debug in params
What is the expected result?
Actual memory usage to be recorded
What do you get instead?
Always the same memory amount is recorded
Additional info

Q
A

Version
3.0@dev

PHP version
7.4.5

Operating system
Win 10 and Ubuntu 20.04

@rustamwin would you please take a look?

@rustamwin would you please take a look?

Yes.


Taking the inverse of an error sometimes returns NaN, not the error.

The expected output of inv(error) is an error, but here, it's NaN for some reason.
link
Example showing the error:

Example with the error wrapped in inv()
Notice the result is NaN, not an error.

A simple error failure - this time it propagates.
Interestingly, it doesn't break for this example. I'm not sure what the difference is.

@berekuk pointed out that the error is happening when the plot is displayed, which makes sense.
We could use better UI of course.


Manifests for fastq files contain duplicates

The manifests for downloading data via the cli contain all fastqs in duplicate. This makes it very difficult to download all the original files an process them. Its also a waste of disk space and bandwidth that is not easy to fix (i.e. deduplicate the files correctly). The specific use case that I have is to go to any project and download all the fastq by selecting the relevant checkbox.
You can then inspect the resulting manifest to see that there are two entries for all fastqs. A quick way to verify this in the command line is:
$ cat hca-manifest-fd577523-43b3-41c2-ab3e-68840b30df9e.tsv | cut -f 5 | sort | uniq -c
      2 01e81e7d8cff674eeb82685e064b20889e33ece6a7bc6f53c329b38bc0b6ea19
      2 0b55f8f21850a52db786e5c34fad871edc34f8418a987fc51a3da4216342b770
      2 3ab2fbde000eb422e46e086929b4100f90085193d7055bb95eac07ac3ef33795
      2 3beb0a8e1af62acc5911a145bbd64d4bee8dd407e6f174eb62472cd5a5c99960
      2 476079c0539062b3caf217ee0b51ea3c6118ebd00b6b0a586a861564bca9d169
[...]

┆Issue Number: HCAB-405

This could possible be caused by both the primary and the secondary bundles being returned, both of which point to the raw fastq files.

The manifest mentions every occurrence of a file in a bundle. It has to because the bundle assigns the file name and two bundles could potentially assign different names to a file. In other words, the manifest needs to continue to mention every occurrence of a file, so we can replicate the bundle structure on the users disk. However, we shouldn't download and store those duplicates.
This will be addressed by https://github.com/HumanCellAtlas/dcp-cli/issues/234.

Do we understand the root cause?  I thought the browser download the secondary bundle rather than the primary one if both were evist? (not that I think this is a go idea.

The root cause is the copy-forward of data and metadata from primary to analysis bundle as @malloryfreeberg suspected and HumanCellAtlas/dcp-cli#234 will fix this.

Sounds good, although this feels like working around a design flaw rather than fixing the root cause.

Copy-forward is a design-flaw in my opinion and we should address it. The underlying DSS features that make copy-forward efficient in the DSS are useful and the CLI download should exploit them. IOW, even without copy-forward, the same file could occur in more than one bundle, under two different names. And even two distinct files could incidentally have the same content. We'd also want to avoid downloading and storing them twice.

Is there a way for now to request only primary bundles? We just need the raw files for testing the pipelines.

Very confusingly if you look at the manifest that comes with the 'ischaemic sensitivity' dataset, all the bundles are duplicated, except one! Do we have an explanation for this? Is it a recently added bundle that hasn't been processed yet?
Here is a grouping of the copy of the files we have in our bucket that should make this clear:
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/31767f5c-2c9c-403e-824a-c040be32ffc2/HCATisStabAug177376561_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/31767f5c-2c9c-403e-824a-c040be32ffc2/HCATisStabAug177376561_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/31767f5c-2c9c-403e-824a-c040be32ffc2/HCATisStabAug177376561_S1_L001_R2_001.fastq.gz

gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/7050c147-8dae-4e47-b260-b729bece41cd/HCATisStabAug177276391_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/7050c147-8dae-4e47-b260-b729bece41cd/HCATisStabAug177276391_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/7050c147-8dae-4e47-b260-b729bece41cd/HCATisStabAug177276391_S1_L001_R2_001.fastq.gz

gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/a5b72567-b534-4c1d-abf7-cde41a451408/HCATisStabAug177376562_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/a5b72567-b534-4c1d-abf7-cde41a451408/HCATisStabAug177376562_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/a5b72567-b534-4c1d-abf7-cde41a451408/HCATisStabAug177376562_S1_L001_R2_001.fastq.gz

gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/91cc94ec-7ba7-4db0-a7ca-6d1449a847ee/HCATisStab7463846_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/91cc94ec-7ba7-4db0-a7ca-6d1449a847ee/HCATisStab7463846_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/91cc94ec-7ba7-4db0-a7ca-6d1449a847ee/HCATisStab7463846_S1_L001_R2_001.fastq.gz

gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/6cdc5687-a087-49ac-a2ba-46085adeffa9/HCATisStab7413619_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/6cdc5687-a087-49ac-a2ba-46085adeffa9/HCATisStab7413619_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/6cdc5687-a087-49ac-a2ba-46085adeffa9/HCATisStab7413619_S1_L001_R2_001.fastq.gz

gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/9d9c985b-e655-4646-8ff5-49dc2e768213/HCATisStabAug177184858_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/9d9c985b-e655-4646-8ff5-49dc2e768213/HCATisStabAug177184858_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/9d9c985b-e655-4646-8ff5-49dc2e768213/HCATisStabAug177184858_S1_L001_R2_001.fastq.gz

// Dataset present only once!
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/81f920f3-8645-439c-868e-1bf16ab9b0a7/HCATisStabAug177078016_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/81f920f3-8645-439c-868e-1bf16ab9b0a7/HCATisStabAug177078016_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/81f920f3-8645-439c-868e-1bf16ab9b0a7/HCATisStabAug177078016_S1_L001_R2_001.fastq.gz

/// Duplicates
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/0cbd9be1-5666-4341-a364-d5fdd6eed2a9/HCATisStabAug177184858_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/0cbd9be1-5666-4341-a364-d5fdd6eed2a9/HCATisStabAug177184858_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/0cbd9be1-5666-4341-a364-d5fdd6eed2a9/HCATisStabAug177184858_S1_L001_R2_001.fastq.gz

gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/84be47a9-3dfd-409e-8ad3-b97c4b6f0f3c/HCATisStabAug177376561_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/84be47a9-3dfd-409e-8ad3-b97c4b6f0f3c/HCATisStabAug177376561_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/84be47a9-3dfd-409e-8ad3-b97c4b6f0f3c/HCATisStabAug177376561_S1_L001_R2_001.fastq.gz

gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/080947fa-4932-48d9-93f2-2f46afb201e0/HCATisStabAug177276391_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/080947fa-4932-48d9-93f2-2f46afb201e0/HCATisStabAug177276391_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/080947fa-4932-48d9-93f2-2f46afb201e0/HCATisStabAug177276391_S1_L001_R2_001.fastq.gz

gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/0f7528b8-8067-43bc-abb5-3115031c4d67/HCATisStabAug177376562_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/0f7528b8-8067-43bc-abb5-3115031c4d67/HCATisStabAug177376562_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/0f7528b8-8067-43bc-abb5-3115031c4d67/HCATisStabAug177376562_S1_L001_R2_001.fastq.gz

gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/8d7bfcb3-fca8-4e34-98d4-c0b2a881004c/HCATisStab7463846_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/8d7bfcb3-fca8-4e34-98d4-c0b2a881004c/HCATisStab7463846_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/8d7bfcb3-fca8-4e34-98d4-c0b2a881004c/HCATisStab7463846_S1_L001_R2_001.fastq.gz

gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/16971394-3dec-4604-a9d8-65c7590652e6/HCATisStab7413619_S1_L001_I1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/16971394-3dec-4604-a9d8-65c7590652e6/HCATisStab7413619_S1_L001_R1_001.fastq.gz
gs://hca-dcp-sc-pipelines-test-data/referenceData/fastq/chemistry_X10_V2/HCA_ischaemic_sensitivity/16971394-3dec-4604-a9d8-65c7590652e6/HCATisStab7413619_S1_L001_R2_001.fastq.gz

Is there a way for now to request only primary bundles?

In data browser, request a manifest for the entire project. It should include references to BAMs.
Make a pass through the resulting manifest to collect the bundle UUIDs for the rows referencing a BAM. Those are the analysis bundles.
Make a second pass through the manifest and drop the rows referencing the UUIDs collected in the first step.

That's the best I can come up with, right now.
https://github.com/HumanCellAtlas/data-browser/issues/502

Very confusingly if you look at the manifest that comes with the 'ischaemic sensitivity' dataset, all the bundles are duplicated, except one! Do we have an explanation for this? Is it a recently added bundle that hasn't been processed yet?

I don't know. Looks like you are already discussing this with the wranglers on Slack.

Obsolete DCP/1 issue.


Prevent underflow when using a MockI2CDevice with a RegisterMap.offset = 0x0

Cast both the offset and the slice.len() to isize so we can safely underflow to a negative number. We lose out on 1-bit of possible offset/slice size, but I'd love to have your computer if you're bumping into that.
Adds a quick test to verify the fix as well.
This fixes #77.

How does that look?

Done.


Unformatted number for "not enough EMC" message when extracting from an EMC Link

As of Project Expansion 1.0.9 for Minecraft 1.18.2, when attempting to extract items from an EMC Link but without having enough EMC to extract the item, the message "You do not have enough emc to purchase this; you need %s" is displayed, where %s is the EMC value of the item.
However, this value is displayed raw, with no formatting whatsoever.
Example when extracting an item worth 1.23 Million EMC: "You do not have enough emc to purchase this; you need 1234567"
Corrected: "You do not have enough emc to purchase this; you need 1.23 Million"
This is because there are still some instances where String.valueOf is used to format a (large) EMC value, where EMCFormat.format should be used.
Example:
https://github.com/DonovanDMC/ProjectExpansion/blob/aab6f320c5d0a543b7e6bc3b81c24065d069284c/src/main/java/cool/furry/mc/forge/projectexpansion/item/ItemMatterUpgrader.java#L120
Non-example:
https://github.com/DonovanDMC/ProjectExpansion/blob/cd2445b6569034298bf014c523aa56edcc6b38e8/src/main/java/cool/furry/mc/forge/projectexpansion/block/entity/BlockEntityEMCLink.java#L394

Fixed in latest releases
1.14: 1.0.10
1.15: 1.0.29
1.16: 1.0.30
1.18: 1.0.10


[0.7] More specific registered constructor is not called

Hi,
I have hierarchy of objects that look like this one:
class A {
  private InnerClass $inner;
}

class InnerClass  {
  private int $property1;
  private int $property 2;

  public function __construct(int $property1) {
    $this->property1 = $property;
    $this->property2 = 42;
  }

  public static function customConstructor(int $property1, int $property2): self {
    $result = new self($property1);
    $result->property2 = $property;
    return $result;
  }
}

With 0.7, I register my constructor trough  $mapperBuilder->registerConstructor(B::customConstructor(...))
Then I try to map A, but the native constructor is always called and not the custom one even though it's more specific  (that was working on 0.6).

Could this be because of the class hierarchy? I do not map directly on SomeClass but on another class that have a property with SomeClass as type

This is also working on my side:
final class SomeParentClass
{
    public readonly SomeClass $someClass;

    public function __construct(SomeClass $someClass)
    {
        $this->someClass = $someClass;
    }
}

final class SomeClass
{
    public readonly int $property1;
    public readonly int $property2;

    public function __construct(int $property1)
    {
        $this->property1 = $property1;
    }

    public static function customConstructor(int $property1, int $property2): self
    {
        $instance = new self($property1);
        $instance->property2 = $property2;

        return $instance;
    }
}

$mapper = (new MapperBuilder())
    ->registerConstructor(
        SomeClass::class,
        SomeClass::customConstructor(...)
    )
    ->mapper();

$resultA = $mapper->map(SomeParentClass::class, [
    'someClass' => ['property1' => 1, 'property2' => 2]
]);

assert($resultA->someClass->property1 === 1);
assert($resultA->someClass->property2 === 2);

$resultB = $mapper->map(SomeParentClass::class, [
    'someClass' => ['property1' => 42]
]);

assert($resultB->someClass->property1 === 42);

I'm afraid I'll need more details about your usecase to understand what's wrong.

Hi,
I've got bitten by the fact that registerConstructor returns a new instance of the mapper builder.
I was doing
$mapperBuilder = new MapperBuilder();
$mapperBuilder->registerConstructor(..);

Sorry for the rookie mistake!
Everything is fine. Still looking forward for a way to whitelist my constructors without having to declare them.
Have a nice week.


Exercise: D&D Character

http://forum.exercism.org/t/select-featured-exercises-for-december-diversions/7971/14

@ErikSchierboom I didn't realize the groovy team is empty.

Thanks for this!