Is _FillValue really the same as zarr's fill_value?

The zarr backend uses the fill_value of zarrs .zarray key as if it would be the _FillValue according to CF-Conventions:
https://github.com/pydata/xarray/blob/1a7b285be676d5404a4140fc86e8756de75ee7ac/xarray/backends/zarr.py#L373
I think this interpretation of the fill_value is wrong and creates problems. Here's why:
The zarr v2 spec is still a little vague, but states that fill_value is

A scalar value providing the default value to use for uninitialized portions of the array, or null if no fill_value is to be used.

Accordingly this value should be used to fill all areas of a variable which are not backed by a stored chunk with this value. This is also different from what CF conventions state (emphasis mine):

The scalar attribute with the name _FillValue and of the same type as its variable is recognized by the netCDF library as the value used to pre-fill disk space allocated to the variable. This value is considered to be a special value that indicates undefined or missing data, and is returned when reading values that were not written.

The difference between the two is, that fill_value is only a background value, which just isn't stored as a chunk. But _FillValue is (possibly) a background value and is interpreted as not being valid data. In my opinion, this mix of _FillValue and missing_value could be considered a defect in the CF-Conventions, but probably that's far to late as many depend on this.
Thinking of an example, when storing a density field (i.e. water droplets forming clouds) in a zarr dataset, it might be perfectly valid to set the fill_value to 0 and then store only chunks in regions of the atmosphere where clouds are actually present. In that case, 0 (i.e. no drops) would be a perfectly valid value, which just isn't stored.  Other formats (e.g. OpenVDB) commonly use this trick.

The issue gets worse when looking into the upcoming zarr v3 spec where fill_value is described as:

Provides an element value to use for uninitialised portions of the Zarr array.
If the data type of the Zarr array is Boolean then the value must be the literal false or true. If the data type is one of the integer data types defined in this specification, then the value must be a number with no fraction or exponent part and must be within the range of the data type.
For any data type, if the fill_value is the literal null then the fill value is undefined and the implementation may use any arbitrary value that is consistent with the data type as the fill value.
[...]

Thus for boolean arrays, if the fill_value would be interpreted as a missing value indicator, only (missing, True) or (False, missing) arrays could be represented. A (False, True) array would not be possible. The issue applies similarly for integer types as well.

I encountered this issue today and totally agree. In Zarr, fill_value=X is used to say all uninitialized parts of the array have value X (useful for sparse arrays), NOT the meaning of _FillValue which is that all values in the array with value X are missing/undefined and so Xarray should convert those values to NaN. This is a minor issue for Zarr v2 and will be a bigger issue in Zarr v3, particularly for certain types of dtypes as @d70-t mentions. To update @d70-t 's comment on Zarr v3, it looks like in v3, fill_value: null is not allowed to be recorded in the metadata.
The default fill_value set by Zarr-Python for an array is 0 for int, 0.0 for float, and False for bool. This means that by default, when Xarray loads a Zarr array with one of those values, that value is converted to NaN.
Here is an MWE with ints:
import zarr
import xarray as xr

store = zarr.DirectoryStore("zarr-xarray-example.zarr")
root = zarr.group(store=store, overwrite=True)
arr = root.create_dataset(name="arr", data=[0, 1, 2, 3, 4], dtype="i4")
arr.attrs["_ARRAY_DIMENSIONS"] = ["phony_dim_0"]
zarr.consolidate_metadata(store)

root = zarr.open("zarr-xarray-example.zarr")
print(root["arr"].fill_value)

dset = xr.open_dataset("zarr-xarray-example.zarr", engine="zarr")
dset.arr.data

Returns:
0
array([nan,  1.,  2.,  3.,  4.])

This was very confusing at first. Why did 0 become NaN? Why did the ints become floats? Upon digging around, I learned that this automatic conversion/masking can be turned off by passing the argument mask_and_scale=False (default is True) or decode_cf=False (default is None) to open_dataset. However, because the same argument controls both masking and scaling, I cannot have Xarray use the scale_factor and add_offset attributes of a sparse zarr array that has fill_value=X, without all X values in the array (which represent value X, not invalid/missing data) being converted to NaN. That's frustrating. This conversion does not happen if the zarr array has fill_value=None, which has to be explicitly set in Zarr-Python, and will be invalid in v3 (I believe).
Now, this is a very specific and probably uncommon use case. But still, Zarr's fill_value and xarray/netcdf's _FillValue mean different things, and I think automatically setting _FillValue to the value of the zarr fill_value on read is incorrect and makes it hard to work with non-Xarray/non-CF Zarr data in Xarray.
To resolve this, I suggest that on write, Xarray writes the dataset _FillValue property to a _FILL_VALUE attribute on the Zarr array (similar to _ARRAY_DIMENSIONS) instead of to the fill_value property, and then on read, Xarray sets the dataset _FillValue property based on that new attribute, NOT the zarr array fill_value property. That would preserve roundtrip behavior of Xarray to/from Zarr, while not messing with the loading of non-Xarray/non-CF Zarr arrays with fill_value into Xarray. This would also help with eventual Zarr v3 support. Naming the attribute something else like _MASK_VALUE or _MISSING_VALUE would might be less confusing.
However, this approach would break the default masking behavior when loading existing Zarr data that lacks this new _FILL_VALUE attribute, particularly those Zarr files written by Xarray which used fill_value to mean _FillValue. Maybe if a user wants to set the dataset _FillValue property to the zarr array fill_value property, they could provide a new argument use_fill_value in open_zarr. It could have default True and then switch to default False after some time and deprecation warnings? I'm not sure how best to handle that.

I ran into the same problem as well. This simple code fails, because xarray interpret the default value of zero as a missing value. It would be great if this gets fixed, because it is easy to overlook.
import zarr
import xarray as xr
import numpy as np

with zarr.open('dump.zarr', mode='w') as z:
    z_time = z.zeros('time', shape=(0), chunks=(1), dtype='f4')
    z_time.attrs['_ARRAY_DIMENSIONS'] = ['time']

for i in range(0, 10):
    with zarr.open('dump.zarr', mode='r+') as z:
        ii = np.array([i], dtype='f4')
        z['time'].append(ii)

with zarr.open('dump.zarr', mode='r') as z:
    print(z['time'].info)
    print(z['time'][:])

with xr.open_dataset('dump.zarr') as z:
    print(z)
    print(z['time'])

The latter print gives a nan for the value time equals 0.0.

In the mean time, you can set mask_and_scale=False in open_dataset and it should avoid this problem

What is the issue? Are you seeing a roundtripping problem?

but it seems surprising.
so I guess the behavior is correct.

The behaviour is correct from Xarray's POV but this example isn't well defined from a data POV. For one, in netCDF land _FillValue is supposed to be outside the range of valid data (reference).
And yeah, Xarray's fill value decoding is already lossy (it doesn't distinguish between _FillValue and missing_value) so the roundtrip test should only work with decode_cf=False.

Zarr's fill_value should be automatically interpreted as the _FillValue when reading and writing. IMO, they seem different.

It's confusing and the meaning has become overloaded over time, but the original intent seem the same to me IMO (treating zarr's write_empty_chunks as an implementation detail)  (reference):
(1) "The _FillValue attribute specifies the fill value used to pre-fill disk space allocated to the variable."
(2) "The fill value is returned when reading values that were never written."

What are our options (spoiler: after writing this up I think 4 is most promising)?

Change the Zarr v3 spec to make fill_value optional.?

This would effectively restore the Zarr v2 behavior: the "surprising" behavior of valid values potentially being masked as invalid would only affect users where _FillValue / fill_value happens be set explicitly.
I don't really like this option. Ideally zarr datasets are portable across implementations, and having some implementations interpret the same dataset differently because they happen to have different default fill values makes me uneasy.

Use different default fill_values for zarr-python v3?

The fact that zarr-python v3 uses 0 as the default fill value when creating an array in memory makes this more problematic in practice. Using other default fill values (NaN for float, NaT / INT_MIN for datetime, INT_MIN for int?) might mean users run into this less often. But it doesn't help for bool at all. uint would need to chose 0 to be consistent with the others, or INT_MAX. Meh.

Change the default in xarray to not mask these values?

I don't know what the tolerance is here.

Make the masking function smarter to only mask uninitialized regions

IIUC, the main purpose of this _FillValue is "reading values that were never written." What if we only applied the masking in CFMaskCoder at https://github.com/pydata/xarray/blob/cc74d3ae226594db23730279711d4379debcf9a8/xarray/coding/variables.py#L221-L232 only to chunks that weren't initialized? I believe that zarr-python knows which chunks of an Array are initialized.
Then we avoid the situation where you have some valid values cast to NaN (or whatever) since they happened to be equal to fill value. You wouldn't have any valid values in the chunk since it was never initialized.
This might be better done in Zarr-Python, and IIRC there's some desire to move some of the CF related logic into Zarr Python as new codecs.

TLDR: perhaps we change the tests instead?

I'm not sure there is a good overarching solution.
fill_value can either be semantically valid (e.g. the sparse array case or "background value"; this issue) or semantically missing (_FillValue or missing_value in CF/netCDF case). We can't decide which meaning the user has assigned to Zarr's fill_value without extra information (NUG/CF provides valid_min, valid_max, valid_range, and missing_value for example).
Xarray's mask_and_scale decoder assumes _FillValue is to be interpreted as semantically missing always (or at least is best represented in-memory by a NaN). IMO this is the core problem. This can be prevented by explicitly setting mask_and_scale=False and manually masking if you need to.

Responding to your suggestions
(1) Change the Zarr v3 spec to make fill_value optional? +1 I don't understand why fill_value must be required. IIRC even netcdf has a no-fill mode.
(2) Use different default fill_values for zarr-python v3? implicit default fill values aren't great especially since the meaning is already not-explicit. netCDF4 does have these but Xarray does not use them.
(3) Change the default in xarray to not mask these values? We should allow more sophisticated control over the decoding pipeline (there are already requests for this). For example, only mask missing_value when present, etc. It seems too late to change this default.
(4) Make the masking function smarter to only mask uninitialized regions when decoding?  This ignores the semantic issues. An uninitialized region could just a sparse array chunk with no non-fill-value data.

In general, I think we should distinguish between what is recorded on disk, and what is interpreted as an in-memory data model. To me, it seems this issue is about two things:

Imperfect recording of semantic meaning on disk
Xarray's imperfect decoding of the recorded information (no support for masked arrays, no handling of valid_* and blindly assuming _FillValue is semantically invalid or missing data).

This implies that we need to have two subtly different roundtripping tests:
(1) being exactly correct roundtripping data from the backend with decode_cf=False and potentially a new encode_cf=False. This was we know we receive and send unmodified values to the backend.
(2) being exactly correct when writing any Xarray object to disk and reading it back. So we ensure that Xarray is internally consistent, and Xarray users can roundtrip their data perfectly.

Thanks, I've been thinking this over since yesterday and am still not sure where I sit. I just keep coming back to this being a really bad behavior
In [1]: import xarray as xr

In [2]: ds = xr.Dataset({"x": xr.DataArray([0, 1, 2], name="x")})

In [3]: store = {}

In [4]: ds.to_zarr(store)
Out[4]: <xarray.backends.zarr.ZarrStore at 0x10914d640>

In [5]: xr.open_dataset(store, engine="zarr").load()
Out[5]:
<xarray.Dataset> Size: 24B
Dimensions:  (dim_0: 3)
Dimensions without coordinates: dim_0
Data variables:
    x        (dim_0) float64 24B nan 1.0 2.0

So I guess we agree that xarray interpreting values equal to fill_value as NaN (at least unconditionally) is the core problem.
With that said, would xarray be open to me implementing https://github.com/pydata/xarray/issues/8359, which AFAICT proposes to only do the masking for values that happen to fall outside of valid range defined by those attributes, and so are "semantically invalid"?
Under that proposal, my snippet would correctly round trip. To restore the old behavior, users would need to specify the valid_min / max / range in the attrs (or encoding?), such that the fill_value (which zarr defaults to 0, but could be overridden) falls outside the valid range.

Yes, I'd agree the core issue is: xarray treats zarr's fill_value as NaN, unconditionally. In many cases, that is a really bad behavior.
With the recent discussion, I guess I see that the original idea of netCDF's _FillValue is similar to zarr's fill_value, so it seems to be a sensible choice to use them interchangeably. I guess the deeper issue here is, that since all the time of netCDF, _FillValue is being confused with missing_value (one to fill empty space, one to indicate no data, which may or may not be the same). I guess this confusion isn't ideal in netCDF world either, but now people are somewhat used to it, and as it's optional, it's not too bad.
So if xarray's zarr ties fill_value to _FillValue, it's also tied to missing_value and we inherit all the confusion. But to make things work (think especially of bool type data), fill_value must be disconnected from missing_value. This either can be done

by disconnecting fill_value from _FillValue or
by disconnecting _FillValue from missing_value

I'd guess the former would be simpler, the latter would be more correct / elegant.

I am the one who originally implemented the current logic (equating Zarr's fill_value with CF _FillValue), and I am quite comfortable agreeing with what everyone has said above: the currently logic is wrong and makes no sense. 😆  When I wrote this code, I just naively saw the name "fill value" and assumed that it was the same thing, without really thinking through the semantics in detail, as has been done above.
I believe we can and should fix this by changing Xarray's behavior, with minimal impact on users.
A useful concept for me is to realize that the Zarr / Xarray stack actually has two distinct layers of encoding (Zarr's own "filters", as opposed to the NetCDF / Xarray stack, which only has one (implemented by Xarray). Using CF conventions with Zarr creates many ambiguities, as it is not always clear where in the stack the encoding / decoding should be applied
Converting _FillValue to NaN is an example of a NetCDF-specific decoding that Xarray applies with reading both NetCDF and Zarr. It makes sense for NetCDF. But it doesn't make sense to do the same with Zarr's fill_value. It may be the case that Zarr data with values equal to fill_value is perfectly valid, not to be interpreted as "missing data". The Zarr fill_value is Zarr specific and should not receive special treatment in Xarray, other than keeping track of it in encoding.
To recreate the behavior we have today, we could set the Zarr array attribute missing_value to be equal to the array fill_value.
I would be inclined not to worry too much about backwards compatibility with existing behavior. This needs to be fixed and the sooner we do so, the better off everyone will be.

I just spent the past hour re-reading all of the NetCDF and CF documentation on this, and reached the same conclusion that @dcherian did a few weeks ago (https://github.com/pydata/xarray/issues/5475#issuecomment-2341713620): this is a mess. It's truly ambiguous how to interpret _FillValue.
From the CF Conventions:

The scalar attribute with the name _FillValue and of the same type as its variable is recognized by the netCDF library as the value used to pre-fill disk space allocated to the variable... The _FillValue should be outside the range specified by valid_range (if used) for a variable...  Applications that process variables that have attributes to indicate both a transformation (via a scale and/or offset) and missing values should first check that a data value is valid, and then apply the transformation. Note that values that are identified as missing should not be transformed. Since the missing value is outside the valid range it is possible that applying a transformation to it could result in an invalid operation.

This is how we do decoding today in Xarray--first mask _FillValue, then scale
https://github.com/pydata/xarray/blob/cde720f21788a1dbff2bdb0775a56b3ee6da6b1a/xarray/conventions.py#L279-L284
However, missing_value is treated differently, according to the  NUG

This attribute is not treated in any special way by the library or conforming generic applications, but is often useful documentation and may be used by specific applications. The missing_value attribute can be a scalar or vector containing values indicating missing data. These values should all be outside the valid range so that generic applications will treat them as missing.
When scale_factor and add_offset are used for packing, the value(s) of the missing_value attribute should be specified in the domain of the data in the file (the packed data), so that missing values can be detected before the scale_factor and add_offset are applied.

This means that we should be reversing the order of the decoding operations depending on whether we are masking a _FillValue (in raw data space) or missing_value in scaled data space. AFAICT this does not happen in Xarray. Instead, we treat the two as identical and raise an error if they differ:
https://github.com/pydata/xarray/blob/cde720f21788a1dbff2bdb0775a56b3ee6da6b1a/xarray/coding/variables.py#L389-L392

Final thought in my rant: another reason this is very hard to solve is the lack of true missing value support in Xarray. We have no good way to actually mask int data. Since int has no obvious sentinel value for missing values (unlike float, which has NaN), we have to resort to treating Zarr's fill_value as a mask. That, combined with the default fill_value of 0, is the cause of the bug described by @TomAugspurger above.
Ok, enough complaining, now time to think about how to actually solve this. 😆
A simple workaround that would solve the most obvious bug (int array with 0 as missing values) could be to just stop interpreting fill_value as missing data for int Zarr arrays. We could perhaps introduce an internal interpret_fill_value_as_missing_data flag in CFMaskCoder and toggle this on and off in different scenarios.
A more comprehensive fix would involve a deep rethinking of how we handle missing data in both Xarray and Zarr.

indeed, the best way to resolve this would involve having true missing value support in our in-memory arrays.
There's two options (as described in NEP 26): masked arrays (i.e. a additional mask array directly on the array) and bit-pattern based missing values on the dtypes.
For the former, there's several promising attempts, most recently marray (as I discovered earlier today, this is already works but is incomplete, see https://github.com/pydata/xarray/issues/1194#issuecomment-2381367576 – to know how much we'll have to either investigate manually or make progress on xarray-array-testing).
For the latter, we'd need to have someone dive into the custom dtype API that has already been used to define the new string dtypes. I also played around with that earlier today, but it looks like it will take me a lot more time to figure out since I don't really know the python and numpy C APIs too well. (In any case I don't want to be the bottleneck, so if someone wants to help out / take over I'd be happy to collaborate – and I'm not involved in that but I guess the same goes for marray)

https://github.com/pydata/xarray/issues/9515#issuecomment-2359242213 has a suggestion / question to set xarray's default zarr version to 2, to punt on this issue a bit.

Tom, we are quite motivated to find a way to write V3, since it is the only thing that works with Icechunk.
I guess I'm failing to see why this is such a problem for V3 but not V2. Is it that V3 doesn't allow an empty fill value?

Ah of course I forgot about that. I was hesitant to suggest xarray not follow the library default in the first place. So let's work through the transition pains.

Is it that V3 doesn't allow an empty fill value?

Correct. So it's a problem for both v2 and v3, just more pronounced in v3 since fill_value must be specified in the array metadata.

Sorry if what I'm saying below is obvious to people. Here's a summary of how it works in V2 vs V3:
Zarr 2.17.1
import zarr
a = zarr.create(shape=10, chunks=10, dtype=int)
# fill_value set to 0 by default
assert a.fill_value == 0
assert a[0] == 0
# None allowed as fill_value
b = zarr.create(shape=10, chunks=10, dtype=int, fill_value=None)
assert b.fill_value is None
# however, 0 is still being used as a fill value (no data have been written); this seems weird
# can we count on this behavior?
assert b[0] == 0

The problem is that Xarray is using the presence of the fill_value attribute to determine whether to apply a mask, via setting the CF _FillValue attribute
https://github.com/pydata/xarray/blob/cde720f21788a1dbff2bdb0775a56b3ee6da6b1a/xarray/backends/zarr.py#L623-L624
An, on the encoding side, setting fill_value as follows
https://github.com/pydata/xarray/blob/cde720f21788a1dbff2bdb0775a56b3ee6da6b1a/xarray/backends/zarr.py#L824-L826
Here's how it works in V3 (3.0.0a7.dev4+ga3221245)
import zarr
# fill_value set to 0 by default, same as V3
assert a.fill_value == 0  # passes; actually it is np.int64(0)
assert a[0] == 0
b = zarr.create(shape=10, chunks=10, dtype=int, fill_value=None)
# fails! it is still np.int64(0)
assert b.fill_value is None
# passes
assert b[0] == 0

So even though the V3 array behaves the same as the V2 array, it lacks the ability to set fill_value to None. This means that V3 always triggers Xarray's masking behavior.
Having written this summary, the answer seems a bit more clear: we need another way for Zarr to tell Xarray whether to apply masking.
Here's an idea: for V3 arrays (and maybe even also V2), we can use the _FillValue attribute for this. Rather than removing it from the attrs:
fill_value = attrs.pop("_FillValue", None)

...we just do
fill_value = attrs.get("_FillValue", None)

If this attr is missing, we will never apply a mask, regardless of what the Zarr .fill_value is. Effectively, we are using _FillValue as a sentinel attr to hint that the data should be masked, decoupling this from the actual Zarr fill_value.
Some questions that arise:

Would applying this retroactively to V2 data break existing datasets? I think so. So perhaps we would only want to do it for V3.
What if _FillValue and the array fill_value are different? Should that be allowed?

however, 0 is still being used as a fill value (no data have been written); this seems weird
can we count on this behavior?

https://github.com/zarr-developers/zarr-python/pull/2274 is a draft PR restoring that behavior on zarr-python 3.x for zarr-v2. But for zarr-v3, fill_value should always be non-None, so xarray can't rely on it in general.

Here's an idea: for V3 arrays [...]

That sounds reasonable to me. Users who want the masking behavior can explicitly set _FillValue as an attribute / encoding.
I do think it has the potential to break reading some v2 datasets, where people were relying on this behavior. FWIW, this is surprising enough to me that I'd be comfortable calling it a bug fix, but I'll leave that up to the experts.

I'm planning to take a swing at this over the weekend.

Here's an idea: for V3 arrays (and maybe even also V2), we can use the _FillValue attribute for this. Rather than removing it from the attrs

I'd vote for this (I guess that's also what I tried to outline in https://github.com/pydata/xarray/issues/5475#issuecomment-2031362185).

Would applying this retroactively to V2 data break existing datasets? I think so. So perhaps we would only want to do it for V3.

I think it could break datasets (but maybe that's fine).

If I get it right, writing _FillValue into new v2 datasets shouldn't hurt at all.
Reading a dataset where fill_value and _FillValue is present and preferring _FillValue over fill_value to set values to NaN in case of a difference between both should also be ok I guess (?). It might create problems when reading new-style datasets with old-style readers...
Not using a lone fill_value to mask values probably would be noticed, but I'd still aggree with @TomAugspurger that it probably should be considered a bugfix.

One might want to explicitly write _FillValue as None to indicate that one wants to opt-in into a potential new behaviour, where fill_value is not used anymore to mask values.

What if _FillValue and the array fill_value are different? Should that be allowed?

In my opinion, this should definitely be allowed. The two serve different purposes. One might however spend a little thought, if _FillValue should have a different name (e.g. missing_value)...

Over in #9552 I have implemented a fix for this issue, with the new approach to fill_value in Zarr V3. It keeps things unchanged for V2 but uses _FillValue attributes in V3.


Feature ion-photo-gallery

Can You make such component?
I see it like this:
ion-photo-gallery fullscreen="true/false"
img src
img src
img src
/ion-photo-gallery
It would show slidable pictures on black screen adjust to screen size keeping aspect ratio.
What do Yuo think?

This would be a great plugin addition to Ionic. +1

+1 !


Update delegate-metrics.md

There may have been too many people making changes to this file at one time.  I rewrote a paragraph that I thought had been worked on before but that I noticed was published.
Harness Developer Pull Request
Thanks for helping us make the Developer Hub better. The PR will be looked at
by the maintainers.
What Type of PR is This?

[ ] Issue
[ ] Feature
[x ] Maintenance/Chore

If tied to an Issue, list the Issue(s) here:

Issue(s) Editorial fixes

House Keeping
Some items to keep track of. Screen shots of changes are optional but would help the maintainers review quicker.

[ ] Tested Locally
[ ] Optional Screen Shoot.

Preview environment: https://ci.pr.harness.io/pr-739


Will Create Calculator app with animation in pure HTML, CSS, Js

I'm a Gssoc22 contributor

Should I start working on the project? Have you assigned?

I WANT TO WORK ON THIS ISSUE UNDER GSSOC'22.

I want to work on this under GSSOC2022. Can you assign it to me.


Changed percent to round two decimal places.

Perfect! Would you mind checking for more places where I didn't round the percentages? I'm not the cleanest coder so I think similar ratios appear in other places without rounding


get_sinks outdated cache

If a sink configuration is updated via set_sink_config method, the following calls to get_sinks will return old cached configurations. I did not use a callback so the execution was synchronous(I also tried with a callback and calling get_sinks from there, but I still got old data).
As a workaround I used set_config_changed_cb to listen for changes and run subsequent calls to get_gateways and get_sinks in order to get latest sink configuration.

Hello,
You are correct. When getting the answer form the set config, the new sink config is included in response so it is quite trivial to update the cached version.
I made a patch for it and will publish it as soon as I made more testing on it.
Thanks!


Revert equally secure accounts.

Firefox Accounts is more secure than GitHub.
Re-enable the OIDC Refresh Token middleware.
Display which account a user must use in order to login in the error message.

Coverage increased (+0.2%) to 89.607% when pulling a1db20ff2c4197d718abb26bc318b7ad6a6ff49b on akatsoulas:revert-equally-secure-accounts into 64b4596d2b43298d2016db05a6116955a3b7526a on mozilla:master.

Let's check if there are any stale data/schema migrations that need to be reverted.


fix CODEOWNERS

Description
I am retiring from DeNA soon so have removed myself from CODEOWNERS.
Contributor License Agreements

[x] By placing an "x" in the box, I hereby understand, accept and agree to be bound by the terms and conditions of the Contribution License Agreement.

Check dist had been broken since https://github.com/DeNA/setup-job-workspace-action/pull/103, so this pull-req is not related.


Az interactive is wrong

This is autogenerated. Please review and update as needed.
Describe the bug
Command Name
az interactive
Errors:
The command failed with an unexpected error. Here is the traceback:
No module named 'prompt_toolkit.interface'
Traceback (most recent call last):
  File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/interactive/custom.py", line 41, in start_shell
    azext_interactive.start_shell(cmd, style=style)
  File "/home/proxxi/.azure/cliextensions/interactive/azext_interactive/__init__.py", line 10, in start_shell
    from .azclishell.app import AzInteractiveShell
  File "/home/proxxi/.azure/cliextensions/interactive/azext_interactive/azclishell/app.py", line 33, in <module>
    from prompt_toolkit.interface import Application, CommandLineInterface
ModuleNotFoundError: No module named 'prompt_toolkit.interface'

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
az interactive

Expected Behavior
Environment Summary
Linux-5.15.0-1021-azure-x86_64-with-glibc2.35, Ubuntu 22.04.1 LTS
Python 3.10.5
Installer: DEB

azure-cli 2.41.0

Extensions:
interactive 0.4.5

Dependencies:
msal 1.20.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

@jiasli for awareness

Duplicate of https://github.com/Azure/azure-cli/issues/24213


Remove tutorials from website

Type of Change

Routes/Pages: Tutorials list, tutorial show, API
Components: Tutorials
Styling: Tutorials, API
Utilities: Routes, sitemap

What issue does this relate to?
cc https://github.com/cdnjs/cdnjs/issues/14130
What should this PR do?
Removes tutorials from the website for libraries, and marks the tutorials API endpoints as deprecated.
What are the acceptance criteria?
Tutorials are not shown as part of the library pages, and the dedicated tutorial routes no longer work. Tutorials are not included in the sitemap, nor in any JSON-LD data.
The tutorial API endpoints are marked with a deprecation notice in the documentation.

Live in staging: https://staging.digitalocean.cdnjs.dev/api#tutorials


Update capability.py

resolves #8841

Problem

Solution

Checklist

[ ] I have read the contributing guide and understand what's expected of me
[ ] I have run this code in development and it appears to resolve the stated issue
[ ] This PR includes tests, or tests are not required/relevant for this PR
[ ] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
[ ] This PR includes type annotations for new and modified functions

@dataders LGTM, it's just a docstring update, and @peterallenwebb has already approved, so I'm going to press the "merge" button


mac m1 怎么安装兼容?

mac m1 怎么安装兼容?

paddle 不支持这个架构，所以paddlespeechw 也不能支持，抱歉

Any workaround then ? I have mac M2, and would like to test this.


Some Questions

Hi @arxyzan ,

Can you tell me what parts I need to change if my input size is 256 instead of 224?

Is it mandatory to load encoder_checkpoint? Can I train my model from scratch?

why is the config file named beit-pretraining.yaml for the vision task?

Could you help me to solve problem below:

Epoch: 1/100      0%|                                                                      | 0/18001 [00:02<?, ?batch/s]
Traceback (most recent call last):
File "/mnt/c/data2vec-pytorch/train.py", line 25, in
trainer.train()
File "/mnt/c/data2vec-pytorch/vision/trainer.py", line 131, in train
train_loss = self.train_epoch(epoch)
File "/mnt/c/data2vec-pytorch/vision/trainer.py", line 97, in train_epoch
loss = self.train_step(batch)
File "/mnt/c/data2vec-pytorch/vision/trainer.py", line 56, in train_step
x, y = self.model(src, trg, mask)
File "/home/bryan/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/c/data2vec-pytorch/data2vec/data2vec.py", line 90, in forward
y = self.ema.model(trg, ~mask, **kwargs)['encoder_states']  # fetch the last transformer layers outputs
File "/home/bryan/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/c/data2vec-pytorch/vision/encoder.py", line 38, in forward
outputs = self.encoder(pixel_values=inputs, output_hidden_states=True, output_attentions=True, **kwargs)
File "/home/bryan/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/bryan/.local/lib/python3.10/site-packages/transformers/models/beit/modeling_beit.py", line 681, in forward
embedding_output = self.embeddings(pixel_values, bool_masked_pos)
File "/home/bryan/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/bryan/.local/lib/python3.10/site-packages/transformers/models/beit/modeling_beit.py", line 154, in forward
embeddings = self.patch_embeddings(pixel_values)
File "/home/bryan/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/bryan/.local/lib/python3.10/site-packages/transformers/models/beit/modeling_beit.py", line 206, in forward
embeddings = self.projection(pixel_values).flatten(2).transpose(1, 2)
File "/home/bryan/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/bryan/.local/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/bryan/.local/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

Hello @bryanwong17, and thanks for your feedback.
Just keep in mind that Data2Vec is by itself a large pretrained model and only if you actually want to pretrain a model for vision/text/audio you can use it. Otherwise, if you want to finetune a model for a downstream task like image recognition, etc you have to use the pretrained weights and finetune from there. With all that:

In order to change the input size, you can only do it for pretraining and you cannot change the model architecture for finetuning, because finetuning needs the pretrained model which is already trained with 224 pixel architecture. (this is not the case for text and audio cause they accept different input sizes)
Loading from encoder_checkpoint only is there so the code would know which base model you are using based on the HuggingFace Hub path you provide. Actually no weight assigning or loading is happening there. It just loads the config file from that path in HF Hub and figure out what model you want to use. This way I was able to provide a general encoder class for vision by using transformers.AutoModel and transformers.AutoConfig. Otherwise one would have to provice one encoder class for any base architecture they wanted to use.
Because the default data2vec for vision, uses the model BEiT as the base encoder model. If you want to use another model you can provide a new config file for that.
Thanks for reporting this issue. I just fixed it. you can now try the code and it works fine.

Hi @arxyzan, thanks for quick response. My goal in training Data2Vec is to replace the Multiple Instance Learning (MIL) framework's feature extractor with Data2vec, allowing all extracted images (patches) to be loaded into the trained embedded. Do you suggest training datavec from scratch? So far, I have around 500k histopathology images.

Data2Vec and similar large image models are all trained on huge amount of data from ImageNet. Considering the fact that the domain of the data is fairly different from what you are working on, I don't think using a large model which is designed for large pretraining is the perfect decision unless you have this much data from your desired domain. I'm not familiar with histopathology data but I think you'd be better look for a base model that is suitable for that kind of data.


Implementation of ID algorithm?

Hi there,
Back door and IV is nice, but for larger graphs, I am wondering if it would be worthwhile implementing Ilya Shpitsers ID algorithm, which is complete.  Implementations of ID are currently available in R and in clojure.
By the way, if you haven't checked out Whittemore, it implements a  causal programming paradigm similar in philosophy to doWhy

Thanks @djinnome ! Definitely, implementing the ID algorithm is on our priority list, but being delayed with work on refutations. Would you be willing to contribute to the development?
Yes, I really the idea behind Whittemore--had a chance to talk to Joshua (who built Whittemore) a few weeks back.

Hi Amit,
Contributing to the development of the ID algorithm (and possibly the ID* and IDC* algorithmshttps://arxiv.org/abs/1206.5294) in DoWhy is exactly the kind of contribution our causality reading group is looking to make.  We are about 1/3 of the way through Chapter 4http://bayes.cs.ucla.edu/PRIMER/primer-ch4.pdf of Judea Pearl's Causal Inference in Statistics: A Primer.
When we are done with that, we will focus our effort on the implementation of these algorithms.
Sincerely,
Jeremy
From: Amit Sharma<EMAIL_ADDRESS>Reply-To: Microsoft/dowhy<EMAIL_ADDRESS>Date: Thursday, April 11, 2019 at 5:43 AM
To: Microsoft/dowhy<EMAIL_ADDRESS>Cc: "Zucker, Jeremy D"<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [Microsoft/dowhy] Implementation of ID algorithm? (#50)
Thanks @djinnomehttps://github.com/djinnome ! Definitely, implementing the ID algorithm is on our priority list, but being delayed with work on refutations. Would you be willing to contribute to the development?
Yes, I really the idea behind Whittemore--had a chance to talk to Joshua (who built Whittemore) a few weeks back.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/dowhy/issues/50#issuecomment-482098319, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAIzUnEzLEml8SiJiJI8SWX5DqPIAMVCks5vfy3KgaJpZM4cdQ_B.

Sounds great. Let me know once you are ready. Happy to chat in case you need you need more details on the library's code structure.

Hi all, was there meanwhile any progress on this front?

yes, the ID algorithm is now implemented for identification.
https://microsoft.github.io/dowhy/example_notebooks/identifying_effects_using_id_algorithm.html
However, estimation based on the probability expressions returned by ID algorithm is not yet supported.

Thank you. Any chance somebody also implemented the Adjustment Criterion by Shpitser (here, Definition 6)? This is a complete criterion for identification via covariate adjustment.

That's not implemented, but it will a simple exercise to implement it by extending the backdoor criterion. Would you like to implement it @gianlucadetommaso ? I've added an issue #402


Issue with building ios version

Here the error log:
0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'run', 'build' ]
2 info using<EMAIL_ADDRESS>3 info using<EMAIL_ADDRESS>4 verbose run-script [ 'prebuild', 'build', 'postbuild' ]
5 info lifecycle test2@~prebuild: test2@
6 silly lifecycle test2@~prebuild: no script for prebuild, continuing
7 info lifecycle test2@~build: test2@
8 verbose lifecycle test2@~build: unsafe-perm in lifecycle true
9 verbose lifecycle test2@~build: PATH: /usr/local/lib/node_modules/npm/bin/node-gyp-bin:/Users/TimKrajewski/TimsDrive/TimCollegeDrive/CSC498Capstone/Test2/node_modules/.bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/git/bin
10 verbose lifecycle test2@~build: CWD: /Users/TimKrajewski/TimsDrive/TimCollegeDrive/CSC498Capstone/Test2
11 silly lifecycle test2@~build: Args: [ '-c', 'ionic-app-scripts build' ]
12 silly lifecycle test2@~build: Returned: code: 1  signal: null
13 info lifecycle test2@~build: Failed to exec build script
14 verbose stack Error: test2@ build: ionic-app-scripts build
14 verbose stack Exit status 1
14 verbose stack     at EventEmitter. (/usr/local/lib/node_modules/npm/lib/utils/lifecycle.js:255:16)
14 verbose stack     at emitTwo (events.js:87:13)
14 verbose stack     at EventEmitter.emit (events.js:172:7)
14 verbose stack     at ChildProcess. (/usr/local/lib/node_modules/npm/lib/utils/spawn.js:40:14)
14 verbose stack     at emitTwo (events.js:87:13)
14 verbose stack     at ChildProcess.emit (events.js:172:7)
14 verbose stack     at maybeClose (internal/child_process.js:829:16)
14 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
15 verbose pkgid test2@
16 verbose cwd /Users/TimKrajewski/TimsDrive/TimCollegeDrive/CSC498Capstone/Test2
17 error Darwin 16.0.0
18 error argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "build"
19 error node v4.5.0
20 error npm  v3.10.8
21 error code ELIFECYCLE
22 error test2@ build: ionic-app-scripts build
22 error Exit status 1
23 error Failed at the test2@ build script 'ionic-app-scripts build'.
23 error Make sure you have the latest version of node.js and npm installed.
23 error If you do, this is most likely a problem with the test2 package,
23 error not with npm itself.
23 error Tell the author that this fails on your system:
23 error     ionic-app-scripts build
23 error You can get information on how to open an issue for this project with:
23 error     npm bugs test2
23 error Or if that isn't available, you can get their info via:
23 error     npm owner ls test2
23 error There is likely additional logging output above.
24 verbose exit [ 1, true ]
I had to change the permissions of the hooks folder with chmod +x hooks/after_prepare/010_add_platform_class.js of the app just to run the command then it still failed

just started getting a similar issue.  my app was building last night, going to try rolling back some changes in my app to see if it helps.  ionic serve runs fine and my app functions.

So, the actual error didn't show up in the npm log file.
I scrolled up a bit more and saw that the ngc process was failing, and complaining that I had something set to private
ngc: Error: Error at /Users/USER/Sites/enterprise-demo/.tmp/pages/dashboard/dashboard.ngfactory.ts:1919:42: Property 'pagesService' is private and only accessible within class 'DashboardPage'.

I set that to public and I'm good to go again.

This issue was moved to driftyco/ionic-app-scripts#257


[FLINK-12846][table-common] Carry primary key information in TableSchema

What is the purpose of the change
The primary key is a standard meta information in SQL. And they are important information for optimization, for example, AggregateRemove, AggregateReduceGrouping and state layout optimization for TopN and Join.
So in this PR, we extend TableSchema to carry more information about primary key.
Bridge the primary key of TableSchema to the planner metadata will be done in other PRs.
Brief change log

Add primary key to TableSchema with some Javadocs.

Verifying this change

Adds some tests in TableSchemaTest to check the primary key.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (not documented)

cc @twalthr @dawidwys

I dropped unique key information, because we only need to support primary key for TPC-DS optimizations.

Thanks @dawidwys for the reviewing. I agree with that the concepts of primary key on source/sink should be discussed and accepted publicly. But the background of this issue is that we would like to support more optimizations for TPC-DS where the sources have primary key information in Hive metastore. So we would like to only expose the primary key interface to CatalogTable, to let the Hive catalog pass the primary key information which is defined by Hive DDL to Flink. If a primary key is defined on Flink DDL or TableSource#getTableSchema or TableSink#getTableSchema, an unsupported exception will be thrown. In this way, I think the primary key concepts is simple and clear which is similar to Hive's definition.
But I'm not sure whether this still requires a FLIP. What do you think?

Hi @wuchong.
First of all thank you for outlining the original purpose of this change. I am a bit hesitant though to introduce a new ground concept to a core part of the Table ecosystem without a good understanding of it. That's why I would appreciate an agreement within community. I do want to emphasize I definitely want to help with finding a solution that we can introduce in 1.10 that can improve the TPC-DS performance.
I did a bit of research on the PRIMARY & UNIQUE keys. I have one concern regarding the semantics of a PRIMARY KEY. What I found out is that according to SQL standard & most of RDBM systems, PRIMARY KEY constraint enforces NOT NULL constraint on all columns that are part of the PRIMARY KEY. That's not the case for UNIQUE keys, but there it is assumed that null values are not equal regarding that constraint (you may have multiple rows having a null unique key). Unfortunately this is not reflected in Hive. In hive (at least pre 3.0) there was/is no support of NOT NULL types. Therefore PRIMARY KEY in hive is actually not a PRIMARY KEY in a SQL standard sense. I think just for that issue it would be good to bring it to the ML.


Phase 1 of improving controller extensibility

Overview

Adds definitions for each input source type that didn't already have one, which contain MixedRealityInputActionMappings that define specific input axes, like buttons or touchpads, for specific hardware (including hands). This allows these to be defined in one place, instead of defining them in each controller class per-platform/API.

This begins the transition away from linking controllers by the SupportedControllerType enum and instead linking them based on the definition they support. In an upcoming phase, the profile will be updated to serialize definitions instead of the existing controller classes which exist per-platform.

Splits out the existing MixedRealityInteractionMapping class into MixedRealityInputActionMapping, which is platform-agnostic and simply defines the capabilities of a physical device, and MixedRealityInteractionMappingLegacyInput, which contains the rest of the information about KeyCodes and input axes that are specific to Unity's legacy input APIs.

This will allow us to serialize only the platform-agnostic data into the profile, while allowing specific classes to define the APIs they call and how they map data to the physical buttons. We won't need to serialize legacy axes into the profile anymore.

Updates the way controllers provide and use their definition. Previously, they were local to a controller class, but now the base class is provided it via the constructor (composition over inheritance!) and uses it via the interface.

 Just a reminder that there are some breaking changes that we might want to document in the release note.

Which breaking changes do you see? I tried to make it as non-breaking as possible.

Overall looks good to me! Just a reminder that there are some breaking changes that we might want to document in the release note.

Which breaking changes do you see? I tried to make it as non-breaking as possible.

/azp run

/azp run

Overall looks good to me! Just a reminder that there are some breaking changes that we might want to document in the release note.

@MaxWang-MS Which breaking changes do you see? I tried to make it as non-breaking as possible.

Just double checked and was not able to spot any breaking change... I must have misread something previously.

Overall looks good to me! Just a reminder that there are some breaking changes that we might want to document in the release note.

@MaxWang-MS Which breaking changes do you see? I tried to make it as non-breaking as possible.

Just double checked and was not able to spot any breaking change...


TLS issues on ubuntu 20 focal

don't have reproduction steps ... it works on a plain focal, but we have a ruby build that adds jmalloc and then crashes with this during packing, hoping there is some obvious "you need to set xyz" fix 🤞
(bionic works fine for some reason)
#14 0.229  INFO exe: loading location=/lib/aarch64-linux-gnu/librt.so.1
#14 0.229  WARN exe: interpreter could not be found. static or compressed executable?
#14 0.229  WARN exe: requesting dynamic libraries of the executable without the interpreter
#14 0.230 error: Unable to lookup shared library: /lib/aarch64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block
#14 ERROR: executor failed running [/bin/bash -xc /usr/bin/magicpak -v $(which ruby) /magicpak]: exit code: 1

Assuming the same cause as https://github.com/jemalloc/jemalloc/issues/937, I was able to reproduce is as follows:
https://gist.github.com/coord-e/6384621a1916b0c36ce3f906edee2d3c
I think my guess is not that far off since changing FROM to ubuntu:bionic passes the build...

So adding --disable-initial-exec-tls to ./configure parameters of jemalloc could at least solves the problem.
From magicpak's side, I confirmed that the error goes away by loading jemalloc at the startup of the resolver (by -Wl,--no-as-needed -ljemalloc in compilation, but just the modification of DT_NEEDED should be sufficient) and using RTLD_NOLOAD flag for dlopen(3) to obtain the handler in the resolver. Nevertheless, I need to consider a little more whether this will work in any case...

awesome, I'll give that a try 🤞

FYI tried these and they did not work
LDFLAGS="-Wl,--no-as-needed" ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --with-jemalloc --enable-dtrace --disable-install-doc --enable-shared --disable-initial-exec-tls

LDFLAGS="-Wl,--no-as-needed -ljemalloc" ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --with-jemalloc --enable-dtrace --disable-install-doc --enable-shared --disable-initial-exec-tls

Sorry for the confusing wording, -Wl,--no-as-needed -ljemalloc is about the implementation of magicpak, and --disable-initial-exec-tls should be sufficient to be passed to ./configure.
If it doesn't work, I will need more time to figure it out. I'll try building jemalloc on my side.

trying to build something reproducible now 🤞

FROM ubuntu:focal as ruby

ARG ARCH

ENV DEBIAN_FRONTEND=noninteractive \
    RUBY_VERSION=3.1.2 \
    GEM_HOME=/usr/bundle \
    BUNDLE_PATH=/usr/bundle \
    BUNDLE_BIN=/usr/bundle/bin \
    BUNDLE_SILENCE_ROOT_WARNING=1 \
    BUNDLE_APP_CONFIG=/usr/bundle \
    PATH=/usr/bundle/bin:$PATH

RUN apt-get update && \
      # Compile and install ruby
      apt-get -y install libffi-dev libgdbm-dev libssl-dev libyaml-dev \
      procps zlib1g-dev libjemalloc-dev libgmp-dev libsasl2-dev libreadline-dev \
      build-essential libbz2-dev libncurses-dev libxml2-dev libxslt-dev xz-utils \
      libtool curl systemtap-sdt-dev && \
      curl -sSLo /tmp/ruby.tar.xz \
      https://cache.ruby-lang.org/pub/ruby/${RUBY_VERSION%.*}/ruby-${RUBY_VERSION}.tar.xz && \
      tar -C /tmp -xJf /tmp/ruby.tar.xz && \
      cd /tmp/ruby-${RUBY_VERSION} && \
      LDFLAGS="-Wl,--no-as-needed" ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --with-jemalloc --enable-dtrace --disable-install-doc --enable-shared --disable-initial-exec-tls && \
      sed -i 's/ENABLE_PATH_CHECK 1/ENABLE_PATH_CHECK 0/' file.c && \
      make && \
      make install && \
      cd /tmp && \
      # Cleanup
      rm -rf /tmp/ruby-${RUBY_VERSION} /tmp/ruby.tar.xz && \
      rm -rf /var/lib/apt/lists/* && \
      echo "install: --no-document" >> /etc/gemrc && \
      echo "update: --no-document" >> /etc/gemrc && \
      mkdir -p "$GEM_HOME" "$BUNDLE_BIN" && \
      chmod 777 "$GEM_HOME" "$BUNDLE_BIN" && \
      # ruby 2.6+ does not ship with `irb`
      # pinning reline and irb due to https://github.com/ruby/reline/issues/95
      [ $(echo $RUBY_VERSION | cut -f2 -d.) -lt 6 ] || gem install reline:0.0.7 irb:1.2.5 cgi:0.3.1 && \
      rm -rf /root/.gem

# pack up the environment
# note: without includes http requests fail see https://github.com/coord-e/magicpak/issues/12
ADD https://github.com/coord-e/magicpak/releases/download/v1.3.2/magicpak-${ARCH}-unknown-linux-musl /usr/bin/magicpak
RUN chmod +x /usr/bin/magicpak
RUN /usr/bin/magicpak --verbose --include "/lib/${ARCH}-linux-gnu/libnss_*" --include "/lib/${ARCH}-linux-gnu/libresolv*" $(which ruby) /magicpak

produces
docker build --progress plain --build-arg ARCH=aarch64 .
#10 0.226 error: Unable to lookup shared library: /lib/aarch64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block

I thought you were building jemalloc from source... when building jemalloc, ./configure with the --disable-initial-exec-tls flag, the error goes away.
https://gist.github.com/coord-e/8e8355bf3de42bcd24ddd5455a8759d3

Anyway, I've released v1.4.0 with --experimental-noload-resolver flag to resolve this issue without building jemalloc from source. Can you try it like the following?
https://gist.github.com/coord-e/016f2a879167a07ef6fa0700265d4161
...

# pack up the environment
# note: without includes http requests fail see https://github.com/coord-e/magicpak/issues/12
ADD https://github.com/coord-e/magicpak/releases/download/v1.4.0/magicpak-${ARCH}-unknown-linux-musl /usr/bin/magicpak
RUN chmod +x /usr/bin/magicpak
RUN /usr/bin/magicpak --verbose --include "/lib/${ARCH}-linux-gnu/libnss_*" --include "/lib/${ARCH}-linux-gnu/libresolv*" --experimental-noload-resolver $(which ruby) /magicpak

confirmed working with --experimental-noload-resolver

also works on ubuntu jammy 🎉

https://github.com/coord-e/magicpak/issues/19#issuecomment-1328318843 has --disable-initial-exec-tls and it still does not work still:
#9 0.231 error: Unable to lookup shared library: /lib/aarch64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block

also --disable-initial-exec-tls disables a performance-optimization so not having to use it is a plus (... unless --experimental-noload-resolver does the same under the hood)
... so all good afaik
maybe rename to remove experimental- and talk about downsides of --disable-initial-exec-tls in the readme


Fix to two-space convention in user manual

Fix to two-space convention in runtime/doc/usr_*.txt

@zeertzjq Fixed. Thanks for reviewing!

thanks!


Protocol processor test fixed

Hi @AndreasMager  could you describe better which problem fixes this PR?
Thanks
Andrea

Hi, I've changed the protocol codec part, passingo to Netty's 4.1 codec, this shouldn't be yet applicable after I merge.

No problem i'll rebase it tomorrow.

I like the new netty codec the test setup error is gone. I have rebase the check for a correct test setup


feat: refactor: Tracking overhaul

What
I've found another, better package for tracking to matomo. It's a fork of the old package with some more automation.
New features;

Page tracking
UserAgent (Operation system, maybe even phone type)
Automatic storing of visits, visit count, last visit etc (Removed our code)
Automatic storing of optout status

Also less code for more features is always good to have

Codecov Report

Merging #1913 (c800c07) into develop (2ea0da3) will decrease coverage by 0.86%.
The diff coverage is 15.42%.

@@            Coverage Diff             @@
##           develop   #1913      +/-   ##
==========================================
- Coverage     8.86%   8.00%   -0.87%
==========================================
  Files          161     175      +14
  Lines         6623    8098    +1475
==========================================
+ Hits           587     648      +61
- Misses        6036    7450    +1414

Impacted Files
Coverage Δ

...h_app/lib/cards/category_cards/abstract_cache.dart
0.00% <0.00%> (ø)

...p/lib/cards/category_cards/asset_cache_helper.dart
0.00% <0.00%> (ø)

...p/lib/cards/category_cards/raster_async_asset.dart
0.00% <0.00%> (ø)

...oth_app/lib/cards/category_cards/raster_cache.dart
0.00% <0.00%> (ø)

..._app/lib/cards/category_cards/svg_async_asset.dart
0.00% <0.00%> (ø)

...smooth_app/lib/cards/category_cards/svg_cache.dart
0.00% <0.00%> (ø)

...th_app/lib/cards/data_cards/image_upload_card.dart
0.00% <0.00%> (ø)

...es/smooth_app/lib/cards/data_cards/score_card.dart
4.54% <0.00%> (-0.22%)
:arrow_down:

...t_cards/knowledge_panels/knowledge_panel_card.dart
0.00% <0.00%> (ø)

...knowledge_panels/knowledge_panel_element_card.dart
0.00% <0.00%> (ø)

... and 130 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 814c224...c800c07. Read the comment docs.

@M123-dev small merge conflict

Happy birthday @M123-dev!


Stretched launch screen on Android

Hey aparajita,
I came here because of this thread:
https://github.com/ionic-team/capacitor/issues/1627
And I actually updated your plugin to capacitor 3 because I thought it would fix the issue with stretched android launchscreens. but I think it does not fix that, right?
Your config from the demo repo only shows the splash from the plugin not from native android. so there is this short amount of time, after clicking the app icon, where nothing happens or at least it looks like that to a user.
So my question is can I show a native image splash screen (which is not stretched) before the splash screen of the plugin?
thanks for you answer!
I will push my capacitor 3 code changes to my fork. It works as it is on ios and android. but i am not sure about some of the code changes and have not tested everything. maybe somebody will test and finish it. ( or i would if the plugin can address my issue)
all the best,
lukas

The issue describe in https://github.com/ionic-team/capacitor/issues/1627 happens because the native splash image is full screen (no status bar) and the capacitor image does have a status bar. So of course they are going to scale differently.
I was never able to reliably avoid that (working with Android is a nightmare), so I just show an empty native splash screen and wait for the capacitor splash screen to appear.

ok thanks fro the clarification. thats how i will do it as well. and i think cordova did that also.


MQTT - blank message

Hi
I'm trying to send a blank message via MQTT to AWS
I've tried both MQTTPUB and MQTTPUBRAW, but they both return an ERROR

AT+MQTTPUB=0,"$aws/things/Thing_Test02/shadow/get","",1,0
ERROR

AT+MQTTPUBRAW=0,"$aws/things/Thing_Test02/shadow/get",0,1,0
ERROR

Any advice would be appreciated
Thanks

Just a quick update.
For AWS a blank message can also be "{}" i.e. a blank json message!
so the following has worked for me

AT+MQTTPUBRAW=0,"$aws/things/Thing_Test02/shadow/get",2,1,0
{}
OK

+MQTTPUB:OK
+MQTTSUBRECV:0,"$aws/things/Thing_Test02/shadow/get/accepted", .....

Sorry if this was obvious... maybe it can help someone else.

Hello, Can you please provide me with the MQTT binaries as I'm having problems in generating them ?

@Wtpaul you could take a look at AT log port, it will print MQTT error code, please refer to: https://github.com/espressif/esp-at/blob/master/docs/ESP_AT_Commands_Set.md#MQTTErrCod,
if there is any MQTT command return ERROR.

@Maher9494 could you please describe the issue in detail?


[PDI-20003] Add a new extension point before the Carte webserver is started

This extension point allows the embedded Jetty Server for Carte to be further configured beyond the default Handler setup in WebServer.java.

:white_check_mark: Build finished in 51m 6s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core,engine

:ok_hand: All tests passed!
Tests run: 5329, Failures: 0, Skipped: 14    Test Results

:information_source:  This is an automatic message


added section3 in home page

@CodiumAI-Agent /review

PR Review

 ⏱ Estimated effort to review [1-5]
3, because the PR involves multiple changes across CSS and React components which require careful review to ensure consistency in styling and functionality. The addition of a new section and route also demands a thorough check to ensure integration with existing features.

 🧪 Relevant tests
No

 🔍 Possible issues
Possible Bug: The background-attachment: fixed; style in .section3 might cause performance issues on mobile devices. Fixed backgrounds can lead to repaint problems and janky scrolling.

 🔒 Security concerns
No

 Code feedback:
relevant fileclient/frontEnd/src/Styles/Navbar.css
suggestion

Consider adding a fallback background color for older browsers that do not support transparency. This can improve cross-browser compatibility. [important]

relevant linebackground-color: transparent;
relevant fileclient/frontEnd/src/Styles/home.css
suggestion

Since .section3 uses the same background settings as .page1, consider creating a common class for shared background properties to reduce redundancy and improve maintainability. [important]

relevant linebackground-image: url('../assets/background3.png');
relevant fileclient/frontEnd/src/App.jsx
suggestion

Ensure that the new route /signup added to App.jsx has corresponding authentication and authorization checks if necessary, to secure user data and access control. [important]

relevant line}/>
relevant fileclient/frontEnd/src/Components/Home.jsx
suggestion

Consider adding key props to the new sections added in Home.jsx to optimize React's rendering process by helping it identify which items have changed, are added, or are removed. [medium]

relevant line

 ✨ Review tool usage guide:
Overview:
The review tool scans the PR code changes, and generates a PR review which includes several types of feedbacks, such as possible PR issues, security threats and relevant test in the PR. More feedbacks can be added by configuring the tool.
The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.

When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...

With a configuration file, use the following template:

[pr_reviewer]
some_config1=...
some_config2=...

See the review usage page for a comprehensive guide on using this tool.


Viewshed (Location) sample

This sample demonstrates the configurable properties of viewshed analysis.

Thanks for the review. I believe I've addressed everything.
The Android deploy issue (class not found) appears to be a Visual Studio or Xamarin. Clearing the 'obj' folder in the Android project and re-building should cause the new sample to be deployed properly. Note that you may need to unload the project or close Visual Studio before the obj folder can be deleted.


Show standard expressions correctly

Currently, time structures and pitches print correct Haskell expressions.
We also want to do this with

The PartT, DynamicT and ArticulationT transformers
The actual standard types (Part, Dynamic and Articulation)

I.e. this should print as a Haskell expression using standard prelude:
>>> (mcatMaybes $ (set parts' violins $ legato $ mconcat [rest, c', ab, level ff db', e^*2, f, g, cresc () _f $ mconcat [ab^*3, bb]^/2, c'^*2])^/4) :: Score (PartT Part (ArticulationT Articulation (DynamicT Dynamics Pitch)))
[(0 <-> (1/4),PartT {getPartT = (Violin,ArticulationT {getArticulationT = ((Average {getAverage = []},Average {getAverage = [-1.0]}),DynamicT {getDynamicT = (Average {getAverage = []},c')})})})^.event,(0 <-> (1/4),PartT {getPartT = (Violin,ArticulationT {getArticulationT = ((Average {getAverage = []},Average {getAverage = [-1.0]}),DynamicT {getDynamicT = (Average {getAverage = []},ab)})})})^.event,(0 <-> (1/4),PartT {getPartT = (Violin,ArticulationT {getArticulationT = ((Average {getAverage = []},Average {getAverage = [-1.0]}),DynamicT {getDynamicT = (Average {getAverage = [2.5]},db')})})})^.event,(0 <-> (1/2),PartT {getPartT = (Violin,ArticulationT {getArticulationT = ((Average {getAverage = []},Average {getAverage = [-1.0]}),DynamicT {getDynamicT = (Average {getAverage = []},e)})})})^.event,(0 <-> (1/4),PartT {getPartT = (Violin,ArticulationT {getArticulationT = ((Average {getAverage = []},Average {getAverage = [-1.0]}),DynamicT {getDynamicT = (Average {getAverage = []},f)})})})^.event,(0 <-> (1/4),PartT {getPartT = (Violin,ArticulationT {getArticulationT = ((Average {getAverage = []},Average {getAverage = [-1.0]}),DynamicT {getDynamicT = (Average {getAverage = []},g)})})})^.event,(0 <-> (3/8),PartT {getPartT = (Violin,ArticulationT {getArticulationT = ((Average {getAverage = []},Average {getAverage = [-1.0]}),DynamicT {getDynamicT = (Average {getAverage = [1.5]},ab)})})})^.event,(0 <-> (1/8),PartT {getPartT = (Violin,ArticulationT {getArticulationT = ((Average {getAverage = []},Average {getAverage = [-1.0]}),DynamicT {getDynamicT = (Average {getAverage = [1.5]},bb)})})})^.event,(0 <-> (1/2),PartT {getPartT = (Violin,ArticulationT {getArticulationT = ((Average {getAverage = []},Average {getAverage = [-1.0]}),DynamicT {getDynamicT = (Average {getAverage = []},c')})})})^.event]^.score

This repo has been merged into: https://github.com/music-suite/music-suite
Closing :)


Implement Russian Draughts

Some interesting information here: https://sudonull.com/post/92610-Magic-bitboards-and-Russian-checkers
Russian Draughts is played on 8x10 which requires at least 40 bits * 3 to represent board state. The article above describes some efficient implementations.

[ ] Engine strategy
[ ] Move generation
[ ] Engine factory
[ ] 1D adapter strategy
[ ] Game factory
[ ] Evaluation function
[ ] Computer factory


update ci

Ⅰ. Describe what this PR does
Ⅱ. Does this pull request fix one issue?
Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

Can you describe the differences between v2 and v3 versions? And what good does it bring?

You can see that version 3 has solved a lot of dependency compatibility and loopholes see https://github.com/EPICLab/synectic/pull/1030


Error on opening this plugin (/public/emotion status=404)

I get these errors while loading this plugin.
INFO[01-17|08:44:07] Request Completed                        logger=context userId=1 orgId=1 uname=admin method=GET path=/api/plugins/marcusolsson-calendar-panel/markdown/help status=404 remote_addr=[::1] time_ms=11 size=68 referer="http://localhost:3000/plugins/marcusolsson-calendar-panel?page=overview"
INFO[01-17|08:44:11] Downloaded marcusolsson-calendar-panel v0.5.0 zip successfully logger=plugin.installer
INFO[01-17|08:44:11] Plugin registered                        logger=plugin.manager pluginId=marcusolsson-calendar-panel
INFO[01-17|08:44:11] Request Completed                        logger=context userId=1 orgId=1 uname=admin method=GET path=/public/emotion status=404 remote_addr=[::1] time_ms=21 size=37403 referer="http://localhost:3000/plugins/marcusolsson-calendar-panel?page=overview"

Should be fixed in 0.6.0 and 0.7.0. Let me know if not, and I'll reopen the issue!

I am this issue again on 0.7.0


Migration

Hi,
In the version 0.5.0, I've called on() method like this:
hubConnection?.on(method: method) { (args, typeConverter) in
     let json = try! typeConverter.convertFromWireType(obj: args, targetType: Any.self)
     callback(json)
}

How sould I call on() method in the latest version? Server's response is in json format like [String: Any].
Thank you!

See the read me - there is a special note about this there. The type needs
to be decodable. Let me know if you still can’t get it to work
On Tue, Dec 10, 2019 at 7:44 AM Misha Torosyan<EMAIL_ADDRESS>wrote:

Hi,
In the version 0.5.0, I've called on() method like this:
hubConnection?.on(method: method) { (args, typeConverter) in
let json = try! typeConverter.convertFromWireType(obj: args, targetType: Any.self)
callback(json)
}
How sould I call on() method in the latest version? Server's response is
in json format like [String: Any].
Thank you!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/moozzyk/SignalR-Client-Swift/issues/82?email_source=notifications&email_token=AAK7JJESVNZYB6AD5MH5QCDQX62MTA5CNFSM4JZASXF2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H7P3W2Q,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAK7JJHMEUHS7TQHEJ63EULQX62MTANCNFSM4JZASXFQ
.

Considering the question answered.


Migrate shcore functions to 0.3

This currently fails because of issues with shcore not being exposed as a library. @retep998 seems to understand the complete issue, so this PR is pending resolving that library.

I've added the rest of the import libraries for MinGW targets: https://github.com/retep998/winapi-rs/commit/f628c80106e97516e8031880995727b7a14d6615

Then let's run it again and see what we get!

Hot-diggity-damn it worked!

I ran it without rebasing and it worked, which is interesting. This one is properly rebased, but it should still work if the last one did.

I believe on PRs appveyor actually tests the result of merging the PR onto the target branch.

Interesting, didn't know that! I've used bors and homu to solve that problem, didn't realize it.
So does this look good to merge? This is a relatively small one here!


ajuste funcao youWin

copilot:all all the content, in one go

copilot:all all the content, in one go


Found end marker on line x without a start marker. Please manually edit

This was introduced by #2 Since the start marker has changed since the first release it is unable to find the Start marker.
To fix, edit your gitignore and remove all SSAutoGitIgnore related lines. Then rerun SSAutoGitIgnore, by either running
composer update

or
composer run-script post-update-cmd

I think this is resolved?

Indeed!


Add article on augmentation and zero-shot learning in NLP

Hi @NirantK,
I have added a few blog articles on data augmentation and zero-shot learning in NLP. Please review and merge them. Thanks.

I'm afraid that so many blog links would simply reduce the point of having a curated resource.
Can you please create a tag and a single link to the 4 blogs there?

@NirantK Updated. I've combined all of them to a single link with a "nlp" tag. Thanks.

Perfecto! Thanks!


fix: change pubKey input from Buffer to Uint8Array

This PR addresses an issue that occurred when running the deriveProof test case in the W3cCredentialService tests. Strangely enough this issue only occurred when running in a NodeJS environment (which uses node-bbs-signatures) instead of the WASM fallback.
Huge thanks to @blu3beri for helping debug this issue. This one wasn't easy to catch.
Signed-off-by: Karim<EMAIL_ADDRESS>

Codecov Report

Merging #935 (f844e22) into 0.3.0-pre (d045dd6) will increase coverage by 0.23%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##           0.3.0-pre     #935      +/-   ##
=============================================
+ Coverage      86.54%   86.77%   +0.23%
=============================================
  Files            512      512
  Lines          12478    12478
  Branches        2104     2104
=============================================
+ Hits           10799    10828      +29
+ Misses          1583     1555      -28
+ Partials          96       95       -1

Impacted Files
Coverage Δ

...c/signature-suites/bbs/BbsBlsSignatureProof2020.ts
81.74% <ø> (+3.96%)
:arrow_up:

...ckages/core/src/modules/vc/W3cCredentialService.ts
77.77% <0.00%> (+0.74%)
:arrow_up:

packages/core/src/modules/vc/jsonldUtil.ts
95.00% <0.00%> (+25.00%)
:arrow_up:

...src/modules/vc/signature-suites/bbs/deriveProof.ts
67.56% <0.00%> (+35.13%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d045dd6...f844e22. Read the comment docs.

Hmm after looking at it again, this looks like a different issue than the flaky tests as it's only failing on BBS tests. Maybe a linux + node 18 issue?

Ok, I've added the environment variable in 49c35d4 which should hopefully make sure the tests succeed for all Node versions (waiting on CI to finish atm, wish me luck 😄). One thing I'd like to double check: in order to not have to overhaul the entire workflow structure, I've added the environment variable top level. This means that the tests for Node 14 and 16 (which did succeed using node-bbs-signatures) will also use the WASM build.
Please let me know if there are any objections regarding this.
@TimoGlastra, @blu3beri, would love to hear your thoughts on this.

Documenting would be good. In addition I think there's two approaches here (maybe more):

set the environment variable by default before importing the bbs library
logging a warning if the node version is used and we're in node 17/18. The hard part is, how do we know which version is used? We could check if the node bbs signatures install succeeded (by trying to import it) and then logging the warning
logging a warning if the env variable is not set and we're in node 17/18. This will push the user to always set the env variable explicitly

Could you maybe open an issue in the node bbs signatures repo so the maintainers are aware of the issue?

Ahh great! Now we broke Node 14 again! 💯

Ahh great! Now we broke Node 14 again! 💯

Lol. Maybe we should take the first route and only set it for node 17 & 18?

Ahh great! Now we broke Node 14 again! 💯

Lol. Maybe we should take the first route and only set it for node 17 & 18?

Yeah... I tried running the tests on my machine (M1) locally with 14 and it succeeds. I will try to run it with Node 14 on an Ubuntu system now, just to get a better understanding where the problem lies, but I agree, that solution is probably better.
However, should we really set this environment variable just before importing the package? Isn't it cleaner to do this more 'top level' e.g. in the Agent class?

I will try to debug the issue tomorrow with the node library, however a temporary fix would be best for now.
WASM should not be that much slower so it's for sure not the end of the world.

Ahh great! Now we broke Node 14 again! 💯

Lol. Maybe we should take the first route and only set it for node 17 & 18?

Yeah... I tried running the tests on my machine (M1) locally with 14 and it succeeds. I will try to run it with Node 14 on an Ubuntu system now, just to get a better understanding where the problem lies, but I agree, that solution is probably better.
However, should we really set this environment variable just before importing the package? Isn't it cleaner to do this more 'top level', e.g. in the Agent class?

If we do it in the agent class the bbs library is probably already imported in which case setting the env variable doesn't matter anymore. So we need to make sure it happens before the import of bbs library

Even if we do this at the very top of the Agent class (before importing any of the containers)? I'm OK with anything tho, but this seems like a more evident place. I just want to avoid forgetting this in the future because it's being done in a deeply nested file.
Anyhow, will do it right before the import for now. We can change it in a follow up PR :)

Maybe the index.ts file is a better suited place? That's the entry point of the framework. I'm almost certain the agent class won't work

Ok, I ran some local tests. I ran it on Ubuntu hoping to reproduce the error, and on macOs (M1) for future reference. On both platforms all tests succeeded. This is going to be fun!
Ubuntu
Specs

Results for Node v14.19.3

Results for Node v16.16.0

Results for Node v17.9.1

Results for Node v18.5.0

macOs (M1)
Specs

Results for Node v14.20.0

Results for Node v16.16.0

Results for Node v17.9.1

Results for Node v18.5.0

Ok, I ran some local tests. I ran it on Ubuntu hoping to reproduce the error, and on macOs (M1) for future reference. On both platforms all tests succeeded. This is going to be fun!
Ubuntu
Specs

Results for Node v14.19.3

Results for Node v16.16.0

Results for Node v17.9.1

Results for Node v18.5.0

macOs (M1)
Specs

Results for Node v14.20.0

Results for Node v16.16.0

Results for Node v17.9.1

Results for Node v18.5.0

But isn't that expected with WASM? We're having issues with Node.JS. Or do I mistunderstand the issue?


Unfinished sentence

Sentence with only a subordinate clause
"Even though the underlying result set is different."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 40f6532a-085e-3d44-6af0-53da297b906d
Version Independent ID: 8e400cca-5227-074b-d6ac-7f4ff0558051
Content: Azure Monitor workbook chart visualizations - Azure Monitor
Content Source: articles/azure-monitor/visualize/workbooks-chart-visualizations.md
Service: azure-monitor
Sub-service: visualizations
GitHub Login: @rboucher
Microsoft Alias: abbyweisberg

Thanks for bringing this to our attention. We will investigate it further and update you shortly.

Hi @ciufudean,
Thank you for pointing this out to us. I've updated the page, and you should see the new text in there within a day or two.
Thank you for your continued dedication to our documentation.
Abby
#please-close


feat: update schema for feature improving security group lookups

My PR on the aws-cdk repo was reverted here: https://github.com/aws/aws-cdk/pull/31065
I'm adding these changes to this new repo so I can re-open that PR

My git state was all weird, so opened a new PR for #30 here @iliapolo


VSCode not asking whether I want tasks configured with "runOn": "folderOpen" to be executed with "task.allowAutomaticTasks": "auto".

Type: Bug

Without VSCode, create a folder named, for instance, "XXXX".
In that folder create another called .vscode
In .vscode create a file named tasks.json with this content:

{
  // See https://go.microsoft.com/fwlink/?LinkId=733558
  // for the documentation about the tasks.json format
  "version": "2.0.0",
  "tasks": [
    {
      "label": "echo",
      "type": "shell",
      "command": "echo Hello",
      "runOptions": {
        "runOn": "folderOpen"
      }
    }
  ]
}

Open VSCode
Make sure the setting "task.allowAutomaticTasks" is not set or is set to its defautl value "auto".
Open "XXXX" with VSCode.

VSCode will run "echo Hello" without asking you whether you want to run tasks configured with "runOn": "folderOpen" or not.
VS Code version: Code 1.70.2 (e4503b30fc78200f846c62cf8091b76ff5547662, 2022-08-16T05:35:13.448Z)
OS version: Windows_NT x64 10.0.19043
Modes:

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz (4 x 2712)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
7.43GB (2.23GB free)

Process Argv
--crash-reporter-id 78bf48ec-3ac2-4ab2-9796-20d8802d00af

Screen Reader
no

VM
0%

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vsdfh931:30280409
vshan820:30294714
vstes263:30335439
vscorecescf:30445987
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30410667
wslgetstarted:30449410
cppdebug:30492333
pylanb8912:30545647
vsclangdf:30486550
c4g48928:30535728
hb751961:30553087
dsvsc012:30540252
azure-dev_surveyone:30548225
i497e931:30553904

The described behavior is how it is expected to work. If you disagree, please explain what is expected and what is not in more detail. See also our issue reporting guidelines.
Happy Coding!

If I understood it correctly, the popup that supposedly asks you whether you want to run these tasks is not actually meant to appear ever because if the workspace is trusted it will be executed without asking you and if it is not trusted then it will just not exeute them instead of asking you. Doesn't it look kinda pointless to have in the description of the auto option of the task.allowAutomaticTasks configuration "Prompt for permission for each folder"? You are never prompted anyways.

I added this configuration and I am still not prompted anyways:
"security.workspace.trust.enabled": false

This happens because you have presumably opened a trusted workspace - which is now when we allow these to be run if the setting is "auto" for allow automatic task

@meganrogge No I did not. Please next time ask the user if he did X before assuming it and closing the issue.

i'll investigate this that shouldn't happen

Tasks will only run in trusted workspaces, so I will update the setting description to indicate that.
Also need to fix the prompt not happening per folder.


[help needed] custom rebalance function to subscribe to a single partition

I am working on a response-request feature based on partition. Let's say you have 20 partitions in a topic, and worker A subscribe to 0, worker B joins later and subscribe to 1, and so on.
How to write such rebalance assignment function that only takes care of its own member and its assignment, without rebalancing any existing workers?
Reopening #294

I'm not 100% sure, but would #373 do what you want to do?

I'm not 100% sure, but would #373 do what you want to do?

Yes, the java document describes the behaviour I want.

Also looking for this feature; is this possible with the current kafkajs apis?
In my case each consumer maintains state relative to a single partition, and I'm trying to assign each partition to a dedicated consumer (preferably without unnecessary rebalances).

You can technically work around it and sort of get the behaviour you want, but it's not idiomatic Kafka (manual assignments would be, see #373):

Create a consumer with a random groupId (uuid would work well).
Subscribe to the topic you want to consume. You'll be assigned all the partitions, as you'll be theo only consumer in the group.
Consume all messages through eachMessage or eachBatch, but skip all the messages that don't have the partition you wish to consume.

While this should work, it also creates a bunch of noise on your kafka cluster in form of lots  and lots of consumer groups. On top of that, you're actually fetching messages for all partitions, which depending on how many partitions the topic has, can cause significant overhead to the client and load on the cluster.

Thank you for the input @JaapRood,
Processing all partitions is not feasible in my case, so I came up with the follow:

Create a manual partition assigner that ensures a consumer group is only assigned a single fixed partition
Create a consumer with a fixed groupId for each partition (e.g., group-0, group-1, etc), configured with the manual partition assigner

At first glance, this implementation seems to be working nicely.
Apart from the need for a consumer group for each partition, do you see any glaring performance/maintenance issues with this strategy?

@dhessler nice one 👌 Reminds me of actually discussing something similar a while back, when I first ran into this (I didn't actually work around it at the time). Should be  the most efficient way of doing it at the moment, gets rid of all the extra fetching overhead.


[STORM-1573] Add batch support for MongoInsertBolt

https://issues.apache.org/jira/browse/STORM-1573

This PR can be merged to both master and 1.x branch.

Could you take a look at this please, @harshach ?

ping @harshach

@vesense
Could we apply standard batching behavior among external modules?
I mean, this PR doesn't flush when batch size is set to 100 but there're 99 tuples in several hours, while storm-hbase, storm-hive, and more flushes batch when tick tuple is received, and it sets tick tuple interval.

@HeartSaVioR Thanks for your reply. I planed to handle tick tuples in a follow up JIRA. But I'm OK for adding these changes in one JIRA, in order to provide standard batching behavior.
Furthermore, I think it's a great job if we can abstract batch processing to an common api among external modules.

@vesense
I'd really like to add that change in this issue, since it's not hard to address.
Btw, totally agreed about abstracting batch process to something common.

@HeartSaVioR OK. I will add the change later.

@HeartSaVioR Updated. also 1.x-branch: https://github.com/apache/storm/pull/1266

@HeartSaVioR Could you take a look?

+1

Could you rebase these three commits? After rebasing I'll merge.

@HeartSaVioR Done. rebased both master and 1.x-branch.

+1 Thanks for the patch.

Btw, totally agreed about abstracting batch process to something common.
@HeartSaVioR I have created a PR for abstracting batch process. see https://github.com/apache/storm/pull/1303

@HeartSaVioR Could you take a look at https://github.com/apache/storm/pull/1303 ?


fluentd config error: Unknown filter plugin 'sumo_fields'

FYI wrong plugin name. Should be "kubernetes_sumologic".
2016-12-15 05:51:58 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2016-12-15 05:51:58 +0000 [info]: starting fluentd-0.12.31
2016-12-15 05:51:58 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.26.2'
2016-12-15 05:51:58 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.8.2'
2016-12-15 05:51:58 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '0.0.3'
2016-12-15 05:51:58 +0000 [info]: gem 'fluentd' version '0.12.31'
2016-12-15 05:51:58 +0000 [info]: adding match pattern="containers.**.fluentd**" type="null"
2016-12-15 05:51:58 +0000 [info]: adding filter pattern="containers.**" type="kubernetes_metadata"
2016-12-15 05:51:59 +0000 [info]: adding filter pattern="containers.**" type="kubernetes_sumologic"
2016-12-15 05:51:59 +0000 [info]: adding filter pattern="docker.**" type="sumo_fields"
2016-12-15 05:51:59 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error="Unknown filter plugin 'sumo_fields'. Run 'gem search -rd fluent-plugin' to find plugins"
2016-12-15 05:51:59 +0000 [info]: process finished code=256
2016-12-15 05:51:59 +0000 [warn]: process died within 1 second. exit.

@rangedev oops, must of missed that.
I'll fix shortly.

@stevezau Any update here?  Would like to get this working today if possible.

@nelsonenzo I originally wrote a plugin to set the Sumo metadata fields for docker and kubernetes log files.
I intended to remove that plugin as there are generic plugins that can do the same thing. I was looking at the record transformer plugin. I've raised an issue trying to understand if it's possible to achieve what I need. https://github.com/sonots/fluent-plugin-record-reformer/issues/45
If i don't get a response i'll copy the plugin back in from the old repo

should be fixed now, i tested it in my env and it appears to work.
Open another issue if you encounter any problems.


Mentioning test login data

I just realized that the project (which gets deployed to http://localhost:8080/) doesn't state the test credentials whereas the demo site (deployed on http://demo.polymer-dashboard.com/pages/auth/auth.html) does. So I thought it's worth mentioning the test login in the README file. :)
Screenshot

Thanks bennyn!


Traefik Proxy + Mesh

Why traefik proxy is deployed alongside this helm chart? It does have its own helm chart and I quite do not understand the reason for having it here. Would be nice to have an option to disable it.

bump


"Output too long for Excel"

Is there an option to keep the columns in AllProteinGroups.tsv that have "output too long for Excel?"
Maybe we should make an additional file AllProteinGroups.forexcel.tsv that has these instead of the information.

It makes the column useless when trying to pull info out of the whole file. 😅

Okay, yes, it turns out there's a setting for this.
For future reference: Settings menu -> Close MetaMorpheus and open global settings -> change WriteExcelCompatibleTSVs = true to WriteExcelCompatibleTSVs = false


Mention usage of Message Authentication Codes

The security of this library is dubious anyway, but I believe that it is somewhat irresponsible to not mention Message Authentication. Otherwise, users will copy the insecure code in the README without realizing e.g. CBC without a MAC is trivially broken.

Might want to also mention that AES CTR is malleable (ciphertext can be easily modified and remain unnoticed if no authentication is done as mentioned above) and also if encrypted data belongs to a small domain of possible values, those values can be guessed if the same counter initialization value is reused for every encryption.


Signtool Issues with Device Guard Signing Method

This is similar to MSIX #137
Windows Build = 10.0.19042 N/A Build 19042
signtool.exe  Version = 10.0.1904.1
signtool.exe sign /debug /fd sha256 /dlib DgssLib.dll /dmdf c:\temp\MSIX\Lightaria-MSIX.json c:\temp\msix\notepad-plus-plus_<IP_ADDRESS>_x64_Lightaria.msix
SignTool Error: This file format cannot be signed because it is not recognized.
SignTool Error: An error occurred while attempting to sign: c:\temp\msix\notepad-plus-plus_<IP_ADDRESS>_x64_Lightaria.msix
Number of files successfully Signed: 0
Number of warnings: 0
Number of errors: 1
MSIX #137 indicates the issue is the wrong version of signtool.exe.  What is the correct version?

I have tested with insider version of signtool and have same results
I have installed signtool from insider kit 20140
signtool file version = 10.0.20140.1001 / 31 May 2020
and I get the same error message
c:\Program Files (x86)\Windows Kits\10\bin\10.0.20140.0\x64>.\signtool.exe sign /debug /fd sha256 /dlib DgssLib.dll /dmdf c:\temp\MSIX\Lightaria-MSIX.json c:\temp\MSIX\notepad-plus-plus_<IP_ADDRESS>_x64_Lightaria.msix
SignTool Error: This file format cannot be signed because it is not recognized.
SignTool Error: An error occurred while attempting to sign: c:\temp\MSIX\notepad-plus-plus_<IP_ADDRESS>_x64_Lightaria.msix
Number of files successfully Signed: 0
Number of warnings: 0
Number of errors: 1

Discussion moved to https://github.com/microsoftDocs/msix-docs/issues/137


setup.py: Provide instructions to install cmake

If cmake_path is not found, either cmake is not installed
or cmake is not in the PATH. Hence, we also give instructions
on how to install cmake if the path is not found.
Fixes https://github.com/davisking/dlib/issues/132

@jayvdb If it refuses to run on non-linux platforms then we definitely can't require it.

They have set it up so it can be required on any platform, but cant be imported on any platform.
We can work around that, by only importing after it has been confirmed that the platform is linux (of unknown flavour).

Hey, so i've updated the code to use distro.
Also, Just incase (if the pip version is too old to support setup_requires, or whatever reason) I've made "distro" optional. i.e. If distro is not found, it just gives the link to the cmake installation instructions page.
I've never used setup_requires before, so if @jayvdb could check it's the right way to use it - that'd be great.

It looks right, but I'm not comfortable making that call on sight only.  I suggest checking a few older versions of setuptools and, separately, older versions of pip, doing pip install -e . and pip install ..

See https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2473 ; it appears that setup_requires may have some quirks.  Worth check to see whether they crop up on this project.  Possibly a better option is to fetch https://raw.githubusercontent.com/nir0s/distro/master/distro.py & compile it into a module on the fly only if it is needed when cmake wasnt found.

I'm beginning to thing it may be a better idea to make a page in http://dlib.net/ which explains in detail about what to do in each platform and just show that link.
The reason for this is that, distros change things, for example yum -> dnf and now it seems Ubuntu is changing apt-get -> apt ? it makes it easy to change these instructions (It does not depend on the dlib release cycle). Rather than keep track of these in code, it would be easier to keep track of this in documentation.
@davisking @jayvdb Opinions ?

Maybe distro.py could also hold code for determining the packaging manager... ?

I don't think asking distro to do that is a good idea, because even if they detect the package manager, the rest of the command to install the package manager would be different. In our case above, yum, dnf, apt-get happen to use install, but others may have something else ... ?
Also, I've created https://github.com/nir0s/distro/issues/132 to directly ask the guys at distro what they think is a good way to use distro in setup.py.

I would put the message in the code since the sort of users that need to be
told things like the name of their package manager generally won't read
documentation.  I very frequently get questions like "I read page X but
still can't do Y.".  Then I go look at page X and their question is
answered directly at the top of the page.  You then copy and paste the text
to the user and they go "oh, finally, thank you!".  Obviously they didn't
read the documentation at all.

@davisking Point noted :D
I've updated the PR with a method which installs distro with pip if the package is not already installed. I thought this would be a better way than downloading the distro.py file directly.

Does that require setup python.py install to be run as root?

If 'python setup.py install' is being run with system python, the pip will be installing to system directories => both will be root.
If 'python setup.py install' is run in a virtual environment, then dlib and distro will be installed in the virtualenv and won't need sudo.

Sounds good so long as commands like python setup.py install --user still
work.

One scenario which i think fails now is a normal user running the following on a system without cmake or distro
python setup.py build
or
python setup.py test
The pip invocation needs --user in that case.
Also I think ensurepip should be loaded.
(Sorry i am without a computer to test atm)

@jayvdb Yes, those would indeed fail.
I'm going to make another PR which checks the pip command's returnvalue and skip distro in such cases. It will simply give the URL in such cases rather than the recommended command.
I think users that use setup.py directly have a higher probability of reading the docs too :D So, they should probably not need this message

Ha, maybe :)