jax and xarray integration for automatic differentiation?

I've been wondering if there has been any recent progress in integrating jax and xarray, specifically for automatic differentiation. For context, we have a simulation project that relies on xarray for our simulation output data but recently added jax support so users can automatically differentiate through these simulations. To make this work, we added code to emulate xr.DataArray functionality but with jax internals. However, this approach has been a headache to maintain and extend. It would be amazing if xarray had native support for gradient tracking in jax.
As an example, the code snippet below multiplies a Jax-traced value by an xarray.DataArray, does an interpolation, and then a jax-traved operation. It would be great if we could differentiate through this. The forward pass works, but the backwards pass gives a TracerArrayConversionError.
I've tried many other workarounds based on issues, such as this and some other discussions eg but without any luck. Are any updates on the status of this, whether it would be possible eventually, or suggestions for possible workarounds? Any discussion or pointers towards a good approach to this are really appreciated.
@shoyer
import numpy as np
import jax
import jax.numpy as jnp
import xarray as xr

shape = (3, 4, 5)
values = np.random.random(shape)
coords = {dim: np.arange(length) for dim, length in zip('xyz', shape)}
xarr = xr.DataArray(values, coords=coords)

def f(x):
    xarr_multiplied = x * xarr
    val = xarr_multiplied.interp(x=1, y=1, z=1)
    return jnp.sqrt(val.values)

f(1.0)
jax.grad(f)(1.0)

---------------------------------------------------------------------------
TracerArrayConversionError                Traceback (most recent call last)
Cell In[30], line 17
     14     return jnp.sqrt(val.values)
     16 f(1.0)
---> 17 jax.grad(f)(1.0)

    [... skipping hidden 10 frame]

Cell In[30], line 12, in f(x)
     11 def f(x):
---> 12     xarr_multiplied = x * xarr
     13     val = xarr_multiplied.interp(x=1, y=1, z=1)
     14     return jnp.sqrt(val.values)

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/_typed_ops.py:282, in DataArrayOpsMixin.__rmul__(self, other)
    281 def __rmul__(self, other):
--> 282     return self._binary_op(other, operator.mul, reflexive=True)

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/dataarray.py:4622, in DataArray._binary_op(self, other, f, reflexive)
   4616 other_variable = getattr(other, "variable", other)
   4617 other_coords = getattr(other, "coords", None)
   4619 variable = (
   4620     f(self.variable, other_variable)
   4621     if not reflexive
-> 4622     else f(other_variable, self.variable)
   4623 )
   4624 coords, indexes = self.coords._merge_raw(other_coords, reflexive)
   4625 name = self._result_name(other)

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/_typed_ops.py:488, in VariableOpsMixin.__rmul__(self, other)
    487 def __rmul__(self, other):
--> 488     return self._binary_op(other, operator.mul, reflexive=True)

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/variable.py:2707, in Variable._binary_op(self, other, f, reflexive)
   2703 with np.errstate(all="ignore"):
   2704     new_data = (
   2705         f(self_data, other_data) if not reflexive else f(other_data, self_data)
   2706     )
-> 2707 result = Variable(dims, new_data, attrs=attrs)
   2708 return result

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/variable.py:366, in Variable.__init__(self, dims, data, attrs, encoding, fastpath)
    346 def __init__(self, dims, data, attrs=None, encoding=None, fastpath=False):
    347     """
    348     Parameters
    349     ----------
   (...)
    364         unrecognized encoding items.
    365     """
--> 366     self._data = as_compatible_data(data, fastpath=fastpath)
    367     self._dims = self._parse_dimensions(dims)
    368     self._attrs = None

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/variable.py:293, in as_compatible_data(data, fastpath)
    290     return data
    292 # validate whether the data is valid data types.
--> 293 data = np.asarray(data)
    295 if isinstance(data, np.ndarray) and data.dtype.kind in "OMm":
    296     data = _possibly_convert_objects(data)

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/jax/_src/core.py:605, in Tracer.__array__(self, *args, **kw)
    604 def __array__(self, *args, **kw):
--> 605   raise TracerArrayConversionError(self)

TracerArrayConversionError: The numpy.ndarray conversion method __array__() was called on traced array with shape float32[3,4,5].
See https://jax.readthedocs.io/en/latest/errors.html#jax.errors.TracerArrayConversionError

[x] Check for duplicate requests.
[x] Describe your goal, and if possible provide a code snippet with a motivating example.

Hi - I think I recall this kind of thing coming up before – I don't know of any effort to do the full integration of JAX and xarray that you have in mind. The problem is that xarray is fundamentally built on the assumption that its arrays are numpy arrays, and so for example np.asarray is used frequently in its implementations. For traced operations in JAX, np.asarray will return a TracerArrayConversionError, because traced values cannot be converted to NumPy arrays.
To move forward, either one of two things would have to happen:

xarray would have to loosen its assumptions about the internal array representation. One way this could happen is if xarray adopted the Python Array API standard. It looks like there is some thought about this, but it would be a very big project.
Somebody could write an entirely new xarray-like wrapper for JAX. I don't know of any projects like this (though I wouldn't be surprised if folks have experimented with it), but it would also be a very big project.

Short of a team of people undertaking one of those very big projects, I don't think there's any good way to do what you have in mind.

I think would be quite exciting!
I think the Python Array API standard would probably be the way to go. Xarray's support for the API standard is pretty close to complete, and most missing features would not be hard to add. Xarray in fact already supports wrapping many types of non-NumPy arrays so this supporting JAX arrays as well would not be a big lift.
To get Xarray objects working with JAX transforms like jax.grad, they need to be registered with tree_util. But I think that is also straightforward.
Deepmind's GraphCast project contains a bundled Xarray-JAX wrapper, which I think already does some verison of both of these (maybe in a non-ideal way):
https://github.com/deepmind/graphcast/blob/main/graphcast/xarray_jax.py

(Side note: for the Array API approach, we'd also have to land some version of #16099 to make JAX compliant)

CC @mjwillson who wrote the Xarray-JAX wrapper in GraphCast.

Thanks @shoyer ! I'll have to study that graph cast code, I tried something similar but never could get it working properly.

I played around a bit with this GraphCast wrapper. It worked for the intended use case of applying @jax.jit to functions mapping from DataArray -> DataArray.
Unfortunately for jax.grad() still seems to give a TracerArrayConversionError. It seems like it might be occurring on the VJP function for multiplying the jax-traced scalar by the DataArray.
It's pretty likely I'm doing something wrong here so if @mjwilson / @shoyer spots something wrong here let me know!
import numpy as np
import jax
import jax.numpy as jnp
import xarray as xr

shape = (3, 4, 5)
values = np.random.random(shape)
coords = {dim: np.arange(length).tolist() for dim, length in zip('xyz', shape)}
xarr_jax = DataArray(values, dims=('x', 'y', 'z'), coords=coords)

def f(x):
    val = x * xarr_jax
    val = val.interp(x=1, y=1, z=1)
    val = jnp.array(val.data)
    return jnp.sum(val)

f(1.0) # works
jax.grad(f)(1.0) # TracerArrayConversionError

---------------------------------------------------------------------------
TracerArrayConversionError                Traceback (most recent call last)
Cell In[16], line 18
     15     return jnp.sum(val)
     17 f(1.0) # works
---> 18 jax.grad(f)(1.0) # TracerArrayConversionError

    [... skipping hidden 10 frame]

Cell In[16], line 12, in f(x)
     11 def f(x):
---> 12     val = x * xarr_jax
     13     val = val.interp(x=1, y=1, z=1)
     14     val = jnp.array(val.data)

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/_typed_ops.py:282, in DataArrayOpsMixin.__rmul__(self, other)
    281 def __rmul__(self, other):
--> 282     return self._binary_op(other, operator.mul, reflexive=True)

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/dataarray.py:4622, in DataArray._binary_op(self, other, f, reflexive)
   4616 other_variable = getattr(other, "variable", other)
   4617 other_coords = getattr(other, "coords", None)
   4619 variable = (
   4620     f(self.variable, other_variable)
   4621     if not reflexive
-> 4622     else f(other_variable, self.variable)
   4623 )
   4624 coords, indexes = self.coords._merge_raw(other_coords, reflexive)
   4625 name = self._result_name(other)

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/_typed_ops.py:488, in VariableOpsMixin.__rmul__(self, other)
    487 def __rmul__(self, other):
--> 488     return self._binary_op(other, operator.mul, reflexive=True)

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/variable.py:2707, in Variable._binary_op(self, other, f, reflexive)
   2703 with np.errstate(all="ignore"):
   2704     new_data = (
   2705         f(self_data, other_data) if not reflexive else f(other_data, self_data)
   2706     )
-> 2707 result = Variable(dims, new_data, attrs=attrs)
   2708 return result

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/variable.py:366, in Variable.__init__(self, dims, data, attrs, encoding, fastpath)
    346 def __init__(self, dims, data, attrs=None, encoding=None, fastpath=False):
    347     """
    348     Parameters
    349     ----------
   (...)
    364         unrecognized encoding items.
    365     """
--> 366     self._data = as_compatible_data(data, fastpath=fastpath)
    367     self._dims = self._parse_dimensions(dims)
    368     self._attrs = None

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/xarray/core/variable.py:293, in as_compatible_data(data, fastpath)
    290     return data
    292 # validate whether the data is valid data types.
--> 293 data = np.asarray(data)
    295 if isinstance(data, np.ndarray) and data.dtype.kind in "OMm":
    296     data = _possibly_convert_objects(data)

File ~/.pyenv/versions/3.10.9/lib/python3.10/site-packages/jax/_src/core.py:605, in Tracer.__array__(self, *args, **kw)
    604 def __array__(self, *args, **kw):
--> 605   raise TracerArrayConversionError(self)

TracerArrayConversionError: The numpy.ndarray conversion method __array__() was called on traced array with shape float32[3,4,5].
See https://jax.readthedocs.io/en/latest/errors.html#jax.errors.TracerArrayConversionError

The error is happening because the gradient computation results in calling xarray's __rmul__, which attempts to cast the inputs to numpy arrays, because xarray is built on the assumption that all its buffers are numpy arrays. Casting a traced JAX array to a numpy array results in a TracerConversionError.
There's no way to fix this without changing how xarray is implemented.

xarray is built on the assumption that all its buffers are numpy arrays

This isn't true -- xarray supports a number of duck arrays. As soon as JAX implements __array_namespace__ from the array API, you'll be able to wrap JAX arrays directly into xarray objects.
If you use the GraphCast Xarray-JAX wrapper, you need to use its special constructors for DataArray/Dataset.

Oh, good to know! Progress on __array_namespace__ is in #16099, though it's been hampered by the fact that JAX arrays are immutable, and some corners of the Python array API assume mutability.

If you use the GraphCast Xarray-JAX wrapper, you need to use its special constructors for DataArray/Dataset.

Could you explain a bit more?

Is this something I could fix by building on top of the GraphCast wrapper? For example, defining a custom jax VJP for DataArray.__rmul__?
Or maybe something more subtle that I'm not doing in my example?
Is it something more internal to jax / xarray that can't be fixed right now?

@jakevdp Indeed, Xarray doesn't rely on the mutation APIs (unless a user tries to mutate an array)
@tylerflex I see, it looks like you were already using the GraphCast wrapper. I don't know exactly what's going on, then.

Hiya,
Firstly just to note that xarray_jax isn't something we're officially supporting outside the GraphCast project for now, as it does have some rough edges and is in part a bit of a stop-gap measure until JAX supports the new array protocol which will allow it to integrate better with xarray.
That said, about your example, you'll find the following very similar code works:
import numpy as np
import jax
import jax.numpy as jnp
import xarray as xr
from graphcast import xarray_jax

shape = (3, 4, 5)
values = jnp.asarray(np.random.random(shape))
coords = {dim: np.arange(length).tolist() for dim, length in zip('xyz', shape)}
xarr_jax = xarray_jax.DataArray(values, dims=('x', 'y', 'z'), coords=coords)

def f(x):
    val = x * xarr_jax
    val = xarray_jax.unwrap_data(val)
    return jnp.sum(val)

f(1.0)
jax.jit(f)(1.0)
jax.grad(f)(1.0)

Some issues in your code were:

It looks like xarray's interp method uses scipy to do the interpolation, so there's no way this is going to work with JAX (it will try to convert the JAX tracer to a numpy array and fail)
To access the underlying jax array use xarray_jax.unwrap_data -- jnp.array(val.data) isn't going to work as it'll try to go via a numpy array
It appears there is a corner case bug where multiplying a numpy-array-backed DataArray with a JAX tracer causes an error. To work around it you should create the xarray_jax.DataArray with a jax array for the data, or you could pass the DataArray in as an explicit argument to the function rather than just closing over it, that way the JIT process will ensure it gets converted to a JAX array under the hood.


Seventh commit

Added following scripts
Tests related to configuration
Tests related to DHCP configuration
Tests related to interface
Tests related to Loadbalancer Stats

We can't merge this in yet. I will send you the details later


Add missing types

[x] Added back types which were removed in 9708025fcafd3d5b02bb69e59939e2db23f83dcd and made minor improvements
[x] Bumped package version

Thanks! There were some merge conflicts, so I added the types manually and pushed them. Hence I will close this PR.


Enable wakeup for USB input devices which connected via USB Switcher/KVM

Fixes GH-894

11月 30 21:57:30 algae (udev-worker)[949]: 3-1.1.2:1.1: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:20 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.2/3-1.1.2:1.1/power/wakeup}="enabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[949]: 3-1.1.2:1.1: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:18 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.2/3-1.1.2:1.1/power/wakeup}="disabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[949]: 3-1.1.2:1.1: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:16 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.2/3-1.1.2:1.1/power/wakeup}="enabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[963]: 3-1.1.1:1.1: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:22 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.1/3-1.1.1:1.1/power/wakeup}="enabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[963]: 3-1.1.1:1.1: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:18 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.1/3-1.1.1:1.1/power/wakeup}="disabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[951]: 3-1.1.2:1.0: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:20 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.2/3-1.1.2:1.0/power/wakeup}="enabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[959]: 3-1.1.1:1.0: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:22 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.1/3-1.1.1:1.0/power/wakeup}="enabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[1009]: 3-1.1.1:1.2: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:22 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.1/3-1.1.1:1.2/power/wakeup}="enabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[963]: 3-1.1.1:1.1: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:16 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.1/3-1.1.1:1.1/power/wakeup}="enabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[951]: 3-1.1.2:1.0: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:18 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.2/3-1.1.2:1.0/power/wakeup}="disabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[959]: 3-1.1.1:1.0: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:18 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.1/3-1.1.1:1.0/power/wakeup}="disabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[1009]: 3-1.1.1:1.2: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:18 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.1/3-1.1.1:1.2/power/wakeup}="disabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[951]: 3-1.1.2:1.0: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:16 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.2/3-1.1.2:1.0/power/wakeup}="enabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[1009]: 3-1.1.1:1.2: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:16 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.1/3-1.1.1:1.2/power/wakeup}="enabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[959]: 3-1.1.1:1.0: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:16 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1.1/3-1.1.1:1.0/power/wakeup}="enabled", ignoring: No such file or directory
11月 30 21:57:30 algae (udev-worker)[965]: 3-1.1:1.0: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:18 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1:1.0/power/wakeup}="disabled", ignoring: No such file or directory
11月 30 21:57:30 algae kernel: snd_rn_pci_acp3x 0000:04:00.5: enabling device (0000 -> 0002)
11月 30 21:57:30 algae (udev-worker)[965]: 3-1.1:1.0: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:16 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.1/3-1.1:1.0/power/wakeup}="enabled", ignoring: No such file or directory
11月 30 21:57:30 algae kernel: tpm_tis MSFT0101:00: 2.0 TPM (device-id 0xFC, rev-id 1)
11月 30 21:57:30 algae (udev-worker)[1006]: 3-1:1.0: /nix/store/l7gq7dp4zx1qvg423j7vlsfc6kgsvd82-udev-rules/99-local.rules:16 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1:1.0/power/wakeup}="enabled", ignoring: No such file or directory
...
10月 28 16:06:09 algae (udev-worker)[994]: 3-1.2.2:1.1: /nix/store/0ygr7fb0l8rg7sqs33d9abfx7dci7zgp-udev-rules/99-local.rules:21 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.2/3-1.2.2/3-1.2.2:1.1/power/wakeup}, ignoring: No such file or directory
10月 28 16:06:09 algae (udev-worker)[983]: 3-1.2.1:1.0: /nix/store/0ygr7fb0l8rg7sqs33d9abfx7dci7zgp-udev-rules/99-local.rules:21 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.2/3-1.2.1/3-1.2.1:1.0/power/wakeup}, ignoring: No such file or directory
10月 28 16:06:09 algae (udev-worker)[967]: 3-1:1.0: /nix/store/0ygr7fb0l8rg7sqs33d9abfx7dci7zgp-udev-rules/99-local.rules:21 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1:1.0/power/wakeup}, ignoring: No such file or directory
10月 28 16:06:09 algae (udev-worker)[1013]: 3-1.2:1.0: /nix/store/0ygr7fb0l8rg7sqs33d9abfx7dci7zgp-udev-rules/99-local.rules:23 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.2/3-1.2:1.0/power/wakeup}, ignoring: No such file or directory
10月 28 16:06:09 algae (udev-worker)[1013]: 3-1.2:1.0: /nix/store/0ygr7fb0l8rg7sqs33d9abfx7dci7zgp-udev-rules/99-local.rules:21 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-1/3-1.2/3-1.2:1.0/power/wakeup}, ignoring: No such file or directory

much errors since this PR.


⚠ BeyondHD has degraded performance

In 3fbc560, BeyondHD ($BHD) experienced degraded performance:

HTTP code: 200
Response time: 19585 ms

Resolved: BeyondHD performance has improved in dca7572 after 12 minutes.


Make type check utilities

Make an expression tree to validate types and show errors
Add a function to check types to Function. The function is called with expression trees
Make examples of type check

fixes #1
For example, when 4th dimension of 1st argument of an input is expected to be 2, you can write like this:
def ceck_type_forward(self, in_types):
    in_types[0].shape[3].should_be(2)

And, actually it is 1, a user can get a error message like this:
Expect: in_types[0].shape[3] == 2
Actual: 1 != 2

In offline discussion, we decided to use this style:
def ceck_type_forward(self, in_types):
    type_check.expect(
        in_types[0].shape[3] == 2,
        in_types[0].shape[2] == 1,
    )

LGTM :+1:


Repo sync for protected CLA branch

The pull request is created from main637873022913746294 to main to fix git push error for protected CLA branch

@ghogen : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Task :app:stripDebugDebugSymbols FAILED & Task :app:generateDebugBuildConfig FAILED

Description
I ran the command "react-native run-android"
and the return is this:

Task :app:stripDebugDebugSymbols FAILED
Task :app:generateDebugBuildConfig FAILED

React Native version:
System:
OS: Windows 10 10.0.18362
CPU: (4) x64 Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz
Memory: 1.30 GB / 7.82 GB
Binaries:
Node: 12.16.1 - C:\Program Files\nodejs\node.EXE
Yarn: 1.16.0 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
npm: 6.13.4 - C:\Program Files\nodejs\npm.CMD
IDEs:
Android Studio: Version  <IP_ADDRESS> AI-192.7<IP_ADDRESS>41897
npmPackages:
react: 16.9.0 => 16.9.0
react-native: 0.61.5 => 0.61.5
Java --version : 13.0.2
Steps To Reproduce
Expected Results
I must be able to compile my project.

Snack, code example, screenshot, or link to a repository:
"dependencies": {
"@react-native-community/async-storage": "^1.7.1",
"@react-native-community/cli-platform-android": "^3.0.3",
"@react-native-community/netinfo": "^5.5.1",
"axios": "^0.19.2",
"moment": "^2.24.0",
"numeral": "^2.0.6",
"qs": "^6.9.1",
"react": "16.9.0",
"react-native": "0.61.5",
"react-native-gesture-handler": "^1.5.2",
"react-native-image-picker": "^1.1.0",
"react-native-paper": "^3.4.0",
"react-native-reanimated": "^1.4.0",
"react-native-screens": "^2.0.0-alpha.22",
"react-native-snackbar": "^2.2.0",
"react-native-unimodules": "^0.7.0",
"react-native-vector-icons": "^6.6.0",
"react-native-web": "^0.12.0-rc.1",
"react-native-webview": "^8.0.3",
"react-native-youtube": "^2.0.0",
"react-navigation": "^4.0.10",
"react-navigation-stack": "^1.10.3",
"react-navigation-tabs": "^2.6.2",
"react-redux": "^7.1.3",
"redux": "^4.0.5",
"redux-persist": "^6.0.0"
},

It just happened to me too. I don't know how.
I had to downgrade gradle dependecy
classpath 'com.android.tools.build:gradle:3.4.2'

Execution failed for task ':app:stripDebugDebugSymbols'.
> No version of NDK matched the requested version 20.0.5594570. Versions available locally: 21.0.6113669

Open Android Studio -> Preferences -> Appearance & Behaviour -> System Settings -> Android SDK -> SDK Tools -> Show Package Details -> Select "NDK" -> select missing package -> Apply

It just happened to me too. I don't know how. I think from upgrading AndroidStudio
I had to downgrade gradle dependecy
classpath 'com.android.tools.build:gradle:3.4.2'

I had this issue when cloning a repo, i did a similar approach for example
the repo was using
  classpath('com.android.tools.build:gradle:3.6.2')
and i used this build instead
`classpath("com.android.tools.build:gradle:3.5.2")`

Thank you. @xlcrr
I have updated the android studio. While updating, NDK version has mismatched I think.
I installed the missed version and then it's working.

It just happened to me too. I don't know how. I think from upgrading AndroidStudio
I had to downgrade gradle dependecy
classpath 'com.android.tools.build:gradle:3.4.2'

Done this after a Gradle clean. (y)

For visual people

Show Package Details

@xlcrr  it helped thanku.

@xlcrr Thank you very much. You saved me!

check your react-native-image-picker configuration properly.

check your react-native-image-picker configuration properly.

Can explain what you mean? What configuration could be causing it?

check your react-native-image-picker configuration properly.

Can explain what you mean? What configuration could be causing it?

check your react-native-image-picker configuration properly.

Can you explain what you mean? What configuration could be causing it?

@godwin-c  check this:  https://github.com/ivpusic/react-native-image-crop-picker/issues/1416#issuecomment-700644075

check your react-native-image-picker configuration properly.

Can you explain what you mean? What configuration could be causing it?

@godwin-c  check this:  https://github.com/ivpusic/react-native-image-crop-picker/issues/1416#issuecomment-700644075

Bende böyle bir hata alıyorum NDK versiyonu olarak son sürümü yükledim ama hata devam ediyor

inside the build.gradle file there's a specified version given for ndk
ndkVersion = "20.1.5948944"
Make sure you are not having any other NDK version in your system.

Go inside your SDK location and find the directory called ndk then delete the content inside that directory. Then build the app again. Probably the ndk version has corrupted and Android Studio will replace it with the latest.
16622 silver badges77 bronze badges
This solution is for Unable to find bundled Java version error in your terminal. Its from August 9, 2021 Check the link for the solution stackoverflow.com/a/68703869/8555332 –

same problem, don't know how to deal it

I'm also facing same issue.  I'm using below build gradle version and NDK.
classpath 'com.android.tools.build:gradle:3.5.4'
ndkVersion "24.0.8215888"

classpath 'com.android.tools.build:gradle:3.5.4'

Closing as this issue is really old and the version of Android Gradle Plugin are far beyond (7.x). If you're having a similar problem please open another issue


Storage: Support MultiBlobTransfer

There is some skeleton code in the project to support this, but the MultiBlobTransfer is not functional. This would allow the user to upload or download a batch of blobs and track them as a single unit.

Closing this issue for now. When/if this feature is needed, we can file a new issue.


Updating Segmented Control

Platforms Impacted

[X] iOS
[ ] macOS

Description of changes
Updating Segmented Control to add support for having full screen equal width pill buttons, customizable colors with pill buttons, the primary and onBrand styles for pill buttons, and the ability to retrieve the view of a button at an index. Also updated the demo app to show the new styles with and without equal width
Verification
Before

After
 |
Pull request checklist
This PR has considered:

[X] Light and Dark appearances
[X] VoiceOver and Keyboard Accessibility
[ ] Internationalization and Right to Left layouts
[ ] Different resolutions (1x, 2x, 3x)
[X] Size classes and window sizes (iPhone vs iPad, notched devices, multitasking, different window sizes, etc)

Microsoft Reviewers: Open in CodeFlow

can you also add darkmode images and ipad images in the description of the PR? you can use markdown for table to make it look more organized with screenshots


[GDnative] add api version field to godot_gdnative_api_struct

Forget to add this in my previous PR ;p

While you're at it, can you add a void *next as the first field? It will allow us to extend the API at a later point without breaking backwards compatibility, a lá Vulkan

@karroffel there you go ;-)


Invalid XML declarations are slipping through the cracks.

Tested in opf and html files:
Passes validation:
<?xxml version="1.0" encoding="utf-8" standalone="yes"?>
<?xl version="1.0" encoding="utf-8" standalone="no"?>

Besten Dank für Ihre Nachricht.
Ich bin bis am Dienstag, dem 15. Mai 2018 abwesend und kann Ihre Nachricht erst ab dann wieder bearbeiten. Ihre Email wird nicht weitergeleitet.
Beste Grüsse
Anton Bolfing

Thank you for your message.
I'm out of office until Tuesday, May 15, 2018.  I will not respond to your email until then.
Best regards
Anton Bolfing

There's actually nothing technically wrong here. You can have processing instructions to start a file, and when the xml declaration is misnamed it just becomes a regular old processing instruction.
The XML declaration doesn't really do anything in EPUB, either, since version 1.0 and UTF-8 are the defaults, so the lack of a correct xml declaration isn't going to negatively affect the rendering.
I haven't heard of anyone releasing EPUBs with processing instructions still in their documents, though, so it probably would be harmless to warn if any are found in the document prolog (ignoring actual xml declarations, of course).

So XML parsers are supposed to ignore this?
On Fri, May 4, 2018, 8:31 PM Matt Garrish<EMAIL_ADDRESS>wrote:

There's actually nothing technically wrong here. You can have processing
instructions to start a file, and when the xml declaration is misnamed it
just becomes a regular old processing instruction.
The XML declaration doesn't really do anything in EPUB, either, since
version 1.0 and UTF-8 are the defaults, so the lack of a correct xml
declaration isn't going to negatively affect the rendering.
I haven't heard of anyone releasing EPUBs with processing instructions
still in their documents, though, so it probably would be harmless to warn
if any are found in the document prolog (ignoring actual xml declarations,
of course).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/IDPF/epubcheck/issues/842#issuecomment-386765770, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ANTgmrltp-kKfppRACFDUQz_F5GTrTHGks5tvPLDgaJpZM4TzDm-
.

Such a PI is certainly correct as a part of XML.  In my understanding, neither WHATWG HTML nor W3C HTML 5.2 disallows the use of such PIs.  So, such PIs are perfectly legitimate, as I see it.  WHATWG DOM allows PIs.  I thus conclude that your example (<?xl ...>) will appear in the DOM tree as a PI, but is not used for rendering.

neither WHATWG HTML nor W3C HTML 5.2 disallows the use of such PIs

No, but they're also generally useless. XML style sheet PIs haven't gone anywhere in epub, so other than relics from internal processes and applications what function would they serve? In all cases, there's no harm in telling someone they're an odd choice in the output.
But I would keep the warning at the info level if implemented, as they are legitimate.

So isn't this an enhancement and not a bug?

So isn't this an enhancement and not a bug?

I agree.

But I would keep the warning at the info level if implemented, as they are legitimate.

But there currently is no warning. So what to do? Create one? I don't think it's really necessary...

Yeah, I would be tempted to just do nothing on that one: EpubCheck is a validation tool, not a best practice tool.
That said, if several people upvote this issue, I guess it doesn't harm to implement that as an INFO-level. But if the usage is rare, I doubt it's worth implementing it?


Invalid resource error with PUT replace action

Hi there. I'm working on integrating this gem with an internal rails service, and running into an issue as I'm setting up tests to confirm the functionality we need.
I'm getting the error here: https://github.com/RIPAGlobal/scimitar/blob/main/app/controllers/scimitar/resources_controller.rb#L183, as the validation fails when the name field I have on my model is empty.
In my test i'm sending the request as so:
  describe '#update' do
    subject { put :replace, params: params }
    let(:group) { create(:group) }
    let(:params) do
      {
        id: group.id,
        schemas: ["urn:ietf:params:scim:schemas:core:2.0:PatchOp"],
        Operations: [{
          'op' => 'replace',
          'value' => {
            'id' => group.id,
            'displayName' => 'Test SCIMv2',
          },
        }],
      }
    end

    it 'gets the expected response' do
      subject
      expect(json_response).to eq({
        'id' => group.id,
        'schemas' => ["urn:ietf:params:scim:schemas:core:2.0:Group"],
        'displayName' => "Test SCIMv2",
      })
    end
  end

The group model contains a name field, and I set it up in the scim attributes map method to correspond name & displayName. However since the with_scim_resource doesn't map the parameters being passed to the constructor of the type returned by storage_class(), the validation then fails. Is there something I'm missing here? I can't seem to find anything in the mock example app you've provided.
The scim attributes map for reference:
def scim_attributes_map
  {
    id: :id,
    externalId: :scim_uuid,
    displayName: :name,
    members: [
      list: :scim_users_and_groups,
      using: {
        value: :id,
      },
      find_with: ->(scim_list_entry) {
        id = scim_list_entry['value']
        type = scim_list_entry['type'] || 'User' # Some online examples omit 'type' and believe 'User' will be assumed

        case type.downcase
        when 'user'
          User.find_by_id(id)
        when 'group'
          Group.find_by_id(id)
        else
          raise Scimitar::InvalidSyntaxError.new("Unrecognised type #{type.inspect}")
        end
      },
    ],
  }
end

Ah so I misunderstood how it's setting up the Scimitar::Group & not my own model. Though i think it's getting confused with me passing an example operation I got from Okta.
Based on the code for the resource controller, I'm thinking my test is invalid, but Okta specifies this could be PATCH or PUT depending on the configuration. https://developer.okta.com/docs/api/openapi/okta-scim/guides/scim-20/#update-a-specific-group-name


[FEATURE] Refactor error handling at OCPP incoming requests handling

Duplicate issue

[X] I've searched for any related issues and avoided creating a duplicate issue.

Description
No response
Attachments
No response

Commit https://github.com/SAP/e-mobility-charging-stations-simulator/commit/56d09fd7cdb49418af30cef6b9a47ea5007fb044 partially address that issue.


No registered 'MirrorPad' OpKernel for XLA_CPU_JIT devices

It seems that mirror padding is not supported with XLA. The message is triggered when running tfcompile. Any ideas on when this (or other ops like #11905) might be available?

nvm #11890


STPSource is not inherited from ISTPAPIResponseDecodable and ISTPSourceProtocol

Hi NAXAM.
I'm Ky. Mobile developer of mobile Xamarin. I had project using Stride payment sdk, so i use package Naxam.Stripe.iOS 11.3.0.
And i found that STPSource is not inherited from ISTPAPIResponseDecodable and ISTPSourceProtocol on dll.
It means: STPSource can't found property StripeID ( ISTPSourceProtocol has StripeID).
It's very serious, because we will can't access to some properties or has a correct type from STPSource to make a charge payment with backend-api.
So i had investigated in the project binding find something:
Code on Apidefinition.cs

[BaseType(typeof(NSObject))]
[DisableDefaultCtor]
interface STPSource : ISTPAPIResponseDecodable, ISTPSourceProtocol
...

I have solution for the issue.

[BaseType(typeof(NSObject))]
[DisableDefaultCtor]
interface STPSource : STPAPIResponseDecodable, STPSourceProtocol
...

So i hope it will make the binding of stripe is better.

Hi Ky,
Thank you very much for finding the issue.
Plz try to make the change and verify at your side then submit a pull request.
I am very happy to merge to the main code base and publish the update.
Cheers.

Commit 217def6eda92c434d0bea0304b7d8da928841e1f


fix: various bugs of cloudrun v2 module

take it or leave it, I don't mind but please test(a principle in software engineering called test) TF before publishing a module and making it available to public

/gcbrun

up up up


What is the type for generators that passing in injector

const injector = async (generators : which type?) => {
........
};

You won't need it cause it should be inferred automatically.
But if you want to manually type, it's GeneratorCallbak[]
You can also see that in the types.ts if you look closely.


Android application crashed after selecting deselecting an images

We are working on phonegap project and using this plugin. We have provided its reffernce in config.xml file of the project.

Steps to reproduce the issue.

Select the multiple pictures
Click on Ok button
Processing message is appeared on the screen
Click any where on screen .Processing message is getting hide.
Select /deselect some of the image.
Click on Ok button again
Application gets crashed, " Unfortunately ,application has stopped" message is coming on the screem

Kindly let us know how it can be solved.

I reproduce this bug also

https://github.com/divyasakpal/MultiImagePicker.git

+1


Use of non global tiles

We have a first proof of concept for the tiling of a large diskarray at PyramidScheme.Jl
Would it be possible to expose these tiles, so that they could be plotted by Tyler?
I can post an example of the pyramids later.

@SimonDanisch this is the same thing we need for using Tyler as a datashader.
Tiles that don't necessarily use the standard positions or zoom levels of XYZ tiles


Custom model on javascript

Hey!
I'm trying to use my own custom model on API javascript mediapipe.
I already saw this issues https://github.com/google/mediapipe/issues/2761 but no javascript solution has been provided.
Do you have any tips ? tuto ? documentation ? in order to do this, and make my tflite model working one mediapipe?

Hi @elYeodeol ,
Did  you get a chance to check this closed issue https://github.com/google/mediapipe/issues/1741

Hi,
thank you for your fast reply.
I read this issue and the issues related.
I'm facing this particulary error :

It seems that it doesn't really like my tflite conversion : i built a keras model and converted it with
converter.allow_custom_ops = True
converter.target_spec.supported_ops = [
  tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
  tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]

tflite_model = converter.convert()

with open('my_tflite_cnn.tflite', 'wb') as f:
    f.write(tflite_model) ```

Did i do the conversion right ? Maybe i am going in the wrong way and my tflite conversion is not really the issue in this...

Thank you in advance for your help

Custom models are not yet supported by the JS APIs.  So there's no guarantee that this process will work.
However, if you do want to try to use your own, then be aware that:

custom ops are often not supported by the graphs being used on web, so you'll probably need to remove this
SELECT_TF_OPS are not supported by our ML backends, so you'll definitely need to remove this in order to convert to .tflite format successfully
the model input parameters are hard-coded into the graph, so your model would need to match (looking at your error message, I believe the graph used here would be pose_landmark/pose_landmark_by_roi_gpu.pbtxt, so your model would need to accept 256x256 input)

Thank you a lot for your reply!
I have tried every ideas you have suggested me, but same error.
So there is no solution to use my custom model on the web except, maybe the android solution ?


Use properties for kamelets

do not use property with key / value but use properties

In Camel 4.6 we have improved the DSL to let beans be using properties for setting parameters
There is a few kamelets that use old style, such as
avro-deserialize-action.kamelet.yaml:        property:
avro-serialize-action.kamelet.yaml:        property:
protobuf-deserialize-action.kamelet.yaml:        property:
protobuf-serialize-action.kamelet.yaml:        property:


recursive clone for microns branch

You will either need to do
git clone --recursive git@github.com:openconnectome/open-connectome.git
or, if already cloned
git submodule update --init --recursive
in the root directory of the open-connectome repo.

Added.


[ERROR] interface conversion: interface {} is nil, not map[string]interface {}

Version: git (source)
Built with: go, as in:

$ git clone https://github.com/ByteDream/crunchyroll-go
$ cd crunchyroll-go/cmd/crunchyroll-go/
$ go build -o crunchy
$ sudo cp crunchy /usr/local/bin

OS: Ubuntu 20.04

Go: go1.17.6 linux/arm64

Platform: ARM64

What happens:

When running `crunchy login user passwd the binary throws the error:
=> interface conversion: interface {} is nil, not map[string]interface {}

I do not own any 64 bit arm processor to check / reproduce this, but on armv8 and amd64 it works fine. Could you paste the full error output (since the error can have multiple causes)?

Building from 1c42e52:
ubuntu@srv:~/crunchyroll-go/cmd/crunchyroll-go$ ./crunchy login user passwd -v
[debug] Executing `login` command with 2 arg(s)
[debug] Logging in via credentials
[err] interface conversion: interface {} is nil, not map[string]interface {}
[debug] goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x88
github.com/ByteDream/crunchyroll-go/cmd/crunchyroll-go/cmd.Execute.func1()
        /home/ubuntu/crunchyroll-go/cmd/crunchyroll-go/cmd/root.go:57 +0x134
panic({0x3455a0, 0x40004d21e0})
        /usr/local/go/src/runtime/panic.go:1038 +0x224
github.com/ByteDream/crunchyroll-go.LoginWithCredentials({0xfffffca6b6a3, 0x12}, {0xfffffca6b6b6, 0x7}, {0x0, 0x0}, 0x655d20)
        /home/ubuntu/crunchyroll-go/crunchyroll.go:70 +0x5b8
github.com/ByteDream/crunchyroll-go/cmd/crunchyroll-go/cmd.loginCredentials({0xfffffca6b6a3, 0x12}, {0xfffffca6b6b6, 0x7})
        /home/ubuntu/crunchyroll-go/cmd/crunchyroll-go/cmd/login.go:34 +0xb8
github.com/ByteDream/crunchyroll-go/cmd/crunchyroll-go/cmd.glob..func2(0x650860, {0x40000a3110, 0x2, 0x3})
        /home/ubuntu/crunchyroll-go/cmd/crunchyroll-go/cmd/login.go:22 +0xac
github.com/spf13/cobra.(*Command).execute(0x650860, {0x40000a30e0, 0x3, 0x3})
        /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:856 +0x678
github.com/spf13/cobra.(*Command).ExecuteC(0x650360)
        /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x424
github.com/spf13/cobra.(*Command).Execute(...)
        /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
github.com/ByteDream/crunchyroll-go/cmd/crunchyroll-go/cmd.Execute()
        /home/ubuntu/crunchyroll-go/cmd/crunchyroll-go/cmd/root.go:61 +0x60
main.main()
        /home/ubuntu/crunchyroll-go/cmd/crunchyroll-go/main.go:10 +0x20

It can be a pain for you to do so, but you can get an always free 4 cores + 24GB RAM ARM64 instance from Oracle Cloud for testing. I'm not sure how to properly debug this to provide more information as I've never touched any Go code, but let me know if there's anything else I could do to provide the necessary logs and such.
Thank you!

Does the same thing happen if you use a pre-compiled binary from the release?

Those wouldn't work as they are built only against AMD64:
ubuntu@srv:~$ file crunchy-v1.2.4_linux
crunchy-v1.2.4_linux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=QjT9KUa6DUJV1H8hZgq5/PXtNnZ6zPCboolZhdHpn/k02lxOGr3YUIeLti9E3I/cZggxkeRffe-0fNrsEpT, stripped

I suppose I could run it using qemu, but the performance would suffer a fair bit. Is there any other information I could provide to help solve this bug?

Oh lol forgot that.
I cannot track down the error, the build works on all of my machines (multiple architectures). As far as I can see, the error happens because a new session cannot be started. Could you execute curl https://api.crunchyroll.com/start_session.0.json?version=1.0&access_token=LNDJgOit5yaRIWN&device_type=com.crunchyroll.windows.desktop&device_id=Az2srGnChW65fuxYz2Xxl1GcZQgtGgI on the machine where the error happens and post the output here?

I had the same error.  I use a VPN.  Changing my connection cleared the error for me.

Closing due inactivity and no further information provision


What are your plans for components?

In the original raspberry-sharp libraries there were quite a few components that I don't find here. Additionally I have a couple of components myself that I wonder if I should include; The BME280 temperature/humidity sensor with extensive implementation and TSL2561 light sensor with a basic implementation.
What are your plans regarding components?
Thanks

I have many components ready to add. I did some positive tests this summer, so I planned to update the library this autumn and move nuget from beta to production.
If you  have some others, we can merge them.


Remove postToAMQ (post_to_amq) and fix typo in doc_type_amq

Fixes https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/214
Related to https://github.com/dmwm/WMCore/issues/11429
Status
ready
Description
Removed post_to_amq configuration parameter and local variable postToAMQ since they are not used in a code and fixed typo doc_to_amg to doc_to_amq which is used in MSPIleup configuration.
Is it backward compatible (if not, which system it affects?)
YES
Related PRs
<If it's a follow up work; or porting a fix from a different branch, please mention them here.>
External dependencies / deployment changes
<Does it require deployment changes? Does it rely on third-party libraries?>

squash is done.

Done, see https://github.com/dmwm/WMCore/pull/11574


Druid SQL - natively support mathematical functions ROUND(), RADIANS(), EXP(), LN(), POWER(), RAND()

Description
Currently these widely known mathematical operations are not officially implemented by Druid SQL:
Remote driver error: RuntimeException: org.apache.calcite.tools.ValidationException: org.apache.calcite.runtime.CalciteContextException: From line 1, column 18 to line 1, column 23:
No match found for function signature RAND() -> ValidationException: org.apache.calcite.runtime.CalciteContextException: From line 1, column 18 to line 1, column 23:
No match found for function signature RAND() -> CalciteContextException: From line 1, column 18 to line 1, column 23: No match found for function signature RAND() -> SqlValidatorException: No match found for function signature RAND()

Similar error for all.
These functions used to work in Druid version 0.10 when the useFallback option existed, calcite had a way to call these. However it looks like this parameter has been deprecated since 0.15: https://github.com/apache/incubator-druid/commit/c648775b5b9001dafd26192abf2a61f44bfc16d4 and https://github.com/apache/incubator-druid/pull/7567
The docs here claim that these functions do exist: https://druid.apache.org/docs/latest/querying/sql#numeric-functions
However I believe these are out of date, or had been written at an earlier time/version when useFallback was being used and they were callable.
Motivation
These would be useful for Business Intelligence applications connecting to Druid SQL clusters, and are very well known operations for any numeric data.

Hi @ntkawasaki, Druid does support all of those functions today except RAND. Most of them were added in version 0.15. (You can access old Druid docs at e.g. https://druid.apache.org/docs/0.14.0-incubating/querying/sql; note that the 0.14 docs don't include most of these functions.)
I raised an issue specifically for RAND: https://github.com/apache/incubator-druid/issues/8789, which is still missing today.
Btw, we deprecated and removed useFallback because it led to unscalable query plans in many cases. We found that often, queries with useFallback would pull all raw data into the Druid Broker and process it there, which was a bad approach for big datasets. This risk outweighed the benefits, and so we decided to implement more functionality in Druid's native query system instead of relying on SQL interpreter fallback.

Thanks @gianm
Made a Druid 0.15 database and its all good:

We can come up with a workaround for RAND() until thats native in Druid.


Broadcast cancel to other nodes if there is no local match

Summary of the changes / Why this improves CrateDB
To support setups with load-balancers. Otherwise cancel only has an
effect if it happens to reach the correct node, which has a 1 / <num_nodes> chance if using a load-balancer with round-robin.
Closes https://github.com/crate/crate/issues/13067
Checklist

[x] Added an entry in CHANGES.txt for user facing changes
[x] Updated documentation & sql_features table for user facing changes
[x] Touched code is covered by tests
[x] CLA is signed
[x] This does not contain breaking changes, or if it does:

It is released within a major release
It is recorded in CHANGES.txt
It was marked as deprecated in an earlier release if possible
You've thought about the consequences and other components are adapted
(E.g. AdminUI)

Will try another variant before merging this to see if we can encode clusterState version and node-index into a pid/secretKey. If we can squeeze that in, we could send the cancel to the correct node and ensure there are no collisions and false positive cancellations.

if we can encode clusterState version and node-index into a pid/secretKey. If we can squeeze that in, we could send the cancel to the correct node and ensure there are no collisions and false positive cancellations.

version is already a long and it would probably change too often without any node change.
We'd need at least 20-24 bits to encode node index and node-size. That would introduce a 4096 max-node limit and I think cut too much off of the session-id/secretKey space.
I'll go ahead with this version as is


Some blog posts are not properly rendered

Liquid is caching blog posts rendered in feed.xml and uses those cached templates in the blog itself. Passing a nil layout to the engine prevents that behaviour.

Nice work!!


No respose from the wingman chat

VS code version:
Version: 1.94.2 (user setup)
Commit: 384ff7382de624fb94dbaf6da11977bba1ecd427
Date: 2024-10-09T16:08:44.566Z
Electron: 30.5.1
ElectronBuildId: 10262041
Chromium: 124.0.6367.243
Node.js: 20.16.0
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22631
Download the models:
deepseek-coder:6.7b-base-q8_0
deepseek-coder:6.7b-instruct-q8_0
Have tested to post  a http request with a question to "http://<IP_ADDRESS>:11434//api/generate" and that work
Would be greaeful if you could help out here

Hi,
What are your machine specs? Also, two things to check:
1.) do you have a workspace or folder open?
2.) can you send over any logs from “Output” -> “Wingman” ?

Hi,
I have a Windows 11 pro with 32GB of ram and 2 tb hard drive.
I have tested both folder and workspace.
There are nothing in the output  window.
When I ask a question,  the Wingman  respond window start to pulses, but nothing else happens
here is my settings:

Hi, I have a Windows 11 pro with 32GB of ram and 2 tb hard drive. I have tested both folder and workspace.
There are nothing in the output window.
When I ask a question, the Wingman respond window start to pulses, but nothing else happens here is my settings:

What gpu do you have? Wingman output will log some timings around requests, but if you don’t have a gpu powerful enough to run inference then it may take a while.
There are a few things you can do such as running a q4 model in that case and reducing some settings under interaction settings.

NVIDIA RTX A1000 6GB Laptop GPU

Ok so this could be related to the performance of that GPU. Here is how chat works:
In the configuration settings - under Interaction Settings, you can lower the chat context window - this takes effect when you have files open in the editor. It will pull surrounding text from your cursor. In the settings, you can also lower chat max tokens.
if you have embedding enabled, it will use RAG to pull additional context when you use chat.
together they can add a considerable amount of tokens to the prompt which may be the issue.
another trick you can try is just open a file a highlight a small section of text/code. This will bypass pulling surrounding text so it focuses on the highlighted code.
also you only have 6gb of ram, the Deepseek 6.7b model in q8 will require more ram. You can downgrade to a q4 for more performance. But let me know if this helps.
in general performance is based on the amount of tokens passed to the LLM

Hi @RussellCanfield Russel,
Just to confirm,  the chat is now responding after your update.
BR
@OboMarvin


[bug 1216998] Move all l10n under fjord.l10n WIP

create a fjord.l10n app to hold all l10n stuff
move extract and merge commands to l10n app
move fjord/base/l10n.py to fjord/l10n/ext.py and clean up
add tests for collapse_whitespace and TowerI18nExtension
add basic tests for extract and merge that mostly test to see if the
commands run at all

Things left to do:

[ ] move settings to central place with sane defaults
[ ] reinstate some of the settings I removed because I didn't need them in fjord
[ ] look into what happened to the language activation code--do we need it anymore?

After that, I'll un-WIP this and land it. Then I'll pull it all into the tower-fork and additionally do the following:

[ ] write tests for extract command
[ ] write tests for merge command
[ ] write a compilemessages command--it's silly we all have bash scripts for this

I remember now. We ditched all the translation activation stuff because we switched to using django.po as the .po file. If we allowed for other filenames, then we'd need to override the activation code to allow for that. So... figuring out activation stuff--check!

I think I've finished up all the work I need to do in fjord and it's ready to be moved to the tower-fork.
One key part of the docs will be "how do i migrate from old tower to tower-fork?" I'm keeping that in mind as I do things.

Need to do a pass on the code, fix whatever that turns up and rewrite the collapse_whitespace code. Then this will be ready for review.

Scratch that. I decided to put the whole thing into a library. I'll do another PR to switch to the library when things solidify.
Closing this crazy out.


Left hand side of For...of cannot use type annotation, but tslint requires it

When using the TypeScript for...of loop, TSLint throws a typedef error expecting the first variable to have a type definition, however that is not allowed in TypeScript.
Sample:
let names: string[] = [...]
for (let name in names) {
    ...
}

TSLint wants name to have a typedef, however that is not allowed in TypeScript.

duplicate of #743


Charts[Network]: update configMaps for network node config files to use filename instead of removing the dot

see comments in: https://github.com/hashgraph/full-stack-testing/pull/797

:tada: This issue has been resolved in version 0.27.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Typings for built-in/core modifiers

I'm running into an issue with TypeScript typings, namely I don't know what to do in the function here:
	{
		name: 'arrow',
		options: {
			element: arrow,
			padding: ({popper, reference, placement}) => {
				// ...
			}
		}
	},

My compiler is screaming at me that the parameters aren't typed, I don't know what the type is, I know nothing about them except that two of them have a width property... I can barely work with this, it's going to take me a lot longer to figure out what is going on than it would have if these modifiers had been typed (so I could've done something like options: {...} as ArrowModifierOptions)
So here is my feature request for typing these, so future developers can be productive in TypeScript when they want to do super advanced things like, having the arrow point at a different element than the top level reference :/

Modifiers should already be typed, here's an example:
https://unpkg.com/browse/<EMAIL_ADDRESS>

Modifiers should already be typed, here's an example:
https://unpkg.com/browse/<EMAIL_ADDRESS>
Ah, the reason I don't have that must be because I only have react-popper. Even though @popperjs/core is installed, since it's not in my package.json I can't use types from it.
Adding @popperjs/core to my package.json allowed me to slap as ArrowModifier on the modifier item and that fixed the issue, thank you!


Error: Twit config must include consumer_key when using user auth.

After setting up the configuration in the .env file and running npm start I get the above error.

Hey @GoodFaithParadigm8
Twit config must include consumer_key when using user auth indicates there is an issue with how the config is set up, so are there any "" or '' around the API keys in the .env file?
closing this as it's a configuration issue and not an issue with the bot itself
I'll happily carry on the thread until you have it resolved though :+1:

Ended up building a bot using a different scheme. Love the 12 Factor App Methodology used in yours though. Knew raising an issue wasn't the best place to ask about the error I got but didn't think it would have gotten a response on Stack Exchange. I'll revisit the bot at a later time to see if I properly set up the configuration. Thanks!

for some reason I didn't get a notification for this issue and only noticed it when I logged in to log one myself 🙃

i got this problem and have all keys sat up properly ( no spaces or  any "" or '' around the API keys)

You have your .env file set up like in the example @mathias3?
TWITTER_CONSUMER_KEY=kjfjsd8kjfjsd8kjfjsd8
TWITTER_CONSUMER_SECRET=kjfjsd8kjfjsd8kjfjsd8kjfjsd8kjfjsd8kjfjsd8kjfjsd8kjfjsd8kjfjsd8
TWITTER_ACCESS_TOKEN=kjfjsd8kjfjsd8kjfjsd8kjfjsd8kjfjsd8kjfjsd8
TWITTER_ACCESS_TOKEN_SECRET=kjfjsd8kjfjsd8kjfjsd8kjfjsd8kjfjsd8kjfjsd8

And you have the rest of your query information set up??
QUERY_STRING=my super awesome query string!,google,android

RANDOM_REPLY=Hi @${screenName} thanks for the follow! What are you working on today?|@${screenName} thanks for following! What are you working on today?

RESULT_TYPE=mixed
TWITTER_LANG=en

TWITTER_RETWEET_RATE=.1
TWITTER_SEARCH_COUNT=20

Exactly:
TWITTER_CONSUMER_KEY=xYs05WxxxxxxxxxxxnEHRa
TWITTER_CONSUMER_SECRET=hiHikxaINsVbWxxxxxxxxxe844EvNHDAWqH
TWITTER_ACCESS_TOKEN=9588xxxxxxx9297-Gd8tZKnkiqyxxxxxxxNUXsYeUsPyYT
TWITTER_ACCESS_TOKEN_SECRET=xZ4e3y6MFnQxxxxxxxxxxxxxxxqBAf99nufq6hkOr5
QUERY_STRING=my super awesome query string!,google,android
RANDOM_REPLY=Hi @${screenName} thanks for the follow! What are you working on today?|@${screenName} thanks for following! What are you working on today?
RESULT_TYPE=mixed
TWITTER_LANG=en
TWITTER_RETWEET_RATE=.1
TWITTER_SEARCH_COUNT=20


RotWK users who install 2.02 after using 2.22 launcher can't play with RotWK users who install it all by hand

Apparently it's a known problem, but creating an issue for tracking
(Didn't test this myself, someone told me about it)

The order tells me that this is a patch 2.02 issue then and not a launcher issue.
Im just delivering the game as patch 2.01.
I also wont support 2.02 in any way since we are not allowed to feature it in the first place.
So no action will be done here for now.

This issue is obsolete and will be solved with the next update anyway ~closed


Set controller names in Demo pick and place task

To remove the following warnings when running the Demo Pick and Place task we need to specify the controllers the task should use.
[move_group-4] [moveit_task_constructor_visualization.execute_task_solution<PHONE_NUMBER>.106751682]: The trajectory of stage '4' from task '' does not have any controllers specified for trajectory execution. This might lead to unexpected controller selection.
[move_group-4] [moveit_task_constructor_visualization.execute_task_solution<PHONE_NUMBER>.106813314]: The trajectory of stage '5' from task '' does not have any controllers specified for trajectory execution. This might lead to unexpected controller selection.
[move_group-4] [moveit_task_constructor_visualization.execute_task_solution<PHONE_NUMBER>.106851123]: The trajectory of stage '7' from task '' does not have any controllers specified for trajectory execution. This might lead to unexpected controller selection.
[move_group-4] [moveit_task_constructor_visualization.execute_task_solution<PHONE_NUMBER>.106876960]: The trajectory of stage '11' from task '' does not have any controllers specified for trajectory execution. This might lead to unexpected controller selection.
[move_group-4] [moveit_task_constructor_visualization.execute_task_solution<PHONE_NUMBER>.106931256]: The trajectory of stage '14' from task '' does not have any controllers specified for trajectory execution. This might lead to unexpected controller selection.
[move_group-4] [moveit_task_constructor_visualization.execute_task_solution<PHONE_NUMBER>.106975585]: The trajectory of stage '16' from task '' does not have any controllers specified for trajectory execution. This might lead to unexpected controller selection.
[move_group-4] [moveit_task_constructor_visualization.execute_task_solution<PHONE_NUMBER>.107018325]: The trajectory of stage '18' from task '' does not have any controllers specified for trajectory execution. This might lead to unexpected controller selection.
[move_group-4] [moveit_task_constructor_visualization.execute_task_solution<PHONE_NUMBER>.107050142]: The trajectory of stage '21' from task '' does not have any controllers specified for trajectory execution. This might lead to unexpected controller selection.
[move_group-4] [moveit_task_constructor_visualization.execute_task_solution<PHONE_NUMBER>.107091647]: The trajectory of stage '24' from task '' does not have any controllers specified for trajectory execution. This might lead to unexpected controller selection.
[move_group-4] [moveit_task_constructor_visualization.execute_task_solution<PHONE_NUMBER>.107175876]: The trajectory of stage '25' from task '' does not have any controllers specified for trajectory execution. This might lead to unexpected controller selection.

If the joints of motion can be executed by multiple controllers (i.e. position, position with compliance, torque) how do you tell MoveIt which one it should use to execute?

Often, there is only one controller applicable. Also, we have defaults as far as I remember. The warning should actually only trigger when there is ambiguity.

Sorry that rebase didn't go as expected.
I'll get this cleaned up and tested the ros2 branch. Is that targeting ROLLING?


chore: add devilkiller-ag as codeowner

Description
With his latest work on the playground: https://github.com/asyncapi/modelina/pull/1628
I am suggesting we add @devilkiller-ag as a codeowner for the website 💪
cc @kennethaasan @magicmatatjahu @Ferror @leigh-johnson @Samridhi-98 any objections?
Lazy consensus will be used with @devilkiller-ag acceptance after 1 week from this PR creation 🙂

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 92.516%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
5600

Relevant Lines:
5901

💛  - Coveralls

@devilkiller-ag please approve the PR (or decline ;)) if you want to become a codeowner and take on that responsibility 🙂

Hi @jonaslagoni, I accept to be the code owner of the Modelina Website. However, I am not getting an option for approving this PR.

@devilkiller-ag if you go into files changed you should see a review button 🙂

Oh got it 😁. I thought I had to Approve the PR Merge.

Welcome again @devilkiller-ag 🎉

Welcome again @devilkiller-ag 🎉

Thanks @jonaslagoni 😊

@devilkiller-ag you can now add yourself to the AsyncAPI TSC incase you are interested ✌
Follow the same approach as here: https://github.com/asyncapi/community/pull/971

@devilkiller-ag you can now add yourself to the AsyncAPI TSC incase you are interested ✌
Follow the same approach as here: https://github.com/asyncapi/community/pull/971

Sure, I will do this today✨

Welcome!

Welcome!

Thanks @kennethaasan ✨

@jonaslagoni kind reminder that you need to invite @devilkiller-ag as maintainer to this repo, otherwise GitHub will not respect his approvals

🤦 Completely forgot thanks @derberg, @devilkiller-ag check your email, should have received the invite 🙂

Thanks @jonaslagoni, I accepted the invite.


"Override MTU" for a Guest WiFi Interface

Setting "Override MTU" for a guest WiFi network interface does not take effect after Save & Apply; a reboot is required to see a new MTU on the interface. The same change for a wired interface or a bridge does not require a reboot and is active right after Save & Apply.

Not quite sure if this is LuCI issue, or an issue of the underlying wifi driver components (driver, hostapd, etc.).
with exactly which LuCI version?
17.01? master?  (or master with new rollback code?)

The latest master built earlier today, but I not think that is a new issue though; I saw it a while back.

The value is saved, it just does not take effect until a reboot. Maybe something does not get restarted?

The latest master built earlier today

That is the reason why I was asking about the exact version.
jow has been rewriting some of the code, so it is possible that this is rather recent regression.
Most of the wifi configuration will get revamped along the rollback functionality. see https://github.com/openwrt/luci/pull/1769/commits for the current patches.
Similar symptoms were discussed e.g. started from https://github.com/openwrt/luci/pull/1769#issuecomment-385228343 but that is mostly in the context of the first version of rollback. But still, the old wifi section code has some peculiarities, so a rewrite of that code may be right on the spot.

That needs to be fixed in OpenWrt. LuCI relies on netifd/procd doing the right things on reload.


Imagetype: minimal-raw support for rhel8/9 with xz compression

This pull request includes:

minimal-raw image type support for rhel8/9
fedora already had this image type , just added zstd compression stage.
add compression with newly added zstd stage in osbuild. (https://github.com/osbuild/osbuild/pull/1322)
This brings down the image size to something around 700MB.
updated README.md under image-types/ folder.
Closed https://github.com/osbuild/osbuild-composer/pull/3373 and updated this PR
[ ] adequate testing for the new functionality or fixed issue
[ ] adequate documentation informing people about the change such as

[ ] submit a PR for the guides repository if this PR changed any behavior described there: https://www.osbuild.org/guides/

Out of curiosity, how does the compression time and ratio compare to the alternatives?

Out of curiosity, how does the compression time and ratio compare to the alternatives?
@teg I have ran zstd and xz on my vm so timing might not be correctbut good for comparison.
`

time zstd -T0 --keep minimal-raw.img
minimal-raw.img      : 16.92%   (  3.69 GiB =>    639 MiB, minimal-raw.img.zst)
real	0m8.300s
user	0m12.950s
sys	0m2.535s
time xz -T0 --keep minimal-raw.img
real	8m58.167s
user	17m14.061s
sys	0m3.840s `
and sizes
# ls -al -rw-------. 1 root root  603381024 Jun 27 18:21 minimal-raw.img.xz -rw-------. 1 root root  669902862 Jun 27 18:21 minimal-raw.img.zst
Image size is smaller by around 66MB when xz is used by for compression time zstd is clearly a winner!

Thanks a lot for this PR. I added some comments inline, but I have also some general comments (and some to just reiterate):

I think that minimalRawImage() function should go away in all distros and you should set the compression to "zstd" in the imageType definition instead. It will work just fine with the existing liveImage() function.

Please split the PR into smaller self-contained commits, instead of having just one giant commit. Sometimes it is not possible, but in this case it it and it would make it easier to follow each change. E.g.:

The addition of support for zstd osbuild stage can be a separate commit.
The addition of the Zstd pipeline implementation can be a separate commit.
The addition of zstd compression type in LiveImage can be a separate commit.
Switching Fedora to use the compression + regenerating manifests can be a separate commit.
Adding the image type to el8 and el9 can be a separate commit.

I would like to ask you for additional context and motivation for switching to the compressed file format (in the commit message). My reasoning is that we tend to produce images ideally in ready-to-use format. This change goes to the opposite direction, since the user needs to decompress the image before it can be used. Also this will be a backward-incompatible change in released Fedora version from user's point of view. Having a good justification for this step in the commit message would help (also for the future if we need to ever get back to this change).

Besides these comments, the PR looks really good, great job on that 👍

@thozza Thanks for the review :) , will update accordingly 👍

Thank you @sarmahaj - those are some compelling numbers!

Thanks for your changes. The PR looks great. There are just a few small things that I commented on.
In addition to those, I think that 3b116ae should be split, specifically the regenerated test manifest. The commit contains modified manifests for el8 and el9 (including centos stream), but those distros have been chnaged only after that commit, so in reality, those manifest changes should be part of it instead. The same goes to the test case modification, which would IMO fail if run on that commit (because rhel was not changed yet at that point).
Overall, great work, thanks!

@thozza Thanks !
I actually have messed up a bit in the files while rebasing , I am working on it(some tests are failing) .
I will split the manifest commit too, and convert the state of the PR to open till CI turns green ! :D

Aside from this (but this is up to you so feel free to fully ignore this) it'd likely be better if you have minimal-raw and minimal-raw-zstd in RHEL. Only offering zstd just adds extra actions before the image can be used.

not sure here, when playing with ARM machines remotely, transferring 12gb of stuff is pretty bad anyway, and having it spitting out compressed images is a plus. Decompressing zstd is pretty fast anyway too but the gain is on the transfer side of the world. zstd came originally from @ondrejbudai too as I wanted to use xz for compression so naming it -zstd feels weird.

The issue is that the minimal-raw image type for Fedora has been in osbuild-composer for a few releases by now. Changing it to output files in a different format and with a different name would break anyone that has built tooling over this (I know I personally have some scripts that would break over this that I use to generate my server images).

this is a good point indeed - maybe @nullr0ute can comment on what we want to do it here in fedora

Aside from this (but this is up to you so feel free to fully ignore this) it'd likely be better if you have minimal-raw and minimal-raw-zstd in RHEL. Only offering zstd just adds extra actions before the image can be used.

not sure here, when playing with ARM machines remotely, transferring 12gb of stuff is pretty bad anyway, and having it spitting out compressed images is a plus. Decompressing zstd is pretty fast anyway too but the gain is on the transfer side of the world. zstd came originally from @ondrejbudai too as I wanted to use xz for compression so naming it -zstd feels weird.

In Fedora we use .xz compression, the concern I have with using .zstd is that unless it yields massive size reduction over .xz I would actually prefer to use .xz because if a customer is writing out the image using el7 or even something like windows I'm not sure how widely supported .zstd is in tools where .xz is pretty widely supported.

The issue is that the minimal-raw image type for Fedora has been in osbuild-composer for a few releases by now. Changing it to output files in a different format and with a different name would break anyone that has built tooling over this (I know I personally have some scripts that would break over this that I use to generate my server images).

this is a good point indeed - maybe @nullr0ute can comment on what we want to do it here in fedora

So while it's supported Fedora for a few releases it's not actually used in deliverables, we still use ImageFactory there, it outputs .xz files, so unless there's a massive reduction in size from xz -> zstd I would actually prefer to use .xz

After discussion around using zstd or xz , we have decided to go for xz over zstd.
Summary :

there is no huge difference in compression size ,infact zstd has around 70MB additional size.
xz is widely deployed and supported across a bunch of platforms whereas zstd is fairly new. So user & compatibility  pov it is better to use xz.
I will close this PR and create a new one for minimal-raw with xz compression support for fedora & rhel8/9.

Thanks @thozza @supakeen for reviewing this PR!
cc: @ondrejbudai (Thanks for suggesting to use zstd, Its been added to osbuild stages, even though we are not using it for minimal-raw I hope it ll be used for for some other purpose !)

There's no need for a new PR is there? It can be changed to use xz in this one.
@nullr0ute @runcom it seems we've evaded the question about doing this to Fedora. While it's now been confirmed that Fedora itself doesn't use minimal-raw images this is still a breaking change to anyone that is currently using minimal-raw.

@nullr0ute @runcom it seems we've evaded the question about doing this to Fedora. While it's now been confirmed that Fedora itself doesn't use minimal-raw images this is still a breaking change to anyone that is currently using minimal-raw.

Personally I would move Fedora to the xz by default as well. I don't think Fedora is currently widely used, but I would like to start to use it in the main compose process and for that it would need to be xz as that's the artifact we currently produce.. I personally don't see this as breaking, but actually fixing a bug/oversight.

@nullr0ute @runcom it seems we've evaded the question about doing this to Fedora. While it's now been confirmed that Fedora itself doesn't use minimal-raw images this is still a breaking change to anyone that is currently using minimal-raw.

Personally I would move Fedora to the xz by default as well. I don't think Fedora is currently widely used, but I would like to start to use it in the main compose process and for that it would need to be xz as that's the artifact we currently produce.. I personally don't see this as breaking, but actually fixing a bug/oversight.

In that case I'm ok with approving this PR when it's been swapped over to use xz :)

@sarmahaj I found an issue on RHEL8.9. The journald doesn't have persistent logging. It only keeps the current one after reboot. But RHEL 9.3 does not have this issue. Could you please take a look? Thanks.

@sarmahaj I found an issue on RHEL8.9 and CS8. The journald doesn't have persistent logging. It only keeps the current one after reboot. But RHEL 9.3 and CS9 does not have this issue. Could you please take a look? Thanks.

@henrywang Thanks for updating the minimal-raw.sh and notifying the issue I will take a look it at!

In that case I'm ok with approving this PR when it's been swapped over to use xz :)

I would be still OK with adding the support for zstd compression in pipeline implementations and also adding the support for the zstd osbuild stage. Even if it is not used by any image type. The rest of the discussion seems to be settled down already...

@sarmahaj I found an issue on RHEL8.9, 9.3, CS8 and CS9. The journald doesn't have persistent logging. It only keeps the current one after reboot. But Fedora 37/38/rawhide does not have this issue. Could you please take a look? Thanks. Following test log from Fedora:
TASK [list boots] **************************************************************
changed: [<IP_ADDRESS>] => changed=true
  cmd: journalctl --list-boots
  delta: '0:00:00.013482'
  end: '2023-07-07 04:28:59.640266'
  msg: ''
  rc: 0
  start: '2023-07-07 04:28:59.626784'
  stderr: ''
  stderr_lines: <omitted>
  stdout: |-
    IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
     -1 8441a01b1ee145789e97622d385b9a49 Fri 2023-07-07 04:24:50 UTC Fri 2023-07-07 04:27:21 UTC
      0 aa7e0e675bd3438a90e012de315b1cca Fri 2023-07-07 04:28:03 UTC Fri 2023-07-07 04:28:59 UTC
  stdout_lines: <omitted>

@henrywang After some discussions with team we found out that persistent logging is disabled in rhel by default https://access.redhat.com/solutions/696893 , and has to be enabled. We tried with method 1 mentioned in this doc and it works. (whereas for fedora is probably enabled thats why persistent logging works on fedora) .
Could you please update the test files for 'minimal-raw' accordingly.

This PR is moved to osbuild/images and merged https://github.com/osbuild/images/pull/29
Test are covered in https://github.com/osbuild/osbuild-composer/pull/3565
Closing this PR.


BIGTOP-3448. Building Alluxio fails on all distros.

Tested on Ubuntu 18.04.

Looks good. +1.

+1 tested as well on debian 10 :)

Thanks everyone! Merged into master and branch-1.5.


Generating: "AssertionError: Some objects had attributes which were not restored:"

Hi,
I'm having problems generating a WAV file with the error message copied at the bottom.
 I'm going off the Google Colab file which is here along with the settings I've changed:
https://colab.research.google.com/drive/1JLWJhgFSrhUxzrQ3PCciy5CBBgsxTdQk?usp=sharing
Thanks so much for this! It's going to give me so many options creatively if it works.
Traceback (most recent call last):
File "generate.py", line 177, in
main()
File "generate.py", line 173, in main
args.sample_rate, args.temperature, args.seed, args.seed_offset)
File "generate.py", line 164, in generate
model = create_inference_model(ckpt_path, num_seqs, config)
File "generate.py", line 158, in create_inference_model
model.load_weights(ckpt_path)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 250, in load_weights
return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py", line 1248, in load_weights
status.assert_nontrivial_match()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/util.py", line 998, in assert_nontrivial_match
return self.assert_consumed()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/util.py", line 975, in assert_consumed
"".join(unused_attribute_strings)))
AssertionError: Some objects had attributes which were not restored:
<tf.Variable 'sample_rnn/frame_rnn/upsample:0' shape=(4, 1024, 1024) dtype=float32, numpy=
array([[[-1.02961492e-02,  1.35401851e-02,  1.50977094e-02, ...,
1.01866096e-03,  5.10610989e-04,  1.87566746e-02],
[ 8.87395721e-03, -4.18026838e-03, -6.99102599e-03, ...,
3.04940287e-02, -1.03946663e-02,  2.09112354e-02],
[ 2.33578612e-03,  1.77005269e-02,  1.46402500e-03, ...,
-1.09163010e-02, -1.46275619e-02, -6.82321005e-03],
...,
[ 2.34062839e-02,  7.47851795e-03,  3.84401088e-03, ...,
-2.79776342e-02, -5.57626272e-03,  6.32355595e-03],
[-2.42233202e-02, -7.10393721e-03, -1.93884352e-03, ...,
-1.35604087e-02, -8.97351932e-03,  2.98934337e-03],
[-7.63602601e-03,  3.31373848e-02,  4.70729574e-04, ...,
3.60660697e-03,  2.51634941e-02,  1.27788875e-02]],
   [[ 7.90888385e-04, -3.12626585e-02, -1.91800983e-03, ...,
     -2.86801532e-02, -1.09788710e-02, -1.92267634e-02],
    [ 8.18298850e-03,  2.17494778e-02, -1.31149981e-02, ...,
      1.70598291e-02, -4.57543228e-03,  4.49129613e-03],
    [-1.42808305e-02, -7.20372563e-03,  1.13683217e-03, ...,
     -3.27150226e-02, -7.19945692e-03,  3.56810377e-03],
    ...,
    [-1.28457043e-02,  1.22231981e-02, -3.25182686e-03, ...,
     -1.34714535e-02, -9.11022816e-03, -2.62970291e-02],
    [-1.67476721e-02,  1.26866661e-02, -7.53369601e-03, ...,
     -7.10430043e-03,  1.64546054e-02, -1.77345611e-02],
    [-2.11791955e-02,  2.27067154e-03,  1.12983892e-02, ...,
     -1.88266076e-02,  1.00256186e-02, -2.94636320e-02]],

   [[-3.47011723e-02, -2.10044496e-02, -5.35532739e-03, ...,
     -8.72646458e-04,  1.23625691e-03,  9.46586806e-05],
    [ 8.24875291e-03, -3.31384689e-02,  1.78510528e-02, ...,
      1.44131845e-02,  2.74074776e-03, -1.71584859e-02],
    [ 7.65335234e-03, -1.22760180e-02,  1.58619806e-02, ...,
     -3.30588868e-04,  2.30300762e-02,  9.01468098e-03],
    ...,
    [ 2.68938299e-03, -1.40413400e-02, -2.10206192e-02, ...,
      2.21681856e-02, -2.58824211e-02,  1.42331515e-02],
    [ 1.94714218e-03, -1.04962348e-03,  1.70364808e-02, ...,
     -1.20551186e-02, -5.75437723e-03, -1.83531549e-02],
    [-1.18572414e-02,  1.84859596e-02, -1.52747957e-02, ...,
      2.44354509e-04, -1.44815380e-02,  1.78436679e-03]],

   [[-3.39218825e-02, -1.77963041e-02, -2.75604390e-02, ...,
      1.53806815e-02,  2.57573631e-02,  3.56114772e-03],
    [-3.40288728e-02, -1.82274655e-02,  1.40822551e-03, ...,
      4.86909645e-03, -1.64082851e-02,  2.20462610e-03],
    [-5.24610840e-03, -1.92433354e-02,  1.91187691e-02, ...,
     -4.68099862e-03,  1.86212994e-02,  2.83861328e-02],
    ...,
    [-6.23152545e-03,  5.05297771e-03, -1.06918439e-02, ...,
      1.04744453e-02,  1.99548360e-02, -1.57663561e-02],
    [ 3.34415659e-02, -1.70691370e-03, -1.00803738e-02, ...,
     -1.48876915e-02, -5.55169024e-03, -2.88231717e-03],
    [-3.79145518e-03,  4.98119649e-03,  2.04012338e-02, ...,
      1.38682462e-02,  3.17192413e-02,  8.98275711e-03]]],

Hi there,
I assume this is the generate.py script... Could you post the exact script call that caused this, with the parameters you used?

Thanks - I did work out that the checkpoint_path needed to be changed to model.ckpt-8.index (that seemed to be the last checkpoint that was saved in the predict folder. Should I have just put model.ckpt-8 then? I'll try again with this if necessary.
The code was:
!python generate.py \   --output_path ./generated/default/test.wav \   --checkpoint_path ./logdir/default/predict/model.ckpt-8.index \   --dur 8 \   --sample_rate 11025 \   --temperature 0.1 \   --config_file ./default.config.json
Thanks!

Ah, I can see what your problem is. You've passed model.ckpt-8.index. You don't actually need any extension, if you look at the section on checkpoints in the notebook it explains that a checkpoint is in fact multiple files, but you don't need an extension. Sorry, that can be confusing! So just pass --checkpoint_path ./logdir/default/predict/model.ckpt-8.

Runs great now, thanks!

No problem. Generation stage is the slowest part of the whole thing currently, we are working on speeding things up. Also, you'll probably need to train for quite a few epochs before you get anything useful, at early stages it'll likely just be white noise. Usually about 15 - 20 epochs before anything decent appears, and usually more than 100 before you'll get anything really interesting. Although there are updates coming soon (next 2 weeks) that will improve training times.


Other value for matrix question column

Hi dropdown and checkboxes in matrix question do not have other value property which seems a valid scenario.

It is implemented. The hasOther property is added to a column object.

Hi other value for checkboxes in dynamic table seems to be broken.

Yes, thank you!
The issue is fixed. Please check it out.
Thank you,
Andrew

Thanks it's fixed now :)


build breaks

Hi,
if I try the step python3 build/build.py --project plugin-apt --package-for deb from delivery-optimization-client i get
script_path=/home/gerhard/test-x509/do-client/build
Starting Build for project: plugin-apt
Target OS: Linux
Flavor: linux-debug
Config: debug
Subproject: plugin-apt
CMake Target: all
CMake Generator: Ninja
Clean: False
Source Path: /home/gerhard/test-x509/do-client
Build Path: /tmp/build-deliveryoptimization-plugin-apt/linux-debug
Running command cmake /home/gerhard/test-x509/do-client -G Ninja -DCMAKE_BUILD_TYPE=Debug -DDO_BUILD_TIMESTAMP=20240730.155226 -DDO_INCLUDE_PLUGINS=TRUE.
-- DO Platform: Linux
-- DO Interface: REST
-- DO Client: Agent
CMake Warning at CMakeLists.txt:96 (message):
  /lib is a symlink and /include does not exist.  Updating CMAKE_PREFIX_PATH
  to /usr.

-- Using builder name: DU
-- NOTE: Please modify builder name if modifying project source
-- Build timestamp found: 20240730.155226
-- Including subproject plugins
-- Packaging for DEB
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/build-deliveryoptimization-plugin-apt/linux-debug
Executing: cmake --build /tmp/build-deliveryoptimization-plugin-apt/linux-debug --target all
Running command cmake --build /tmp/build-deliveryoptimization-plugin-apt/linux-debug --target all.
[3/4] Building CXX object plugins/linux-apt/CMakeFiles/deliveryoptimization-plugin-apt.dir/main.cpp.o
FAILED: plugins/linux-apt/CMakeFiles/deliveryoptimization-plugin-apt.dir/main.cpp.o
/usr/bin/c++  -DDO_CLIENT_AGENT=1 -DDO_INTERFACE_REST=1 -DDO_PLATFORM_LINUX=1 -DDO_PLUGIN_APT_CACHE_PATH=\"/var/cache/deliveryoptimization-plugin-apt\" -DDO_PLUGIN_APT_LOG_PATH=\"/tmp/deliveryoptimization-plugin-apt.log\" -I/home/gerhard/test-x509/do-client/common/lib-doversion -isystem /usr/include/deliveryoptimization -isystem /usr/include/uuid -Wall -fPIE -pie -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wreturn-local-addr -fmerge-all-constants -g -DDEBUG -fPIE   -std=c++17 -MD -MT plugins/linux-apt/CMakeFiles/deliveryoptimization-plugin-apt.dir/main.cpp.o -MF plugins/linux-apt/CMakeFiles/deliveryoptimization-plugin-apt.dir/main.cpp.o.d -o plugins/linux-apt/CMakeFiles/deliveryoptimization-plugin-apt.dir/main.cpp.o -c /home/gerhard/test-x509/do-client/plugins/linux-apt/main.cpp
/home/gerhard/test-x509/do-client/plugins/linux-apt/main.cpp: In function ‘DownloadResult SimpleDownload(const string&, const string&)’:
/home/gerhard/test-x509/do-client/plugins/linux-apt/main.cpp:196:73: error: conversion from ‘void’ to non-scalar type ‘std::error_code’ requested
  196 |         std::error_code errorCode = msdo::download::download_url_to_path(url, tempPath);
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
ninja: build stopped: subcommand failed.
Running cmake --build /tmp/build-deliveryoptimization-plugin-apt/linux-debug --target all failed. Rethrowing exception
Traceback (most recent call last):
  File "build/build.py", line 761, in <module>
    main()
  File "build/build.py", line 31, in main
    build.run()
  File "build/build.py", line 460, in run
    super().run()
  File "build/build.py", line 264, in run
    self.build()
  File "build/build.py", line 382, in build
    run_command(build_command)
  File "build/build.py", line 724, in run_command
    _check_call(command)
  File "build/build.py", line 748, in _check_call
    subprocess.check_call(
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '/tmp/build-deliveryoptimization-plugin-apt/linux-debug', '--target', 'all']' returned non-zero exit status 1.
`

I have tested this with main and develop

I use Ubuntu 20.04

I'm unable to reproduce in 20.04, develop branch. You probably have an incompatible version of the DO SDK installed. See the steps to install the SDK in the README.md.
Curious why you are trying to build and use the APT plugin. Are you using Microsoft Connected Cache in your environment?

No response. Closing.


Allow course team to specify date for certificates

As an instructor, I need more time between when my course ends and when I'm ready for certificates to be created. I use this time to review and adjust grades, and to review any proctored exams that were flagged.
Acceptance Criteria:

[ ] Research how to turn on the "Certificates Display Behavior" and "Certificates Available Date" settings in Studio (it's not available on studio.mitxonline.mit.edu, but it is available on studio.edx.org

Syncing Steps

[ ] Sync the certificates available date from Studio to the mitxonline app
[ ] Create certificates after the certificates available date, instead of after the course end date.

The above criteria acceptance might be divided into a couple of PRs as mentioned below (Orderly):

A PR in MITxOnline to add the certificate_available_date field, and change all the certificate generation logic from end_date to certificate_available_date. (This enables us to add certificate available date manually until we have our PR merged in edX. Already in progress WIP-https://github.com/mitodl/mitxonline/pull/1286)
An upstream PR to edX to include certificate_available_date to course details API.
A PR in edx-api-client to enable fetching certificate_available_date in conjunction with another PR in MITxOnline that uses this edx-api-client and syncs the certificate_available_date from edX.

Update:
So I was looking around this and this is what I've got.

Research how to turn on the "Certificates Display Behavior" and "Certificates Available Date" settings in Studio (it's not available on studio.mitxonline.mit.edu, but it is available on studio.edx.org

To enable the Certificate's display behavior and certificate available dates we need to do the following:

Add a waffle switch in admin/waffle/switch/1/change/ with value http://localhost:18000/admin/waffle/switch/1/change/  (Ref from edX).
Add a Django setting/feature in CMS's FEATURES dictionary "ENABLE_V2_CERT_DISPLAY_SETTINGS": True. (Ref from edX).

After the above steps, the certificate display behavior and date options should be visible in studio>your course>Settings>Schedule & Details.

Sync the certificates available date from Studio to the mitxonline app

For this, we need an API from edX to provide the details, while looking around I found two APIs only that send this value in the response:

Course Outline API needs an enrolled user.
Course Progress API needs an enrolled user.
Courseware API needs an enrolled user.

The problem with the above APIs is that they need an enrolled user for the certificate details and we will need to call the APIs independent of enrollment. Turns out we can't use the above APIs.

Create certificates after the certificates available date, instead of after the course end date.

Keeping the current possibilities in mind, I think we can divide this into two possible parts:

Add a new field(certificate_available_date) in MITxOnline, and change our certificate generation flow to follow this value instead of end_date. e.g. if certificate_available_date is available, use this, otherwise use end_date. Create the PR and get it running. This will enable us to at least set this value manually and use it.
Create an upstream PR that adds the certificate_available_date in the course details API. Probably, get it merged, and backport it too so that we can use it in our current edX instances.

@pdpinch FYI. Let me know your thoughts on this.

Update:
So this is probably going to be divided into a couple of PRs(Issues) based on the above comment.

A PR in MITxOnline to add the certificate_available_date field, and change all the certificate generation logic from end_date to certificate_available_date. (This enables us to add certificate available date manually untill we have our PR merged in edX. Already in progress WIP-https://github.com/mitodl/mitxonline/pull/1286)
An upstream PR to edx to include certificate_available_date to course details API.
A PR in edx-api-client to enable fetching certificate_available_date in conjunction with another PR in MITxOnline that uses this edx-api-client and syncs the certificate_available_date from edX.

Steps 1 & 3 can be done independently as mentioned, but we can also first do step#2 and then do steps 1 & 3 together. This might take some time since this required the edX PR to get merged first and probably backported to Olive.

That sounds correct to me @arslanashraf7. Can you update the description of this issue to reflect all the necessary steps and sequence? Thanks for starting on #1. I'm not quite ready to start working on the other steps.

To enable the Certificate's display behavior and certificate available dates we need to do the following:
Add a waffle switch in admin/waffle/switch/1/change/ with value http://localhost:18000/admin/waffle/switch/1/change/ (Ref from edX).
Add a Django setting/feature in CMS's FEATURES dictionary "ENABLE_V2_CERT_DISPLAY_SETTINGS": True. (Ref from edX).

Does either of these steps require deploying the credentials application or any other apps we may not be running in mitxonline?

When I look at course in studio (e.g https://studio-qa.mitxonline.mit.edu/settings/details/course-v1:MITx+14.310x+3T2021) I don't see the fields for setting certificate_available_date. Is there a waffle flag of a setting that needs to be set?

When I look at course in studio (e.g https://studio-qa.mitxonline.mit.edu/settings/details/course-v1:MITx+14.310x+3T2021) I don't see the fields for setting certificate_available_date.

That might be because we haven't probably configured it there.

Is there a waffle flag of a setting that needs to be set?

Yes, we need to set a couple of things that I mentioned in this comment - Edited the comment a bit to be clearer and fixed a value. I can do these settings in the QA tomorrow, This will need a PR in https://github.com/mitodl/ol-infrastructure and a switch through Django Admin.

So, As per my last comment, I added the switch here. And the PR for enabling the feature flag for certificates display behaviour is at https://github.com/mitodl/ol-infrastructure/pull/1294.

I tried setting the waffle flag in production, but it didn't have the desired effect of letting me set the certificate generation date. Any idea what might be going wrong?
https://studio.mitxonline.mit.edu/admin/waffle_utils/waffleflagcourseoverridemodel/24/change/
https://studio.mitxonline.mit.edu/settings/details/course-v1:MITxT+14.310x+1T2023

This is working on RC though, e.g. take a look at https://studio-qa.mitxonline.mit.edu/settings/details/course-v1:Arbisoft+ARB_CRT_0001+2022_T1#schedule.
Maybe that's because I tested it there by setting the switch overall, e.g. https://studio-qa.mitxonline.mit.edu/admin/waffle/switch/.
I can try testing it by just setting an override instead of a switch itself.

Maybe that's because this is a switch and not a flag that's why the override isn't working, Are we looking to enable is just course wise?

Oh, I didn’t know that.
We can turn it on for all courses. That’s OK.

That did it, thanks!

For the record, I set the switch in https://studio.mitxonline.mit.edu/admin/waffle/switch/4/change/


v3.6.0 package contains ~175 MB of .vscode files

I upgraded an electron app from v3.5.3 to v3.6.0 and the build size jumped significantly. Tracked the extra weight back to<EMAIL_ADDRESS>
$ pwd
/tmp/test/node_modules/spellchecker

$ du -hc . | grep M
1.2M	./build/Release/obj.target/spellchecker/src
1.2M	./build/Release/obj.target/spellchecker
2.1M	./build/Release/obj.target
3.4M	./build/Release
3.5M	./build
 83M	./.vscode/ipch/7f56f8a8ad139ab6
 83M	./.vscode/ipch/e0f6371f7634001f
166M	./.vscode/ipch
166M	./.vscode
1.1M	./vendor/hunspell/src/hunspell
1.2M	./vendor/hunspell/src
1.3M	./vendor/hunspell
2.0M	./vendor
172M	.
172M	total

/cc @rafeca

Thanks for the heads up @javan 😅
I'll republish a new version without that folder

Published v3.6.1 without the .vscode files, thanks for creating the issue!


nsh/fsutils: fix build break if CONFIG_NSH_DISABLE_PKILL not disabled

Summary
nsh/fsutils: fix build break if CONFIG_NSH_DISABLE_PKILL not disabled
arm-none-eabi/bin/ld: apps/libapps.a(nsh_proccmds.c.obj): in function `cmd_pkill':
apps/nshlib/nsh_proccmds.c:861:(.text.cmd_pkill+0x38): undefined reference to `nsh_getpid'

Signed-off-by: chao an<EMAIL_ADDRESS>Impact
N/A
Testing
ci-check

Thank you @anchao :-)
We have similar issue with free command not enabled when PROCFS is not enabled. I mean why not use positive selection only (i.e. ENABLE_FREE and ENABLE_PROCFS) instead one thing is disabled and other thing is enabled and that makes things more complicated than should be :-)

I guess that the original design wants to enable the command by default for newbie.


The .com domain expired links are broken

👋 Hey @MichaelCurrin, I haven't seen @SotiriosVrachas on here in a LONG time. You could check out other GitHub projects that call this one out [https://github.com/search?q=remote_theme%3A+SotiriosVrachas%2Fjekyll-theme-startbootstrap-agency&type=Code](Github Search) or this site that i maintain uses this project as well [https://visionnetworkinc.com](Vision Network Solutions) @ https://github.com/Mutmatt/visionNetworkSolutions

Thanks @Mutmatt
I'll have a look at those.
Yes those links.
For interest, this is my site using the theme - private repo
https://secondlinersimprov.co.za/

For interest, this is my site using the theme - private repo
https://secondlinersimprov.co.za/
I found it useful to split home.html layout into several includes files:
{% include header.html %}

{% include about.html %}

{% include events.html %}

{% include contact.html %}

raviriley/agency-jekyll-theme/
I wanted to use this theme but saw that it's outdated and not maintained, so I decided to convert the newest Bootstrap code to a Jekyll theme from scratch. Then, once I started using the theme, I realized I could make some additions to make the original theme more useful, such as Google Analytics support, an actually functional contact form, customizable styling, etc.
Check out the details here!
https://github.com/raviriley/agency-jekyll-theme/
Let me know if you run into any issues!