[WIP][SPARK-23674][ML] Adds Spark ML Events

What changes were proposed in this pull request?
This PR proposes to add ML events so that other developers can track and add some actions for them.
Introduction
This PR proposes to send some ML events like SQL. This is quite useful when people want to track and make some actions for corresponding ML operations. For instance, I have been working on integrating
Apache Spark with Apache Atlas. With some custom changes with this PR, I can visualise ML pipeline as below:

I think not to mention how useful it is to track the SQL operations. Likewise, I would like to propose ML events as well (as lowest stability @Unstable APIs for now - no guarantee about stability).
Implementation Details
Sends event (but not expose ML specific listener)
In events.scala, it adds:
@Unstable
case class ...StartEvent(caller, input)
@Unstable
case class ...EndEvent(caller, output)

object MLEvents {
  // Wrappers to send events:
  // def with...Event(body) = {
  //   body()
  //   SparkContext.getOrCreate().listenerBus.post(event)
  // }
}

This way mimics both:
1. Catalog events (see org.apache.spark.sql.catalyst.catalog.events.scala)

This allows a Catalog specific listener to be added ExternalCatalogEventListener

It's implemented in a way of wrapping whole ExternalCatalog named ExternalCatalogWithListener
which delegates the operations to ExternalCatalog

This is not quite possible in this case because most of instances (like Pipeline) will be directly created in most of cases. We might be able to do that via extending ListenerBus for all possible instances but IMHO it's too invasive. Also, exposing another ML specific listener sounds a bit too much at this stage. Therefore, I simply borrowed file name and structures here
2. SQL execution events (see org.apache.spark.sql.execution.SQLExecution.scala)

Add an object that wraps a body to send events

Current apporach is rather close to this. It has a with... wrapper to send events. I borrowed this approach to be consistent.
Add ...Impl methods to wrap each to send events
1. mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
- def save(...) = { saveImpl(...) }
+ def save(...) = MLEvents.withSaveInstanceEvent { saveImpl(...) }
  def saveImpl(...): Unit = ...

Note that saveImpl was already implemented unlike other instances below.
- def load(...): T
+ def load(...): T = MLEvents.withLoadInstanceEvent { loadImple(...) }
+ def loadImpl(...): T

2. mllib/src/main/scala/org/apache/spark/ml/Estimator.scala
- def fit(...): Model
+ def fit(...): Model = MLEvents.withFitEvent { fitImpl(...) }
+ def fitImpl(...): Model

3. mllib/src/main/scala/org/apache/spark/ml/Transformer.scala
- def transform(...): DataFrame
+ def transform(...): DataFrame = MLEvents.withTransformEvent { transformImpl(...) }
+ def transformImpl(...): DataFrame

This approach follows the existing way as below in ML:
1. transform and transformImpl
https://github.com/apache/spark/blob/9b1f6c8bab5401258c653d4e2efb50e97c6d282f/mllib/src/main/scala/org/apache/spark/ml/Predictor.scala#L202-L213
https://github.com/apache/spark/blob/9b1f6c8bab5401258c653d4e2efb50e97c6d282f/mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala#L191-L196
https://github.com/apache/spark/blob/9b1f6c8bab5401258c653d4e2efb50e97c6d282f/mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala#L1037-L1042
2. save and saveImpl
https://github.com/apache/spark/blob/9b1f6c8bab5401258c653d4e2efb50e97c6d282f/mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala#L166-L176
Inherited ones are intentionally omitted here for simplicity. They are inherited and implemented at multiple places.
Backward Compatibility
This keeps both source and binary backward compatibility. I was thinking enforcing ...Impl by leaving it abstract methods to force to implement but just decided to leave a body that throws UnsupportedOperationException so that we can keep full source and binary compatibilities.

For user-faced API perspective, there's no difference. ...Impl methods are protected and not visible to end users.

For developer API perspective, if some developers want to ... methods instead of ...Impl, that's still fine. It only does not handle events. If developers want to handle events from their custom implementation, they should implement ...Impl. Of course, it is encouraged to implement ...Impl

How was this patch tested?
Manually tested and unit tests were added.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5877/
Test PASSed.

Test build #99859 has started for PR 23261 at commit 56805d6.

Test build #99859 has finished for PR 23261 at commit 56805d6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TransformStart(transformer: Transformer, input: Dataset[_]) extends MLEvent
case class TransformEnd(transformer: Transformer, output: Dataset[_]) extends MLEvent
case class FitStart[M <: Model[M]](estimator: Estimator[M], dataset: Dataset[_]) extends MLEvent
case class FitEnd[M <: Model[M]](estimator: Estimator[M], model: M) extends MLEvent
case class LoadInstanceStart[T](reader: MLReader[T], path: String) extends MLEvent
case class LoadInstanceEnd[T](reader: MLReader[T], instance: T) extends MLEvent
case class SaveInstanceStart(writer: MLWriter, path: String) extends MLEvent
case class SaveInstanceEnd(writer: MLWriter, path: String) extends MLEvent

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99859/
Test FAILed.

Test build #99865 has started for PR 23261 at commit a44e30e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5881/
Test PASSed.

Test build #99867 has started for PR 23261 at commit 6dbe14c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5883/
Test PASSed.

I'm going to rebase and reopen a PR to retrigger AppVeyor tests. The failure looks unrelated

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5884/
Test FAILed.


Add quality check for secrets scanning using TruffleHog

Issue #, if available:
Description of changes:
Add quality check for secrets scanning using TruffleHog
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Reopening


Properly verify that the npm package is hosted on the current repository

As long as the name in the package.json exists somewhere in the registry, it uses that package. Yikes.

Fixed!


[Suggestion] treated wood manufacturing in barrels

many recipes still need treated wood, and rn it requires a vanilla bucket of creosote
for example: windmill blades
or at the very least make it create firmalife treated lumber, and make it recipe-usable

That's incorrect. JEI shows that it requires a vanilla bucket but in fact you can craft with a wooden bucket.

also do IE crates follow size limits?

Currently they don't.


start-all.sh: -d option doesn't work with the new (v2.3) Prometheus

HEAD: 99041c403c932bf75e009bf26746d587e9ec5d23
Description
Prometheus doesn't store/read data in the directory given with -d option.
Root cause
Prometheus changed the root directory name in the docker from prometheus-data to prometheus.

@amnonh actually, it was. ;)
Like I mentioned in the description, the data directory inside the docker is prometheus and not prometheus-data as the patch above assumes:
$ sudo docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                    NAMES
2be59ac09856        grafana/grafana:4.6.3       "/run.sh"                4 hours ago         Up 4 hours          <IP_ADDRESS>:3000->3000/tcp   agraf
42907f375d0c        prom/prometheus:v2.3.2      "/bin/prometheus -..."   4 hours ago         Up 4 hours          <IP_ADDRESS>:9090->9090/tcp   aprom
0b556e0f81b0        prom/alertmanager:v0.12.0   "/bin/alertmanager..."   4 hours ago         Up 4 hours          <IP_ADDRESS>:9093->9093/tcp   aalert
$ sudo docker exec -it aprom sh
/prometheus $ pwd
/prometheus
/prometheus $ ls
data
/prometheus $

It looks like this patch actually a root cause of this (github) issue.

@tzach can you confirm or deny the issue?

I can confirm the prometheus:v2.3.2 does not have a prometheus-data directory, and do have prometheus directory

@tzach The question, can you confirm that version 2.0 worked for you with an external data directory (i.e. using -d flag)

On latest branch-2.0 the issue persists. I checked inside the Prometheus container and the correct path for data dir is /prometheus/data
This is urgent since after restarting the container all the historical data is LOST!
/ $ [root@ip-172-16-99-139 scylla-grafana-monitoring-data]# docker exec -it fed3fc45a282 sh
/prometheus $ cd data
/prometheus/data $ ls -ltr
total 0
-rw-r--r--    1 nobody   nogroup          0 Sep 30 09:45 lock
drwxr-xr-x    2 nobody   nogroup         20 Sep 30 09:45 wal
/prometheus/data $

The issue still exists!!!
Correct data dir volume mount should include ":Z":
-v $DATA_DIR:/prometheus/data:Z
Otherwise the container start fails.

I think it's a permission issue on the data directory, but I can check the Z flag


consider descriptor pattern for GraphicFeature.

While playing around with some examples, I saw this type error reported by mypy:

I dug into a bit and learned about the Graphic/GraphicFeature relationship.  Apologies if you already know this and specifically decided against it... but the general pattern of an abstract _set method on GraphicFeature objects that retain a pointer to some parent object looks very much like it could be replaced/improved with python descriptors.
For example, the current pattern (highly simplified) is:
class GraphicFeature:
    def __init__(self, parent):
        self._parent = weakref.proxy(parent)

    def _set(self, value):
        raise NotImplementedError

class SomeFeature(GraphicFeature):
    def _set(self, value):
        print(f"setting {self.name!r} on {self._parent} to {value!r}")

class Graphic:
    def __setattr__(self, key, value):
        if hasattr(self, key):
            attr = getattr(self, key)
            if isinstance(attr, GraphicFeature):
                attr._set(value)
                return

        super().__setattr__(key, value)

class SomeGraphic(Graphic):
    def __init__(self) -> None:
        self.feature = SomeFeature()

with the descriptor pattern, that all reduces to
class SomeFeature:
    def __get__(self, instance, owner):
        ...

    def __set__(self, obj, value):
        print(f"setting feature on {obj} to {value!r}")

class SomeGraphic:
    feature = SomeFeature()

and when you set the feature attribute on an instance of SomeGraphic,  then SomeFeature.__set__ will be called with the instance and value:
In [18]: g = SomeGraphic()

In [19]: g.feature = 1
setting feature on <__main__.SomeGraphic object at 0x106c33750> to 1

this also works very nicely for typing:

note that it neither complains when setting  .feature = [1,2,3], and it also recognizes that accessing .feature will return a str (i.e. it's been coerced from list to some internal value), based on the type hints of __set__ and __get__

Thanks for all this!! I will have to dig into my old PRs in a few hours to see why we didn't do this, but it's likely I didn't know it could be done like this.
We don't use any getters for GraphicFeatures, they have a view methods on them which are public such as "add_event_handler()", and the "values" property for CmapFeature. So it should be sufficient to just implement set

I looked into descriptors and our own issues and this would generally be within the scope of a GraphicFeatures refactor that we want to do sometime next year in conjuntion with refactoring feature events, probably using observ
By using __set__ as you suggested we can do something like this, if you have ideas on how to improve this that would be welcome!
class SomeFeature:
    def __init__(self, value):
        self._value = value
        self._handlers = list()

    def __set__(self, obj, value):
        print(f"setting feature on {obj} to {value!r}")
        self._call_handlers({"value": value})

    def __call__(self):
        return self._value

    def add_event_handler(self, func: callable):
        self._handlers.append(func)

    def _call_handlers(self, ev: dict):
        for f in self._handlers:
            f(ev)

class SomeGraphic:
    feature = SomeFeature("foo")

>>> g = SomeGraphic()
>>> g.feature
<__main__.SomeFeature at 0x7ff4681baad0>

>>> g.feature()
'foo'

>>> g.feature.add_event_handler(lambda ev: print(f"event happened: {ev}"))
>>> g.feature = "bar"
setting feature on <__main__.SomeGraphic object at 0x7ff4681c44d0> to 'bar'
event happened: {'value': 'bar'}

If you're interested in helping out let us know :smile:

refactor that we want to do sometime next year in conjuntion with refactoring feature events, probably using observ

hah, yes I was also going to bring up psygnal with you at some point, which is also an observer pattern that arose from our needs of having an event layer at napari outside of vispy.  It's rather easy to get into trouble with strong-references to object-bound callbacks, and that lib tries very hard to be careful with both references and types.  thanks for pointing out observ, i'll have a look

yeah I'm happy to discuss this stuff more.  Not sure if I'll have time to be a part of the implementing.  A quick note on your pattern above.  If you want any of those other methods to be instance specific, you will want to add a __get__ method, and it should return an object (bound to the instance) that has all the events.  and the descriptor object itself shouldn't have a _value (since it will be shared across all instances of SomeGraphic in your example above).
In [14]: g1 = SomeGraphic()

In [15]: g2 = SomeGraphic()

In [16]: g1.feature.add_event_handler(print)

In [17]: g2.feature._handlers
Out[17]: [<function print at 0x104cff240>]

so, there's more to be discussed about the descriptor pattern...

I think it would be useful to get your thoughts if you're free to chat briefly in early January?

yep, happy to chat then

If the descriptors are instance attributes rather than class attributes that should be fine right?

descriptors are, by definition, class attributes (not instance attributes) that have special properties when accessed or set, that give them access to the class and/or instance on which they are being accessed/set. more details in the technical tutorial ... but note how, in my last example above, when you call add_event_handler on g1.feature, it actually also adds it to g2.feature._handlers.  Now, it doesn't have to be that way, (it's essentially a bug in the way you've implemented it above), but it does demonstrate that you have to be a bit careful, and understand exactly what the descriptor protocol implies for attribute access.

For what it's worth, I was a bit surprised by the behavior of features:
>>> sinc_graphic.thickness
ThicknessFeature for LineGraphic @ 0x161eba050, call `<graphic>.thickness()` to get value
>>> sinc_graphic.thickness()
5
>>> sinc_graphic.thickness = 3

I think most people to have a symmetry between the setter and getter, e.g.
 line_graphic.thickness = other_line_graphic.thickness

So although I appreciate the idea to add a handler to the feature like that, it results in, IMO, a somewhat whimsical API that may confuse users a bit.
An alternative way to add a handler might be:
>>> sinc_graphic.add_handler("thickness", my_handler)

Anyway, just my 2ct

Yes I agree I'm not the happiest about how we handle this and the events. Can discuss it, potentially next Wednesday. I became very used to Qt's way of doing things, valueChanged.connect(). I eventually want to look into observ as well which was recommended by Ivo. There's also psygnal by Tally which is modelled directly after Qt.

I just tried this and the descriptor pattern doesn't give us what we need, i.e. __setitem__ and __getitem__ functionality as far as I know, I will explore this more:
def parse_colors(value, n):
    if isinstance(value, str):
        return np.array([pygfx.Color(value)] * n)

class ColorFeature:
    def _get_buffer(self, instance) -> pygfx.Buffer:
        return instance.world_object.geometry.colors

    def __get__(self, instance, owner):
        return self._get_buffer(instance).data

    def __set__(self, obj, value):
        # __set__ and __get__ work but we need slicing and fancy indexing
        buffer = self._get_buffer(obj)

        buffer.data[:] = parse_colors(value, obj._n_vertices)

        buffer.update_range()

        self._changed(value)

    def __setitem__(self, *args):
        # setitem and getitem don't do anything in descriptors
        print(args)
        # buffer = self.get_buffer(obj)
        # buffer[key] = self.parse_colors(value, key.stop - key.start)

    def __getitem__(self, *args):
        print(args)

    def _changed(self, value):
        print(f"new val: {value}")

class Graphic:
    colors = ColorFeature()

    def __init__(self, colors="r"):
        xs = np.linspace(0, np.pi * 2, 100)
        ys = np.sin(xs)
        zs = np.zeros(xs.size)

        self._n_vertices = xs.size

        self.data = np.column_stack([xs, ys, zs]).astype(np.float32)

        colors = parse_colors(colors, self._n_vertices)

        self.world_object = pygfx.Line(
            geometry=pygfx.Geometry(positions=self.data, colors=colors),
            material=pygfx.LineMaterial(thickness=5, color_mode="vertex")
        )

In the worst case if this isn't possible I will refactor GraphicFeatures so that there's at least symetry between the setter and getter.

I just tried this and the descriptor pattern doesn't give us what we need, i.e. setitem and getitem functionality as far as I know,

Aah, I see what you mean. I wonder if its at all possible to realize a getter then. Because:
graphic.some_feature[10:20] = 10
                   /\
                    getter is invoked here

That's not quite how descriptors work.  Don't think of __get__ as returning the data itself; think of it as returning "what you get when you access this attribute"
So, if you want something like:
obj = Graphic()
obj.colors[index] = ...

Then you need ColorFeature.__get__ to return an object that supports __setitem__.   Does that make sense?  The descriptor pattern is definitely flexible enough to support what you want here, it's just that it's not the descriptor itself that needs to have the get/setitem methods.

IIUC the problem is that when ColorFeature.__get__returns a special object as to support indexing, this means that a user simply getting the data does not get the familar data (numpy array, I think) but a special object instead.
Not necessarily the end of the road, but then you'll need to think about an object that supports the buffer protocol so that it can easily be converted into an array. Or subclassing ndarray. Nicolas Rougier made something similar to autosync arrays to the GPU here.

Yes, if you want obj.colors itself to be a numpy array, then you'd have to return a numpy array from __get__.  I don't remember all the details of the use case here to know whether that's a problem :)

if the issue is that you're trying to have the user-facing object be a numpy array, but you also want to know whether they've updated that object... then you've got the legitimately difficult design issue that was faced in napari, where it impossible for napari to know whether a user had done in-place mutations on layer.data, because it was decided that the user object would be the numpy array.  If that's the issue you're facing here, you could try psygnal.containers.EventedObjectProxy which was designed for this sort of use case: exposing an object that is as-similar as possible to the interface your users are familiar with, but which lets you know about all mutation events: https://psygnal.readthedocs.io/en/latest/API/proxy/

Now that I understand this better, the sinc_graphic.thickness() does not look so bad after-all 😆

Not necessarily the end of the road, but then you'll need to think about an object that supports the buffer protocol so that it can easily be converted into an array. Or subclassing ndarray. Nicolas Rougier made something similar to autosync arrays to the GPU here.

Been thinking about this a bit and yes I think this would be the way to do it if GraphicFeatures were descriptors, but it adds a layer of complexity which doesn't add additional functionality if I'm understanding this correctly.

if the issue is that you're trying to have the user-facing object be a numpy array, but you also want to know whether they've updated that object... then you've got the legitimately difficult design issue that was faced in napari, where it impossible for napari to know whether a user had done in-place mutations on layer.data, because it was decided that the user object would be the numpy array. If that's the issue you're facing here, you could try psygnal.containers.EventedObjectProxy which was designed for this sort of use case: exposing an object that is as-similar as possible to the interface your users are familiar with, but which lets you know about all mutation events: https://psygnal.readthedocs.io/en/latest/API/proxy/

Thanks for the suggestion! I think that similar to the GPUData array mentioned above it adds more complexity.
I'm going to dig in and play with this over the next few days to try and maximize functionality while minizing API awkwardness :laughing:

Been digging into this more and it's really bothering me :joy: . I think the way forward is something like the GPUData array.
My option is to either implement another layer of wrappers, or submit a PEP to change Python behavior :laughing:

Here's what I've come up with that maybe makes things less awkward while also letting us keep track of in-place buffer data modifications. Having the GraphicFeatures in @properties helps specify the types for getting and setting so that helps? And also an explicit feature.data property instead of the call, i.e. feature(). Thoughts @almarklein ?
import numpy as np
import pygfx

class BufferManager:
    """Small wrapper for pygfx.Buffer"""
    def __init__(self, data, isolated_buffer):
        if isolated_buffer:
            # useful if data is read-only, ex. memmaps
            bdata = np.zeros(data.shape, dtype=data.dtype)
        else:
            # use the user's array as the buffer
            bdata = data

        self.buffer = pygfx.Buffer(bdata)

    @property
    def data(self):
        return self.buffer.data

    def __getitem__(self, key):
        return self.buffer.data[key]

    def __setitem__(self, key, value):
        pass

    def _update_range(self, start, size):
        print(f"update range: {start}, {size}")
        self.buffer.update_range(offset=start, size=size)

    def __repr__(self):
        return f"{self.__class__.__name__} buffer data:\n"\
               f"{self.data.__repr__()}"

    def cleanup_slice(self, key):
        """cleanup slice indices for setitem"""
        start = key.start if key.start is not None else 0
        stop = key.stop if key.stop is not None else self.data.shape[0]
        step = key.step if key.step is not None else 1

        return range(start, stop, step)

def parse_colors(value, n):
    """parse colors using pygfx and return RGBA array for each vertex"""
    if isinstance(value, str):
        return np.array([pygfx.Color(value)] * n)

    return value

class ColorFeature(BufferManager):
    """Manage color buffer for positions type objects"""
    def __init__(self, data: str | np.ndarray, n_colors: int, isolated_buffer: bool):
        if not isinstance(data, np.ndarray):
            # isolated buffer is only useful when data is a numpy array
            isolated_buffer = False

        colors = parse_colors(data, n_colors)

        super().__init__(colors, isolated_buffer)

    def __setitem__(self, key, value):
        if isinstance(value, BufferManager):
            # trying to set feature from another feature instance
            value = value.data

        key = self.cleanup_slice(key)

        colors = parse_colors(value, len(key))

        self.buffer.data[key] = colors

        self._update_range(key.start, key.stop - key.start)

class Graphic:
    @property
    def colors(self) -> ColorFeature:
        """color feature"""
        return self._colors

    @colors.setter
    def colors(self, colors: str | np.ndarray | ColorFeature):
        """set colors"""
        self._colors[:] = colors

    def __init__(self, data: np.ndarray, colors: str | np.ndarray | ColorFeature):
        n_vertices = data.shape[0]

        if isinstance(colors, ColorFeature):
            # allow sharing buffers between world objects, just an idea for now
            self._colors = colors
        else:
            self._colors = ColorFeature(data=colors, n_colors=n_vertices, isolated_buffer=False)

        self.world_object = pygfx.Line(
            geometry=pygfx.Geometry(positions=data, colors=self._colors.buffer),
            material=pygfx.LineMaterial(thickness=20, color_mode="vertex")
        )

Functionality:
xs = np.linspace(0, np.pi * 2, 10)
ys = np.sin(xs)
zs = np.zeros(xs.size)

data = np.column_stack([xs, ys, zs]).astype(np.float32)

g = Graphic(data=data, colors="r")

pygfx.show(g.world_object)

The repr prints the array:
>>> g.colors

ColorFeature buffer data:
array([[1., 0., 0., 1.],
       [1., 0., 0., 1.],
       [1., 0., 0., 1.],
       [1., 0., 0., 1.],
       [1., 0., 0., 1.],
       [1., 0., 0., 1.],
       [1., 0., 0., 1.],
       [1., 0., 0., 1.],
       [1., 0., 0., 1.],
       [1., 0., 0., 1.]], dtype=float32)

returns slice of buffer from ColorFeature.__getitem__
>>> g.colors[5:10]

array([[1., 0., 0., 1.],
       [1., 0., 0., 1.],
       [1., 0., 0., 1.],
       [1., 0., 0., 1.],
       [1., 0., 0., 1.]], dtype=float32)

Set value to a slice:
>>> g.colors[2:7] = "b"

update range: start: 2, size: 5

Set to entire line:
>>> g.colors = "y"

update range: start: 0, size: 10

Make another graphic to set colors from one graphic to another:
ys = np.cos(xs)
zs = np.zeros(xs.size)

data2 = np.column_stack([xs, ys, zs]).astype(np.float32)

g2 = Graphic(data=data2, colors="g")
pygfx.show(g2.world_object)

>>> g2.colors = g.colors

update range: start: 0, size: 10

I think you've found a pretty good sweet spot!
You can also use a descriptor so you can write the @property stuff using one line  colors = ColorFeatureDescriptor, and that descriptor can return the ColorFeature instance.
Idea: could also add a stub array interface to help users. They'd get a helpful error when they tried to use the buffer as an array. On the other hand, I'm not sure if there's code somewhere that checks for the presence of __array_interface__, and providing one that raises an error might be harmful ...
import numpy as np

class Foo:

    @property
    def __array_interface__(self):
        name = getattr(self, "name", "feature")
        raise RuntimeError(f"Cannot use a Feature as an array. Use {name}.data instead.")

f = Foo()
a = np.array([1,2,3])
a + f

Gets:
RuntimeError: Cannot use a Feature as an array. Use feature.data instead.

I think you've found a pretty good sweet spot!
You can also use a descriptor so you can write the @property stuff using one line colors = ColorFeatureDescriptor, and that descriptor can return the ColorFeature instance.

Great to have a comprose, I think I'll do this refactor later in the week!

Idea: could also add a stub array interface to help users. They'd get a helpful error when they tried to use the buffer as an array. On the other hand, I'm not sure if there's code somewhere that checks for the presence of __array_interface__, and providing one that raises an error might be harmful ...

Ah good idea! I think that this will work though:
image.data = image.data + np.random.rand(image.data[:].shape)
                                                   /\ slicing will invoke __getitem__

Not sure if this will work though
image.data += np.random.rand(image.data[:].shape)

or maybe it will because the @feature.setter or descriptor __set__ will call with slice(None), i.e. [:], like I showed above:
    @colors.setter
    def colors(self, colors: str | np.ndarray | ColorFeature):
        """set colors"""
        self._colors[:] = colors

I can checkout these details later, I think for now just the ability to slice and set features after creating graphics is really what we need for all our use cases and I think that's what most users will do.


Fix difficult translation for other languages

Fix https://github.com/go-gitea/gitea/pull/24051#discussion_r1163697643

.

Codecov Report

Merging #24070 (01b7b2b) into main (f521e88) will decrease coverage by 0.06%.
The diff coverage is 39.34%.

@@            Coverage Diff             @@
##             main   #24070      +/-   ##
==========================================
- Coverage   47.14%   47.08%   -0.06%
==========================================
  Files        1149     1161      +12
  Lines      151446   153912    +2466
==========================================
+ Hits        71397    72470    +1073
- Misses      71611    72936    +1325
- Partials     8438     8506      +68

Impacted Files
Coverage Δ

cmd/dump.go
0.64% <0.00%> (-0.03%)
:arrow_down:

cmd/embedded.go
0.00% <0.00%> (ø)

cmd/mailer.go
0.00% <0.00%> (ø)

cmd/manager.go
0.00% <0.00%> (ø)

cmd/manager_logging.go
0.00% <0.00%> (ø)

cmd/migrate_storage.go
5.76% <0.00%> (-0.12%)
:arrow_down:

cmd/restore_repo.go
0.00% <0.00%> (ø)

cmd/web.go
0.00% <0.00%> (ø)

models/actions/run.go
1.63% <0.00%> (-0.10%)
:arrow_down:

models/actions/runner.go
1.44% <ø> (ø)

... and 57 more

... and 143 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


T420 boot loops when trying to boot from ssd after install.

Hello! I cant seem to get my T420 to boot the MacOS install correctly. Installed using Ventura and made sure to follow all instructions exactly. Would be awesome if you had any ideas!
Log: https://pastebin.com/QXGMhTnh

I am trying Monterey sense your other repo for t440s says Ventura is still beta.

I can say that I was successfully able to load Big Sur with this OC, but the video memory is only 5MB and I am working to figure out how to increase it.
But I can confirm with marcocet that Ventura installs to disk fine, but when booting into the installer showing time to go before completion just keeps panicing.

I can say that I was successfully able to load Big Sur with this OC, but the video memory is only 5MB and I am working to figure out how to increase it.
But I can confirm with marcocet that Ventura installs to disk fine, but when booting into the installer showing time to go before completion just keeps panicing.

I can confirm I got the HD3000 working successfully with OpenCore Legacy Patcher.  I'm going to tinker with Ventura / Sonoma, but at least I have a more current macOS to have on my t420 hackintosh.  Also, the base Broadcom wifi card won't work with the typical kext.

I can confirm I got the HD3000 working successfully with OpenCore Legacy Patcher. I'm going to tinker with Ventura / Sonoma, but at least I have a more current macOS to have on my t420 hackintosh. Also, the base Broadcom wifi card won't work with the typical kext.

How did you make it work? I've successfully installed Monterey. After installing and root-patching with OpenCore Legacy Patcher, intelhd3000 is still at 4 mb.
I got a message "SIP is enabled (Required:0x802 or higher)" and "Currently booted SIP: (0x3e7)."
My hardware is also i5-2540m with a SMBIOS of MacBookPro 8,1.
Should I reinstall?


Allow extending of parameters (at least renaming them)

assume schema has this paths:

/store
/store/{store_id}
/store/{store_id}/offers
/store/{store_id}/offers/{offer_id}

and the {store_id} and {offer_id} are id's that are generated in the same manner (e.g. string($uuid))
it would be beneficial if pathId schema could be defined in the components/parameters
    "parameters": {
      "pathId": {
        "name": "id",
        "in": "path",
        "required": true,
        "schema": {
          "type": "string",
          "format": "uuid"
        },
        "example": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
      }
    }

and then later reused for the parameter definition in the paths section

Related (or duplicate of): https://github.com/OAI/OpenAPI-Specification/issues/2026

Duplicate of #2026
thanks @hkosova this is subset of the mentioned issue

sibling to #2498


Add Atlantis by @DrewKiino to Logging section

Project URL

https://github.com/DrewKiino/Atlantis
Description

Add Atlantis by @DrewKiino to Logging section
Checklist

[x] Only one project/change is in this pull request
[x] Addition in chronological order (bottom of category)
[x] Supports iOS 8 or later
[x] Has a commit from less than 2 years ago
[x] Has a clear README in English

          1 Error

  :no_entry_sign:
  Found 6 link issues

Link issues by awesome_bot

Line
Status
Link

449
301
https://github.com/melvitax/AFDateHelper redirects tohttps://github.com/melvitax/DateHelper

1204
301
https://developer.apple.com/library/ios/documentation/DeveloperTools/Reference/UIAutomationRef/ redirects tohttp://developer.apple.com/reference

1716
404
https://github.com/rob-nash/InfiniteScrollKit

1737
301
https://github.com/rob-nash/FooterPull redirects tohttps://github.com/rob-nash/PageFeedControl

1930
301
https://github.com/neonichu/ThisCouldBeUsButYouPlaying redirects tohttps://github.com/segiddins/ThisCouldBeUsButYouPlaying

2138
301
http://code.tutsplus.com/categories/ios-sdk redirects tohttps://code.tutsplus.com/categories/ios-sdk

  Generated by :no_entry_sign: danger

wow thanks guys I really appreciate the link.

@DrewKiino thanks for the really awesome library!


When you're in portrait and open keyboard on Motorola, device js detects landscape

I Have the same issue on Any version of Android

This is because Android Stock and Chrome (on Android, not iOS) resizes the window when the keyboard is open. Firefox also exhibits this behaviour.


[PROBLEMS] Solitare CE

Actual problems:

Fails at the end of the function WinMain. (usually wrong coredll declarations)
CreateWindowEx generating exception in user32.dll

Fixed:

RegisterClass fails - fixed in https://github.com/feel-the-dz3n/wcecl/commit/e3ae04e71f7efd50c8c6fc45cdfd276cbec0ee87
LoadBitmap fails - fixed in https://github.com/feel-the-dz3n/wcecl/commit/e3ae04e71f7efd50c8c6fc45cdfd276cbec0ee87
LoadString fails - fixed in https://github.com/feel-the-dz3n/wcecl/commit/e3ae04e71f7efd50c8c6fc45cdfd276cbec0ee87
Fails after the first call to LoadCursorW. https://github.com/feel-the-dz3n/wcecl/commit/dab450086d106f587e446f349a6383e2a739d992 - workaround. Fixed in https://github.com/feel-the-dz3n/wcecl/commit/e3ae04e71f7efd50c8c6fc45cdfd276cbec0ee87
Fails after the first call to InvalidateRect. Fixed in https://github.com/feel-the-dz3n/wcecl/commit/3f4695a9e34cb4ad2cc9907a97917c3e732414dc

Solitare CE in Wiki

https://discord.gg/u74Pk5e

Current status:

Current status: works fine

By the way, RANDOM function is :b:roken and after clicking "Deal" button background makes glitch

do CE apps act as 16-bit or 32-bit?

do CE apps act as 16-bit or 32-bit?

they are just usual 32 bit applications

do CE apps act as 16-bit or 32-bit?

They could be compiled for different CPU architectures, as well as normal Win32 apps.

for arm wince applications you can use wine for android. wcecl works in wine arm

for arm wince applications you can use wine for android. wcecl works in wine arm

but not tested with real applications yet 🤷♂


Load samhq model?

https://github.com/SysCV/sam-hq#model-checkpoints
https://huggingface.co/lkeab/hq-sam/tree/main
how to load sam_hq model?

Hey
https://github.com/luca-medeiros/lang-segment-anything#usage
here you can find custom checkpoint usage. havent test your case tho

How to soulve the problem??

Hey, I think I found the issue.
https://github.com/luca-medeiros/lang-segment-anything/blob/134a48a81ebd691686187ecce2d016fb327a8852/lang_sam/lang_sam.py#L76
I believe here should be:
 sam = sam_model_registry[self.sam_type](checkpoint=ckpt_path)
Mind trying and writing a PR if it works?


depends, search, sync: add --optdepends

Fixes #945

I wonder if there's some way to incorporate select_deps in aur-depends directly into the jq filter.
https://github.com/AladW/aurutils/blob/master/lib/aur-depends#L39-L47
jq --arg maybe.


I want to do custom design for camera screen.

I want to design custom camera design for barcode scan as per attached design.
Additional context

@ChandraPrakashJangid Thanks for using the plugin. At this moment plugin doesn't support custom overlays. It would be a nice idea to have something to use a custom overlay.


Tasks created on Win7 connected to AD uses DOMAIN\User syntax in the task principal information

I came across this issue when looking into an issue I have with Author not being set, as described in #659.
I just wanted to comment that every task I create with this library on Windows 7 connected to AD uses the DOMAIN\User syntax in the task principal information, which is confirmed when exporting.
This doesn't fix the registration info author being null, I'd hoped it was related.
It also doesn't fix new tasks being created with the DOMAIN\Username principal either. Though it sounds like that might be a Microsoft bug?
Originally posted by @Gwindalmir in https://github.com/dahall/TaskScheduler/issues/835#issuecomment-542440839

@Gwindalmir Will you confirm that you register a task on a Windows 7 system for that same system (not remote to another) and that when you export the task you find the Author blank the UserId value in the "DOMAIN\User" format rather than SID format?

@Gwindalmir Also, does this same thing happen if you use the Windows Task Scheduler applet?

I just spun up my Win7 image connected to an domain. I have confirmed that the exported XML stores the UserId as DOMAIN\User and that with the 2.8.16 release of the library, the Principal.Account retrieves the value correctly. The exported XML does store the Author the same way and RegistrationInfo.Author retrieves that value correctly. Please help me understand what you are seeing and how I can replicate it.

I don't know how to register a task for a remote system (didn't know it was possible), so I don't know how to confirm that.
However I've attached a screenshot from the task scheduler windows app, and the two tasks created.
I did edit out sensitive information though, I hope that's not a problem. I can send privately if needed.

task created in library.xml.txt
task created in scheduler applet.xml.txt
The scheduler service is created like this:
            m_service = new TaskService();

A new task is then created like this:
            // Create Windows task definition
            var def = m_service.NewTask();

            // Add the COM handler action, which is not user-editable
            // This COM handler is our code, which executes database calls.
            var action = (ComHandlerAction)def.Actions.AddNew(TaskActionType.ComHandler);

            // Get GUID for handler from class attribute
            var guid = (GuidAttribute)typeof(TaskRunner).GetCustomAttributes(typeof(GuidAttribute), false)[0];
            action.ClassId = new Guid(guid.Value);

            var folder = m_service.GetFolder(TaskFolder);

            // Create parent folder if it doesn't exist yet
            if (folder == null)
                folder = m_service.RootFolder.CreateFolder(TaskFolder);

            // Register new task with Windows
            task.Task = folder.RegisterTaskDefinition("task name", def);
            return task;

Oh, and Run as Admin doesn't have an effect on the Author field, though it does affect the SID in Task.SecurityDescriptor.Owner, which (obviously) determines who can edit it.

I see. The Author information is populated via the TaskDefinition.RegistrationInfo.Author property. You can set this to any string you wish. The applet sets the current user's name there automatically.

Ah, I thought it would have had a default set. I can set it then.
Thanks for looking into it!


Function code package for ARM deployment not found

Trying to deploy ARM template but it fails deploying function from the following package:
https://csciblob.blob.core.windows.net/deploy/csci-audio-dep.zip
It says Blob is disabled
@rturknett @bkeller108 @brittneek
### Tasks

Thank you for reporting this issue. Our team is currently investigating and working diligently towards a resolution. We appreciate your patience during this process.

Thank you for reporting this issue. Our team is currently investigating and working diligently towards a resolution. We appreciate your patience during this process.

This issue has been resolved.


-add minimap and script

-add minimap camera
-create minimap script to stabilise the camera and follow target
-add gameobject minimap tracker in Car for camera to look ahead the racetrack
-add minimap circular ui

Once the project was opened after this merge, I have found that some of the scenes and files are not up to date with the latest commit so I have reverted this merge.


Concepts of GenericEventMessage & DomainEventMessage gets lost

Basic information

Axon Framework version: 4.x
JDK version:  11
Complete executable reproducer if available (e.g. GitHub Repo): can be provided if required

Steps to reproduce
Using the JPAEventStorageEngine publish a GenericEventMessage directly on the EventBus; after it is persisted to the database and again loaded from the EventStore e.g. via a TrackingProcessor its type is an instance of DomainEventMessage.
Expected behaviour
The type of GenericEventMessages should be preserved after loading the event from the eventstore.
Actual behaviour
Events that get loaded via the JPAEventStorageEngine.fetchTrackedEvents will always and only be loaded as a type GenericDomainEventEntry - a subclass of DomainEventData.
Later when executing the upcasting procedure in EventUtils and creating the InitialEventRepresentation the condition that probes for DomainEventData is therefore always true:
https://github.com/AxonFramework/AxonFramework/blob/11429f924a47d51c428b080081b41bf0de5c9b2c/messaging/src/main/java/org/axonframework/serialization/upcasting/event/InitialEventRepresentation.java#L76
As every event has an aggregate identifier - no matter the original type - the aggregate identifier is not cleared in that InitialEventRepresentation so that the final decision evaluates to true as well:
https://github.com/AxonFramework/AxonFramework/blob/e59e2e4f1b9364abe61a10f535cf698970ac0180/messaging/src/main/java/org/axonframework/eventhandling/EventUtils.java#L101
leaving the objects to be obtained on that way of type GenericTrackedDomainEventMessage and thus loosing the original type of the GenericEventMessage.

Just looked at the interfaces again: AbstractEventStorageEngine - the base of all event storage engines - defines an abstract method to read events using a TrackingToken that requires to return a stream of TrackedEventData:
https://github.com/AxonFramework/AxonFramework/blob/f911231de7ec9233077061a31818f74064a7f85f/eventsourcing/src/main/java/org/axonframework/eventsourcing/eventstore/AbstractEventStorageEngine.java#L209
However the only implementation of TrackedEventData is TrackedDomainEventData which is also a subclass of DomainEventData.
In fact all existing implementations of EventData are also a subclass of DomainEventData, so unless I'm missing something this problem should occur on every storage engine.

Well spotted. Apparently, long time ago, we implemented a workaround to avoid some uniqueness constraint issues in certain relational db implementations. We did that by copying the message identifier into the aggregateIdentifier for non-Domain Event Messages. But on the way back, that equality check wasn't performed, resulting (wrongly) in DomainEventMessage for all events.
This has been fixed in c61a95bff14cda0ed3fea154747067560a670b4d and will be part of the 4.4.7 release.


How to customize a theme?

I found that only a couple of colors doesn't suit my taste, but I can't found any documentation about just changing one or two colors.
Judging by the screenshots, in my laptop the bubblegum airline theme looks brighter.

see the documentation for g:airline_theme_patch_func  here: https://github.com/vim-airline/vim-airline/blob/master/doc/airline.txt#L91

Here's the new link for 2023: https://github.com/vim-airline/vim-airline/blob/master/doc/airline.txt#L172

You could use a permalink, that would not point to master (to whatever specific commit that resolves), but just point to a very specific commit, like this one in 2023:
https://github.com/vim-airline/vim-airline/blob/3b9e149e19ed58dee66e4842626751e329e1bd96/doc/airline.txt#L172

That'll work, but I personally prefer to use those "failing" links (failing as in they will be outdated), because this way a user will immediately check that they are outdated :D and will (hopefully) search for the newest content


Subject line does not appear in FR when email delivery is set

Description
Subject line does not appear on Response page in FR when set.
Steps to reproduce

Go to "Paramètres" (settings)
Select "Protégé A" (Protected A)
Select "Recevoir les réponses par courriel" (email delivery)
Enter subject line "Ligne d'objet" in EN and FR
Go to "Responses"
See how FR subject is not appearing

Details

Browser: Edge
Operating system: Windows
Environment: App router PR
Language: French
Viewport: Tablet

Expected behaviour
FR subject line appears on response page when email subject is not empty
Screenshots or videos

Found a typo -- pushed a commit
https://github.com/cds-snc/platform-forms-client/pull/2583/commits/4bf211d9eceab94e8ed4fb8ec37bbd9b093d1578

Thanks Tim! Fix looks good!


Warmup Schedueler is Not Linear warmup, Increase in Log curve, inscrease Too Fast for big model

Would DeepSpeed Support Linear Warmup ? Thank You

@BitVoyage, thanks for your question. Can you please give more details on what you need? We are currently finishing some changes that will make it easy to integrate existing torch learning rate schedulers into DeepSpeed training.

@tjruwase I have encountered the same issue as well. I hope the learning rate can increase linearly in Warmup Stage.

@tjruwase @stas00  Is the feature being implemented? If not, I'm willing to give it a try.

@skpig, please go ahead with your plans. This will be greatly appreciated.


Workflows updated

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 99.706%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
202

Relevant Lines:
202

💛  - Coveralls


Is it possible to access the test instance done in the beforeEach?

I've read that Expose current test information to beforeEach #794, however when I access this.currentTest.fn.arguments, it turned out to be null, not done, in my code I used spooky for automatic test, and I need to bind test case done with spooky's run.complete event. So in order to handy bind the done method in every test case, I tried to make the bind in the beforeEach, so this is the situation, is it possible to access the test instance done in the beforeEach?
describe('xxx', function() {

		beforeEach(function(done){
			var itDone = this.currentTest.fn.arguments[0];
			// error: this.currentTest.fn.arguments == null
			done();
		});

		it('should open calendar', function (done) {

			var self = this;
			self.ok = true;

		});

		it('should return a 200 OK status', function(done) {

			var self = this;
			self.ok = true;

			//context.spooky.beforeTest(self, done);

			context.spooky.then(function() {

				// casper API
				this.test.assertHttpStatus(200, 'successfully received 200 OK');

			});

			context.spooky.run(function() {
				this.test.done();
			});
		});
	});

Thank you!

I'm also experiencing this. I took from this example: https://github.com/mochajs/mocha/wiki/Shared-Behaviours
(works fine)
then I added in a done() callback to the beforeEach.
https://gist.github.com/blenderer/27a4ce8ff6380af8d90a
(fails: this.user is undefined)

Hi @dickeylth, your example doesn't work as the test function hasn't yet been invoked, so no arguments have been passed or resolved. For example:
$ node
> var example = function(done) { done(); };
undefined
> example.arguments
null

You can't access the done callback function from a hook since it's not created until the Runner actually invokes/runs the runnable: https://github.com/mochajs/mocha/blob/master/lib/runnable.js#L192
I'm gonna close this since it's been a couple months, and you may have found a better solution to handling spooky tests :)


🐞 [Bug]: A lot of nodes with fake GPU cards on mainnet

Is there an existing issue for this?

[X] I have searched the existing issues

which package/s did you face the problem with?
Dashboard
What happened?
I tried deploying a VM on a node with a GPU after renting it, but it kept failing to deploy a lot, and even when it did, I couldn't ssh to the VM.
A lot of the nodes with GPUs, either one or more, are fake or have a problem deploying or connecting to them.
Steps To Reproduce
No response
which network/s did you face the problem on?
Main
version
2.4.2
Twin ID/s
No response
Node ID/s
No response
Farm ID/s
No response
Contract ID/s
No response
Relevant screenshots/screen records

Relevant log output
[
  {
    "version": 0,
    "contractId": 459604,
    "nodeId": 4771,
    "name": "vmjsu3m",
    "created":<PHONE_NUMBER>,
    "status": "ok",
    "message": "",
    "flist": "https://hub.grid.tf/tf-official-vms/ubuntu-22.04.flist",
    "publicIP": null,
    "planetary": "301:7100:a1fd:e83f:e79f:4fc8:32b5:369c",
    "myceliumIP": "",
    "interfaces": [
      {
        "network": "nws8a98",
        "ip": "<IP_ADDRESS>"
      }
    ],
    "capacity": {
      "cpu": 1,
      "memory": 2048
    },
    "mounts": [
      {
        "name": "diskh72",
        "mountPoint": "/",
        "size":<PHONE_NUMBER>0,
        "state": "ok",
        "message": ""
      }
    ],
    "env": {
      "SSH_KEY": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvmSYj5BKWkizdNyIziMfMm0CRrgq0UxFtBfBdArUMXSFkU30rtExbg6dlVFJCufw6UsCESm/QBSwlOyYKRdpsfVARcAw5G5OG0iX/22n+tYmCcWpmOJLJZaAmSynF1kwpBx3XDy40mFL6OMXjqFcU3DirbcucIL185XfAsTzrq3tDtSvmYPwbXMVMbs1ZAZValxOQNuaLty4qGD2awxVNZq4vtAzgpz3FFo3cs2g70jsGiPq/rsH+NxZHYmqvYnF/S0aPGBrbOe2yW8Cz0mqq5CzqbjYd+Qorkmt7uyJrlCBI43ky+pUuiAYeE1eJlWl5svA+ibWk5yYoa6svLUkaL1kcOMeZWLzDjL3OCrno9gglNp8zUkdSagHROoYlGgpc5fnzMGX4MKDLpHdE9xdk4NjpjB2Iq3hV3DamexXAhLEirLYv9fSilSyKVf/2kvYAa9o+NS4DgBVvXg9J54By/mAVsydD4lbpNe/2pX/WoFGCczmoQfWdHgJPYVCf2vU= harby@ahmed-Saleh-Harby"
    },
    "entrypoint": "/init.sh",
    "metadata": "",
    "description": "",
    "rootfs_size":<PHONE_NUMBER>,
    "corex": false,
    "gpu": [
      "0000:01:00.0/10de/2488"
    ]
  }
]

Investigation and Solution:
it took me sometime to set up mainnet locally as there're some relay updates that's not available on mainnet
Issue is  reproducible on same node, still investigating what's causing this

It could be a builtin GPU not an external one @muhamadazmy correct?

I don't believe this is a built in GPU. You can always google the model using the GPU identity string suffix. So by looking up 10de:2488 it showed that this device is an RTX 3070
https://www.techpowerup.com/vgabios/236303/gigabyte-rtx3070-8192-210526
Failure to deploy can be duo other reasons. It would be very useful if we can collect the deployment status after failure (get the changes, to the deployment from the node) not only the final status. This will give more details on what exactly happened.
Would be also very useful if we could collect node logs at the same time of the problem. Since i have no idea when this deployment was created it's not possible to find the corresponding node logs.
In general the node logs looks normal, and I don't believe it's fake.

After some investigation, I noticed that the deployment with gpu crashes, I opened an issue for this

https://github.com/threefoldtech/zos/issues/2349

I also noticed that the vm isn't listed but the contract doesn't get deleted

https://github.com/threefoldtech/tfgrid-sdk-ts/issues/2827

the issue happened on the node so nothing can be done here


2d view is not actually flat

It's an orthographic projection I think, which might not be always obvious. Maybe useful to at least mention in a tooltip or so? Or would it be better to use a projection that completely flattens the z-axis?
Example using the demo bag (https://webviz.io/app/?demo)

I'm not sure what your recording is showing — it looks like you have 3D mode enabled?

LOL my bad 🤦🏻 2d mode does exactly what we want it to hurray!

classic
On Fri, Mar 12, 2021 at 4:25 PM JP Posma @.***> wrote:

Closed #597 https://github.com/cruise-automation/webviz/issues/597.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cruise-automation/webviz/issues/597#event-4453180163,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB6VGJXI77BU3KGNA52MZ3DTDKWHTANCNFSM4ZBI37KQ
.

Lol rub it in Jeff 😂 Glad you're still lurking!


Text output

Hi,
If I make a get request to the api in javascript using request,  get the html of the site.
Is it possible just to get the text of it? I'm not sure how to parse the html,
My qustion is the same as this: https://github.com/chubin/wttr.in/issues/197#issue-323466573

@Mystik01 I am not sure that your question was exactly the same as #197.
If you want to have the response in the plain-text form, without ANSI, and without HTML,
you can specify A (force ANSI) and T (text) options, like this:
wttr.in/Parist?AT

Yeah, I worked that out which is why I closed it back then. But thanks


Add G4.10.x compatibility

This is the implementation of G4.10 inside WCSim.
Tests have been run to compare if there is a difference between version of Geant4 9.4.4 and Geant4 10.1.3 and results shows no differences for electrons and muons between [100-1000] MeV by setting the BERTINI physics list by default.
However, changes can be seen in muons at low energy (100 MeV) using the BINARY physics list. As a recommendation by Geant4 developers, the BERTINI physics list is set to the default one in file joboption2.mac.
Some results can be found at http://pprc.qmul.ac.uk/~zsoldos/G4results.pdf
G4Results.pdf
It's worth noticing that the GEISHA models have been depreciated and are now removed from WCSim.
The biggest changes can be found in the WCSimPhysicsList class. Other changes are mainly made to incorporate the new headers for physical constants introduced by Geant4 version 10.
This work is based on Sam Short and Francesca Di Lodovico implementation. See #178

I have corrected the indentation problem of WCSimPhysicsList class. I work with the CLion IDE and I don't have any indentation problem on my local version, don't know why he got mixed up when I commited my work. Hope the code will be more readable now.
I removed the geisha line from joboptions2.mac.
Let me know when you are happy with the reviewing on the code and I will run the same tests with electrons and muons at 100, 500 and 1000 MeV and check that we still see the same things.

You also need to change things in WCSimPhysicsMessenger.cc. The default is set to BINARY and the GEISHA option is still in there. It works fine since we explicitly set the model in jobOptions2.mac, but we want BERTINI to be the default if no other model is set.

Besides the issue with the min/max energy values, everything looks ok to me, so once you have verified these values, go ahead and do your final checks. We should aim to merge this after the software premeeting on Monday so that no one is caught off-guard when the develop branch of the code stops working.

Here you can find the validation plots comparing the orgin version of WCSim with the forked one.
I attached the pptx as well if needed to recover the plots.
G4FinalResults.pdf
G4FinalResults.pptx

So, there are many plots here, but the interesting ones are missing. For the general validation, please show 4.9.4 (binary) and 4.10 (Bertini) since this was be the change of this pull request. Have you compared your plots from #201 with these plots? Is it possible to show them overlaid so we can see there are no changes?

Apologize, that was a unfortunate oversight...
Here are the plots
G4FinalResults_Update.pdf
G4FinalResults_Update.pptx

I see that the plots are for Geant<IP_ADDRESS>. Is there any reason to not go to the latest Geant4 release - 4.10.3?
@federiconova has found that there is a radioactivity gamma cascade bug in 4.10.1. I'm unsure whether there has been a patch in 4.10.1 to fix it, or whether it is fixed in 4.10.2, but it is fixed in 4.10.3

I heard that the tests was expected with G<IP_ADDRESS> (as Yoshida-san did for pions) so I kept on with this version, but I don't recall seeing any difference by using G<IP_ADDRESS>. I can re-run some simulation if wanted, it was not clear to me to which version WCSim wants to migrate --- just that we want to migrate to G4.10

I guess the main reason is that Geant4.10.3 was just released in December and we started this process before then. Being at the cutting-edge of having the very best Geant physics is a moving target. It's unfortunate that we've already found an issue that will affect our simulations.
Moving to Geant4.10.3 might be more work than just installing a new version and running this PR with it. It looks like there are significant to the sensitive detector code, among other things.

Focussing on getting a G<IP_ADDRESS>-compatible version out sounds like the priority, so let's do that. How about @federiconova checks for the bug when this is merged, and makes an issue if it is an issue?

Do we want some pion plots posting too?

Yes, let's stick with <IP_ADDRESS>. I'm reluctant to have more delays on this upgrade, since it is holding up the fiTQun tuning. I don't love the idea of changing the recommended Geant version on a regular basis either since that also might cause issues for the fiTQun tuning (as well as just being a general pain for users and generating a lot of comparison work for developers), but it sounds like moving to Geant4.10 will solve the major problems for fiTQun and a small perturbation after that might not have a large effect. @federiconova, please try out the new release when we upgrade and make an issue outlining the problem so we can consider it for the future.
And, yes, good idea on including some pion plots. @P3tru, please include some of Yoshida-san's plots after we finalize things at the premeeting today.

Do you mean to just upload her presentation ? Or run some pions simulation at 100 and 500 MeV as for the electrons and muons ?

I think just uploading her presentation (or some select plots from it) is sufficient.

There are the final validation plot for pions, which agrees with Yoshida's plots.
pi+300MeV_90404-100103.pdf

resolved with PR #211


[Bug]: False handling of Custom base path

Describe the bug
Please have a look at https://github.com/kometenstaub/obsidian-version-history-diff/issues/9#issuecomment-1169454462
Using substring len+1, it omits the first character of the path.

I think it's all coming from using a simple and cheap trick of substring +1 instead of proper path handling. Even something like split by the delimiter(slash or backslash) would've been better.

Relevant errors (if available)
No response
Steps to reproduce
Use custom base path or not, you can see

fatal: ambiguous argument 'xxxxxx': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:

Be careful to see that the file name xxxxx's first character is omitted.
Expected Behavior
No response
Addition context
No response
Operating system
Windows
Plugin version
1.26.3

Should be fixed now.


Fix upsample converter not properly registered

Description
Partially #2665
Even though the operator is properly registered along with #2681 being applied, the operator is still decomposed into lower-level operators rather than converted using this converter, just like https://github.com/pytorch/TensorRT/issues/2665#issuecomment-1975168297. Adding aten.upsample_bilinear2d.default and aten.upsample_bilinear2d.vec to torch_disabled_decompositions doesn't help. Compiling the model under with torch.inference_mode() also doesn't help. At the end I find out that I have to remove these two lines and this line in PyTorch to bypass the decomposition and then this converter finally works.
DEBUG:torch_tensorrt.dynamo._compiler:Input graph: graph():
    %arg0_1 : [num_users=1] = placeholder[target=arg0_1]
    %upsample_bilinear2d : [num_users=1] = call_function[target=torch.ops.aten.upsample_bilinear2d.default](args = (%arg0_1, [256, 256], True, 2.0, 2.0), kwargs = {})
    return (upsample_bilinear2d,)
DEBUG:torch_tensorrt.dynamo.lowering.passes.constant_folding:Graph after constant folding:
graph():
    %arg0_1 : [num_users=1] = placeholder[target=arg0_1]
    %upsample_bilinear2d : [num_users=1] = call_function[target=torch.ops.aten.upsample_bilinear2d.default](args = (%arg0_1, [256, 256], True, 2.0, 2.0), kwargs = {})
    return (upsample_bilinear2d,)
DEBUG:torch_tensorrt.dynamo._compiler:Lowered Input graph: graph():
    %arg0_1 : [num_users=1] = placeholder[target=arg0_1]
    %upsample_bilinear2d : [num_users=1] = call_function[target=torch.ops.aten.upsample_bilinear2d.default](args = (%arg0_1, [256, 256], True, 2.0, 2.0), kwargs = {})
    return (upsample_bilinear2d,)
INFO:torch_tensorrt.dynamo._compiler:Compilation Settings: CompilationSettings(precision=torch.float16, debug=True, workspace_size=0, min_block_size=1, torch_executed_ops=set(), pass_through_build_failures=False, max_aux_streams=None, version_compatible=False, optimization_level=None, use_python_runtime=False, truncate_long_and_double=False, use_fast_partitioner=True, enable_experimental_decompositions=False, device=Device(type=DeviceType.GPU, gpu_id=0), require_full_compilation=False, disable_tf32=False, sparse_weights=False, refit=False, engine_capability=<EngineCapability.DEFAULT: 0>, num_avg_timing_iters=1, dla_sram_size=1048576, dla_local_dram_size=1073741824, dla_global_dram_size=536870912, dryrun=False, hardware_compatible=False, output_format='exported_program')

DEBUG:torch_tensorrt.dynamo.partitioning._global_partitioner:
Supported Nodes:
- torch.ops.aten.upsample_bilinear2d.default + Operator Count: 1

DEBUG:torch_tensorrt.dynamo.partitioning._global_partitioner:
All Nodes Supported

DEBUG:torch_tensorrt.dynamo._compiler:Detected support for 1 operators out of 1 in subgraph.
DEBUG:torch_tensorrt.dynamo.partitioning._adjacency_partitioner:
Number of TensorRT-Accelerated Engines Generated: 1
DEBUG:torch_tensorrt.dynamo.partitioning._adjacency_partitioner:
Supported Nodes:
- torch.ops.aten.upsample_bilinear2d.default + Operator Count: 1

DEBUG:torch_tensorrt.dynamo.partitioning._adjacency_partitioner:
All Nodes Supported

DEBUG:torch_tensorrt.dynamo._compiler:Submodule name: _run_on_acc_0
 Input shapes: [(1, 3, 128, 128)]
 graph():
    %arg0_1 : [num_users=1] = placeholder[target=arg0_1]
    %upsample_bilinear2d : [num_users=1] = call_function[target=torch.ops.aten.upsample_bilinear2d.default](args = (%arg0_1, [256, 256], True, 2.0, 2.0), kwargs = {})
    return upsample_bilinear2d
DEBUG:torch_tensorrt.dynamo.conversion.converter_utils:Node meta name arg0_1
DEBUG:torch_tensorrt.dynamo.conversion.converter_utils:Node meta name __/upsample_bilinear2d
DEBUG:torch_tensorrt.dynamo.conversion.converter_utils:Node meta name output
INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT INetwork construction elapsed time: 0:00:00.000980
INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:Build TRT engine elapsed time: 0:00:01.093642
INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT Engine uses: 0 bytes of Memory
DEBUG:torch_tensorrt.dynamo._DryRunTracker:
++++++++++++++++++++++++++++++++++++++++++++++++++ Dry-Run Results for Graph ++++++++++++++++++++++++++++++++++++++++++++++++++

The graph consists of 1 Total Operators, of which 1 operators are supported, 100.0% coverage

Compiled with: CompilationSettings(precision=torch.float16, debug=True, workspace_size=0, min_block_size=1, torch_executed_ops=set(), pass_through_build_failures=False, max_aux_streams=None, version_compatible=False, optimization_level=None, use_python_runtime=False, truncate_long_and_double=False, use_fast_partitioner=True, enable_experimental_decompositions=False, device=Device(type=DeviceType.GPU, gpu_id=0), require_full_compilation=False, disable_tf32=False, sparse_weights=False, refit=False, engine_capability=<EngineCapability.DEFAULT: 0>, num_avg_timing_iters=1, dla_sram_size=1048576, dla_local_dram_size=1073741824, dla_global_dram_size=536870912, dryrun=False, hardware_compatible=False, output_format='exported_program')

  Graph Structure:

   Inputs: List[Tensor: (1, 3, 128, 128)@float16]
    ...
    TRT Engine #1 - Submodule name: _run_on_acc_0
     Engine Inputs: List[Tensor: (1, 3, 128, 128)@float16]
     Number of Operators in Engine: 1
     Engine Outputs: Tensor: (1, 3, 256, 256)@float16
    ...
   Outputs: List[Tensor: (1, 3, 256, 256)@float16]

  ------------------------- Aggregate Stats -------------------------

   Average Number of Operators per TRT Engine: 1.0
   Most Operators in a TRT Engine: 1

  ********** Recommendations **********

   - For minimal graph segmentation, select min_block_size=1 which would generate 1 TRT engine(s)
   - The current level of graph segmentation is equivalent to selecting min_block_size=1 which generates 1 TRT engine(s)
WARNING: [Torch-TensorRT] - Using default stream in enqueue()/enqueueV2()/enqueueV3() may lead to performance issues due to additional cudaDeviceSynchronize() calls by TensorRT to ensure correct synchronizations. Please use non-default stream instead.

Type of change

Bug fix (non-breaking change which fixes an issue)

Checklist:

[x] My code follows the style guidelines of this project (You can use the linters)
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas and hacks
[ ] I have made corresponding changes to the documentation
[ ] I have added tests to verify my fix or my feature
[ ] New and existing unit tests pass locally with my changes
[ ] I have added the relevant labels to my PR in so that relevant reviewers are notified

Thanks for pointing out the above!
I looked at it and looks like in the above case the AOT trace is returning the decomposition for torch.nn.functional.interpolate
graph():
    %l_x_ : torch.Tensor [num_users=1] = placeholder[target=L_x_]
    %clone_default : [num_users=1] = call_function[target=torch.ops.aten.clone.default](args = (%l_x_,), kwargs = {})
    %interpolate : [num_users=1] = call_function[target=torch.nn.functional.interpolate](args = (%clone_default,), kwargs = {size: None, scale_factor: 2, mode
: bilinear, align_corners: True, recompute_scale_factor: None, antialias: False})
    return (interpolate,)

Post the torch.export or the AOT trace the graph decomposes into a big graph

@apbose - does the decomposition into that large set of operators you showed still occur if we remove the following two lines (but don't add anything to torch_disabled_decompositions)?
https://github.com/pytorch/TensorRT/blob/ad74a735056667726692c49a175a790647ef889e/py/torch_tensorrt/dynamo/lowering/_decomposition_groups.py#L160-L161

Hi @gs-olive, yes the above operation decomposes into the large set of ops when the two lines shown above has been commented.
Also a side doubt, when you say that they should not be added to torch_disabled_decompositions, ideally I get why you said that since that would be an unnecessary addition I suppose? Since torch_disabled_decompositions would be effective only when enable_experimental_decompositions would be True where all the core_aten_decompostions decompositions would be considered except the disabled ones? And in this case the above param is False.


Map modifiers (allows dynamic map editing/generation)

About the PR
This PR starts of as a draft so I can have advice as I go along.
On many codebases, there is some kind of dynamic map edition on map load or procedural generation. Map modifiers are classes in which downstream codebases can define simple functions to setup some stuff on the map after it loads.
This can range from making procedurally generated mining asteroid, procedural maints, or even stitch map togethers (let's say you want a spaceship map to be "pasted" on a randomly chosen planet map each round).
In current design as you will see in the PR, you define N map modifiers in your map's prototype. In the final version, I think stations should also be able to have their own map mods.
The first proof of concept of this feature is on empty map where I applied two ExampleMapModifiers. You input a tile ID and a position and this tile is placed there, simple.
# In /Prototypes/Maps/Test/test.yml
  modifiers:
      # This places a floor_steel tile at coordinates 1,1
    - !type:ExampleMapModifier
      priority: 0
      tileId: floor_steel
      tilePos: "1.0,1.0"
      # This places a floor_wood at coordinates 2,2. However, this tile placed here is not wood
    - !type:ExampleMapModifier
      priority: 1
      tileId: floor_white
      tilePos: "2.0,2.0"

MapMods don't work in mapping mode (which would be awful)
Any suggestions about the feature are welcome.
Screenshots

Changelog

:cl: Topy

add: Added map modifiers, so mappers can now dynamically generate content on their maps (procedural maints!)

What's next?

[ ] Adding mapmods to Stations
[ ] Refining the system, implement suggestions/optimizations
[ ] Creating one MapModifier as an example (maybe random maints)
[ ] Creating more map mods as examples, and usable by mappers

For some reason SaveLoadSaveTest failed on empty when the map mod was applied. I tried applying it to saltern instead and it seems to work (tests did succeed locally, waiting for github tests to run).
Here is the result on Saltern, proof of concept seems to work fine

For some reason SaveLoadSaveTest failed on empty when the map mod was applied. I tried applying it to saltern instead and it seems to work (tests did succeed locally, waiting for github tests to run).
Here is the result on Saltern, proof of concept seems to work fine

The tests fail because presumably you're modifying the map, when saveload expects to load the map, save it, and for the original and new files to be identical.

Is there a reason you're doing stuff like procgen entirely in YAML instead of as mapinit-time markers?

The tests fail because presumably you're modifying the map, when saveload expects to load the map, save it, and for the original and new files to be identical.

I triple checked how the test works, it first saves the current map, then loads it again from the file it just saved and saves it again, so the modifications are saved and reloaded and the modifiers are only applied once. I believe it failed on empty for other reasons.

Looks like we don't need a system for that, closing this.


How to build and use MIOpen with AMD Radeon RX 6700 XT (gfx1031)

issue opened on behalf of @aosewski
[Question]
Understandably gfx1031 is not officially supported, while can the user experiment with it and MIOpen?

[ ]  How to build MIOpen with AMD Radeon RX 6700 XT?
[ ]  How to use MIOpen kernels and DBs

# /opt/rocm/bin/rocminfo | grep gfx
  Name:                    gfx1031
      Name:                    amdgcn-amd-amdhsa--gfx1031

[Some Tests and Workarounds]

The default instruction for installation will be stuck at GPU_DETECTION stage, since gfx1031 is not officially supported.

One way is to use cmake -DMIOPEN_BACKEND=HIPNOGPU (not recommended, see comment below: https://github.com/ROCmSoftwarePlatform/MIOpen/issues/1309#issuecomment-982826860)
Line #139 in test/CMakeLists.txt seems to be too strict that everything other than officially supported would fail the build.

Can gfx1031 use kernels and databases created for gfx1030?

If so, how can our users experiment with them? :)

Please note that cmake -DMIOPEN_BACKEND=HIPNOGPU would disable all GPU functionality in MIOpen and should not be used for this purpose.

Can gfx1031 use kernels and databases created for gfx1030?

If so, how can our users experiment with them? :)

Once the build issue is addressed, the usual MIOpen variables to generate databases should work out of the box and need no additional code work. Namely,
MIOPEN_FIND_MODE=1 to generate the find database and MIOPEN_FIND_ENFORCE=4 to tune kernels

Carry over some questions from chat: They are valuable for every potential users of MIOpen from scratch :)

[ ] I do not see any instructions on how to add and from where get the precompiled kernels database.

[ ] Does this local tuning happens also when running tests?

[ ] Are there any examples available which showcase how to use MIOpen?

For pre-compiled kernels it seems that we only support gfx900, gfx906, gfx908:
/devel/MIOpen# apt search miopenkernels
Sorting... Done
Full Text Search... Done
miopenkernels-gfx900-56kdb/Ubuntu <IP_ADDRESS>500-56 amd64
  MIOpenKernels built using CMake

miopenkernels-gfx900-56kdb4.5.0/Ubuntu <IP_ADDRESS>500-56 amd64
  MIOpenKernels built using CMake

miopenkernels-gfx900-64kdb/Ubuntu <IP_ADDRESS>500-56 amd64
  MIOpenKernels built using CMake

miopenkernels-gfx900-64kdb4.5.0/Ubuntu <IP_ADDRESS>500-56 amd64
  MIOpenKernels built using CMake

miopenkernels-gfx906-60kdb/Ubuntu <IP_ADDRESS>500-56 amd64
  MIOpenKernels built using CMake

miopenkernels-gfx906-60kdb4.5.0/Ubuntu <IP_ADDRESS>500-56 amd64
  MIOpenKernels built using CMake

miopenkernels-gfx906-64kdb/Ubuntu <IP_ADDRESS>500-56 amd64
  MIOpenKernels built using CMake

miopenkernels-gfx906-64kdb4.5.0/Ubuntu <IP_ADDRESS>500-56 amd64
  MIOpenKernels built using CMake

miopenkernels-gfx908-120kdb/Ubuntu <IP_ADDRESS>500-56 amd64
  MIOpenKernels built using CMake

miopenkernels-gfx908-120kdb4.5.0/Ubuntu <IP_ADDRESS>500-56 amd64
  MIOpenKernels built using CMake

Please note that cmake -DMIOPEN_BACKEND=HIPNOGPU would disable all GPU functionality in MIOpen and should not be used for this purpose.

Can gfx1031 use kernels and databases created for gfx1030?

If so, how can our users experiment with them? :)

Once the build issue is addressed, the usual MIOpen variables to generate databases should work out of the box and need no additional code work. Namely,
MIOPEN_FIND_MODE=1 to generate the find database and MIOPEN_FIND_ENFORCE=4 to tune kernels

Indeed the user has the following issue with  cmake -DMIOPEN_BACKEND=HIPNOGPU
Any test throws an error "hipErrorNoBinaryForGpu: Unable to find code object for all current devices!"

Resolved by https://github.com/ROCmSoftwarePlatform/MIOpen/pull/1531

Resolved by #1531

Sir, I have seen this issue.
Was gfx1031 supporting added to the develop branch of the official repo (https://github.com/ROCmSoftwarePlatform/MIOpen)? I have a 6700xt GPU and installed rocm 5.4.2. Now I want to install MIOpen. Can I install it by building MIOpen from source?

@Looong01 please see updated answer at https://github.com/ROCmSoftwarePlatform/MIOpen/pull/1531#issuecomment-1411946795


Move rendering of flash messages to the application layout

Since the application layout is included in all pages, we have to set the display of flash messages here, and that will free us from having to set this up in views.

Fixed. See Pull Request #140


Feature Request: @request.query.* for realtime API

Hi,
Would it be possible to pass some extra query parameters to realtime API - in a similar way how it's currently possible to do it for simple records API - to be used in listRule?
Right now, I can create a collection "messages" with a field "key", and add a list API rule like this: @request.query.key = key - this way anonymous users can find all messages if they know a "key" for them, like this: pb.collection('messages').getFullList({key:"..."}).
However, this currently doesn't work for realtime API - it doesn't have a "query" option, so @request.query.key is always empty for them. In other words, realtime API works only for those records which you can see by issuing a simple pb.collection('messages').getFullList() call.
I was able to workaround it by creating a user for anyone accessing such "key-protected" records, recording all known keys into this user's "known" multi-relation field, and adding a listRule like this: @request.auth.known.key?=key. But I wonder if you might consider implementing it in the same way as it's done for records API. Thanks!

I'll consider it together with the planned expand and filter params support (most likely somewhere after the internal hooks refactoring and the JS app hooks integration).

Thanks!

Would it be possible to also specify the fields? If you just want to know that the collection changed, but plan to reconcile separately, you could request only the id field say, and avoid pulling down data you don’t need.

Experimental filter, expand, fields, custom query and headers parameters support for the realtime subscription was added in the develop branch. There are some minor polishing details that needs to be done, but if you are curious you can test it with the v0.20.0-rc prerelease. In general, it looks something like this:
// JS SDK v0.20.0-rc
pb.collection("example").subscribe("*", (e) => {
  ...
}, {
  expand: "someRelField",
  filter: "status = 'active'",
  fields: "id,expand.someRelField.*:excerpt(100)",
})

awesome! Now I have something to play with during the coming rainy weekend :)

Sorry for a delay (was a bit busy at day job), tried it in my private project - works flawlessly for me! Both expand, fields, and extra parameters. Now my code can be quite shorter :) Thanks a lot!

@mukhtharcm It is available in v0.20+. If you have troubles making it work, please file a Q&A discussion with more details about your use case and the problem you've stumbled on.

It's working!
I was confused about typescript was not showing suggestions :)


Fix build issues

Fixes some loose ends that prevent the CI build
/cc @halter73 @cesarbs

/cc @halter73 Travis CI Unhandled Exception: System.Diagnostics.Debug+DebugAssertException: Block being garbage collected instead of returned to pool might be knock on from the port exception

Alas still has the Unhandled Exception: System.Diagnostics.Debug+DebugAssertException: Block being garbage collected instead of returned to pool issue on travis
Rest of the stuff works though

That assertion was only recently enabled by https://github.com/aspnet/KestrelHttpServer/pull/623, and I haven't seen a Travis pass since that inclusion. I'll see if I can reproduce on my Ubuntu machine.

I think its on shutdown; seems to be after all the tests pass

I'm seeing that assertion too. Trying to track down where it's coming from.

All the components of the issues have been merged. Next time we see System.Diagnostics.Debug+DebugAssertException: Block being garbage collected instead of returned to pool, I'm filing an issue.

@halter73 I've seen it and I'm investigating. I've finally managed to get a repro :grin:


chore: golangci-lint: run.skip-dirs is deprecated

WARN [config_reader] The configuration option `run.skip-dirs` is deprecated, please use `issues.exclude-dirs`.

Checklist

[x] Commits are signed with Developer Certificate of Origin (DCO - learn more)

/skip-e2e


Classification data out

I assume the GUI for science teams getting classification data out of Panoptes is a ways down the line. ( @brian-c right? ) What will be the best way to extract classification data from the back-end API? "List all classifications" from the Classification Collection?

@ggdhines thoughts?

@mkosmala - Raw classification data is probably best retrieved via the /api/classifications?project_id=X for your project. You'll need to have owner access to the project to get these classifications. This is not an optimal solution and we will be reviewing how this works in the future, most likely using a different database technology and/or perhaps creating downloadable archives of classification data links.
I asked @ggdhines to contribute here because the aim is to have an aggregation engine(AE) provide you with a form of aggregation over the classification set. What kind of aggregation (consensus) algorithm are you thinking of running for your project?
Finally any data that is produced by the AE will be available on the /aggregations?workflow_id=X&subject_id=Y (subject_id is a more restrictive filter here). FYI - I've still got to document this end point.

Both the aggregation results and the individual classifications should be available to the science teams. While I hope to have useful algorithms for each project, the science teams should obviously be able to access the raw classifications to do their own analysis.

I'm happy to have raw classifications, as some of what I'll be figuring out for this project is aggregation algorithms. We have several different types of questions and each will need its own aggregation method. The "draw an oval around a tree" question can likely use whatever standard aggregating algorithm you use for drawing tasks. But I also have set up a paired question where I'm actually going to use the distribution from the answers as part of the inference as to the "right" answer.
So, in short, raw classifications are fine (and preferred) for me.

@mkosmala if we create a request to construct an archive on demand, what format is best? JSON data (serialised api response) or a csv format?
It would work like this: the project owner requests a classification data archive dump to be created. Background workers would construct a json/csv? archive, upload it to s3 as an expirable asset and then mail the requesting user with a link.

@camallen That would be perfect! As to JSON vs. csv I'm agnostic. Both are easy to parse in Python.


Document new Extensions for 2.7

References:
Background docs
UI Sync

Addressed via https://github.com/rancher/rancher-docs/pull/281.


chore: fix log lvl

Resolves https://github.com/Taraxa-project/taraxa-node/issues/2812

prnet cleaned-up successfully


Please improve build instructions to ultimately make builds reproducible

As discussed here, the compilation still fails with the provided build instructions and I couldn't figure out how to make it work. A detailed protocol of what I used and what I tried can be found here.

Agreed, hopefully this can be clarified. There's related discussion of the issue here:
https://walletscrutiny.com/posts/2019/11/bictcoinlightningwallet/

More apps joined the ranks of reproducible apps. How is this wallet doing on that front?

No reply in half a year :(

Sorry, project is not actively developed currently, the plan is to provide maintenance at most.
Currently I actively work on https://github.com/btcontract/firebird which aims to be a deterministic build when ready.

So lnwallet is defunct? https://play.google.com/store/apps/details?id=com.btcontract.wallet appears to not be live yet.

Firebird is not in fully functional state yet, not present on any app store currently.
BLW is usable and there are no plans to shut it down but there is no active development either.

@Giszmo

So lnwallet is defunct? https://play.google.com/store/apps/details?id=com.btcontract.wallet appears to not be live yet.

It's alive now and is a successor of BLW, repository is at https://github.com/btcontract/wallet. I'd be glad if you could scrutinize it, I'm ready to provide help and answer questions if needed.

Ok. Updated the relevant analysis.


Changed busybox to hello-world to match quickstart

If you follow the quick start from top to bottom this section is wrong. Out of nowhere, it switched from hello-world to busybox.  I updated the markdown and the screenshot to hello-world to follow the flow of the article.

@DarqueWarrior : Thanks for your contribution! The author, @dlepow, has been notified to review your proposed change.

@DarqueWarrior - Donovan, many thanks for submitting the fix for this inconsistency. LGTM.
#sign-off

No Problem! I will check the last one. I read Portal and CLI I will review PowerShell next.


GH-38: Add package-lock.json to excludes for additional checks (resolves #38).

See #39 for details.
@greatislander, I have cut 1.1.4-dev.20210604T105954Z.979a7d8.GH-38 with the changes proposed here.  Please give it a try, hopefully it'll work better for you.

Hi @the-t-in-rtf — this works for the package-lock.json file. Now I'm getting an error from the same check for the package.json file. I created it using npm init and added fluid-lint-all via the NPM CLI so it doesn't match our expected indentations. Do you think it's worth excluding this by default as well? I'm not sure if it's beneficial to be linting the package.json file if it's not being edited by hand otherwise whenever one uses the NPM CLI to add a dependency it will reformat it and necessitate manual re-indentation. But that may be just my preference so I'm fine to manually excluded it in my projects.

I would rather leave package.json in by default.  I vaguely remember us explicitly adding checks for package.json because of indentation and other style issues in other projects.  Although I do issue commands as well, I often use my IDE's package version checking and auto-completion, and certainly write script blocks and other stuff in a text editor, so for me it's an easy sell.
From working with the check for a while, I can say that once you take the time to indent the file (for example letting ESLint "fix" it), npm will follow your style from that point on.  So, it's a one time switch away from npm's defaults and you shouldn't have issues with content added/updated by npm install and other commands.

Or of course you can opt out of checking that file altogether in your fluidlintallrc.


NetBSD style 2-clause BSD license not detected

Running detect on https://github.com/espeak-ng/espeak-ng fails to detect COPYING.BSD2 as the bsd-2-clause license. It gives the following output for that file:
COPYING.BSD2:
  Content hash:  fe2ee5f70e7ea744625c2b8ee84fe4fb83893839
  License:       NOASSERTION
  Closest licenses:
    BSD-2-Clause similarity:        96.08%
    BSD-3-Clause similarity:        89.09%
    BSD-3-Clause-Clear similarity:  85.59%

Running diff on the BSD2 license gives:
$ cat COPYING.BSD2 | licensee diff --license=bsd-2-clause
Comparing to BSD 2-Clause "Simplified" License:
Input Length:      1228
License length:    1222
Similarity:      96.08%
diff --git a/LICENSE b/LICENSE
index 62e1dc0..e538278 100644
--- a/LICENSE
+++ b/LICENSE
@@ -1,14 +1,14 @@
redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met: {+1.+} redistributions
of source code must retain the above copyright notice, this list of conditions
and the following disclaimer. {+2.+} redistributions in binary form must reproduce
the above copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the distribution. this
software is provided by the [-copyright holders-]{+netbsd foundation, inc.+} and contributors [-"as is"-]{+as is''+} and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are
disclaimed. in no event shall the [-copyright holder-]{+foundation+} or contributors be liable for any
direct, indirect, incidental, special, exemplary, or consequential damages
including, but not limited to, procurement of substitute goods or services; loss
of use, data, or profits; or business interruption however caused and on any
theory of liability, whether in contract, strict liability, or tort including

The file text was taken from the license text in the NetBSD getopt.c file comment at https://github.com/espeak-ng/espeak-ng/blob/master/src/compat/getopt.c.

If you get rid of the mixed single/backtick quotes and replace numbered bullets with asterisks (both of which licensee arguably ought to normalize), the match is still at 97.51% (below the 98% threshold).
FWIW the NetBSD variation is treated as a distinct license in the SPDX list: https://spdx.org/licenses/BSD-2-Clause-NetBSD.html ... it might be worth looking into https://github.com/github/choosealicense.com/blob/gh-pages/CONTRIBUTING.md#adding-a-license (from which licensee vendors license texts).

On 2nd thought I don't think it would be a good idea to add BSD-2-Clause-NetBSD to choosealicense.com ... licenses with hardcoded copyright holders aren't really appropriate for copy-paste into your own project that the site encourages, and licenses with a bunch of fill-in fields are very hard for licensee to ever detect.
A couple changes have been merged which would make some other aspects of this text easier to detect (#324, #325) but I think the way it might eventually be detected for sure is to eventually have licensee utilize all of the licenses cataloged by SPDX, this being one of them, but that's a much longer term project, so I'm going to close this issue rather than wait for stalebot to do so, since I don't think there are any other near term actions to take from it.
Thanks for filing the issue @rhdunn! 🙇


Investigate whether we can check "Allow Activities" in all objects

This would facilitate activity management (tasks and events) in SAL. We want to do this, but we have to see if the packaging org will let us do that. This old idea seems to indicate this is not possible: https://success.salesforce.com/ideaView?id=08730000000BqB7AAK. We need to test it.
Also, if it does let us do it, what happens if someone has it uncheck and we check it in the package?

**lurch: add

Tracking W-019458

This effort is included in GUS W-7931287.  It is no longer being considered by the product team, and will be closed.


Drop legacy notices [notify_via_smtp]

As a followup on https://github.com/opnsense/core/issues/2912 and https://github.com/opnsense/core/pull/2907, we should drop the legacy notices which are being send out and add glue for monit where needed.
Currently there are only two spots where the notifications are (kind of relevantly) used:

carp changes https://github.com/opnsense/core/blob/702f9af71f9b2d2e6684a628a89a33daabb4db2f/src/etc/rc.syshook.d/carp/20-openvpn#L55
gateway changes in https://github.com/opnsense/core/blob/6172beb8f4cbc764fa96727405b0c51ecf9144ef/src/etc/inc/gwlb.inc#L708 and https://github.com/opnsense/core/blob/6172beb8f4cbc764fa96727405b0c51ecf9144ef/src/etc/inc/gwlb.inc#L727

The notify() function itself for filter configuration issues should remain for now, since these messages have no other clear feedback to the user.

I've closed #2408 :)

https://github.com/opnsense/core/pull/3031


Error: Exception in callback of async function: TypeError: Cannot read property 'options' of undefined

I am getting this error, coming from case !_.has(this.currentRoute.route.options, 'title'):

Which error?
What are you passing?

It looks like it happens when I make an update to the code locally.  If I refresh the page then the error goes away.

Could you please post here a code which causes the error. And a error itself

is this issue still persists?

Exception in callback of async function: TypeError: Cannot read property 'options' of undefined     at IronRouterTitle.<anonymous> (http://localhost:3000/packages/ostrio_iron-router-title.js?865f455f55d84922fec5a47342195a7f950d7f2a:83:44)     at IronRouterTitle.getTitle (http://localhost:3000/packages/ostrio_iron-router-title.js?865f455f55d84922fec5a47342195a7f950d7f2a:92:8)     at null.<anonymous> (http://localhost:3000/packages/ostrio_iron-router-title.js?865f455f55d84922fec5a47342195a7f950d7f2a:67:19)     at http://localhost:3000/packages/iron_router.js?7b16fe70bccff9182aca02afb2ccd7708ac57c64:1209:36     at _.extend.withValue (http://localhost:3000/packages/meteor.js?43b7958c1598803e94014f27f5f622b0bddc0aaf:955:17)     at hookWithOptions (http://localhost:3000/packages/iron_router.js?7b16fe70bccff9182aca02afb2ccd7708ac57c64:1208:27)     at RouteController.runHooks (http://localhost:3000/packages/iron_router.js?7b16fe70bccff9182aca02afb2ccd7708ac57c64:271:7)     at RouteController._runRoute (http://localhost:3000/packages/iron_router.js?7b16fe70bccff9182aca02afb2ccd7708ac57c64:557:8)     at Function.Route.dispatch (http://localhost:3000/packages/iron_router.js?7b16fe70bccff9182aca02afb2ccd7708ac57c64:854:18)     at route (http://localhost:3000/packages/iron_router.js?7b16fe70bccff9182aca02afb2ccd7708ac57c64:711:11)

error is:   case !_.has(this.currentRoute.route.options, 'title'): (line 83)  from this function: IronRouterTitle.prototype.getTitle = function() {

Your code sample, how do you use it within Router?

var companyName = Meteor.settings.public.ownerSettings.companyName;
    Router.configure({
        title: companyName
    });

   Router.route('/inventory', {
    name: 'inventory',
    title: companyName + ' | Inventory'
}); ```

Meteor version, your platform (Mac/Linux/Win), IronRouter and this package version, please

iron router 1.0.9
ostrio:iron-router-title            1.0.0
mac

Meteor version

meteor version <IP_ADDRESS>

hold on, I need somehow downgrade, it may take few days on my end

Hi @adriamooney
Sorry for delayed response.
This issue was fixed in v1.0.1, see commit: https://github.com/VeliovGroup/Meteor-iron-router-title/commit/58d7825ef12d7269fc458754fdb67d6d0596c787#diff-6a7f5c833a5b6b47ae00f8e910fa5b29R23
Please update your dependencies


add support for compile_commands.json

The clangd language server uses compile_commands.json to provide rich
IDE support.

If there is a way to do that without this commit please tell me :)

Sorry for the late response, been a little busy. Hmm I need to see if I can safely upgrade to scons 4, I'm still on 3. Is it possible to add a check in so we don't loose compatibility with scons 3?

I didn't find a way to test the SCons version from the script, so I try/except the env.Tool call instead...

Let's give it a try and see what people say