Feature: Prometheus Middleware to collect metrics

To improve Observability in FastStream, Prometheus Middleware support is needed.
Suggested metrics:

number of messages received;
ack quantity;
nack quantity;
reject quantity;
number of unprocessed messages (unhandled errors);
number of messages sent.

I'm guessing the usage would look something like this:
from prometheus_client import make_asgi_app, CollectorRegistry
from faststream.asgi import AsgiFastStream
from faststream.rabbit import RabbitBroker
from faststream.rabbit.prometheus import RabbitPrometheusMiddleware

registry = CollectorRegistry()

broker = RabbitBroker(middlewares=[RabbitPrometheusMiddleware(registry=registry)])

app = AsgiFastStream(
    broker,
    asyncapi_path="/docs",
    asgi_routes=[
        ("/metrics", make_asgi_app(registry=registry))
    ]
)

Solution pseudocode:
from prometheus_client import CollectorRegistry, Counter

class RabbitPrometheusMiddleware(BasePrometheusMiddleware):
    @staticmethod
    def get_broker_name() -> str:
        return "rabbit"

class BasePrometheusMiddleware:
    def __init__(self, registry: CollectorRegistry):
        self._registry = registry
        self._received_messages_counter = Counter(
            name="received_messages",
            documentation="Received messages",
            labelnames=["broker"],
        )
        self._processed_messages_with_error_counter = Counter(
            name="processed_messages_with_error",
            documentation="Processed messages with error",
            labelnames=["broker"],
        )
        self._processed_messages_without_error_counter = Counter(
            name="processed_messages_without_error",
            documentation="Processed messages without error",
            labelnames=["broker"],
        )
        self._registry.register(self._received_messages_counter)
        self._registry.register(self._processed_messages_with_error_counter)
        self._registry.register(self._processed_messages_without_error_counter)

    @staticmethod
    def get_broker_name() -> str:
        raise NotImplementedError

    def __call__(self, msg: Optional[Any]) -> "BaseMiddleware":
        return PrometheusMiddleware(
            msg=msg,
            broker_name=self.get_broker_name(),
            received_messages_counter=self._received_messages_counter,
            processed_messages_with_error_counter=self._processed_messages_with_error_counter,
            processed_messages_without_error_counter=self._processed_messages_without_error_counter,
        )

class PrometheusMiddleware(BaseMiddleware):
    def __init__(
        self,
        msg: Optional[Any] = None,
        *,
        broker_name: str,
        received_messages_counter: "Counter",
        processed_messages_with_error_counter: "Counter",
        processed_messages_without_error_counter: "Counter",
    ) -> None:
        self._broker_name = broker_name
        self._received_messages_counter = received_messages_counter
        self._processed_messages_with_error_counter = processed_messages_with_error_counter
        self._processed_messages_without_error_counter = processed_messages_without_error_counter
        super().__init__(msg)

    async def on_consume(
        self,
        msg: "StreamMessage[Any]",
    ) -> "StreamMessage[Any]":
        self._received_messages_counter.labels(broker=self._broker_name).inc()
        return await super().on_consume(msg)

    async def after_processed(self, exc_type, exc_val, exc_tb):
        if exc_val is None:
            self._processed_messages_without_error_counter.labels(broker=self._broker_name).inc()
        else:
            self._processed_messages_with_error_counter.labels(broker=self._broker_name).inc()

        return await super().after_processed(exc_type, exc_val, exc_tb)

This example produces metrics like this:
# HELP received_messages_total Received messages
# TYPE received_messages_total counter
received_messages_total{broker="rabbit"} 314560.0
# HELP received_messages_created Received messages
# TYPE received_messages_created gauge
received_messages_created{broker="rabbit"} 1.723374866369715e+09
# HELP processed_messages_with_error_total Processed messages with error
# TYPE processed_messages_with_error_total counter
processed_messages_with_error_total{broker="rabbit"} 155.0
# HELP processed_messages_with_error_created Processed messages with error
# TYPE processed_messages_with_error_created gauge
processed_messages_with_error_created{broker="rabbit"} 1.723374866552031e+09
# HELP processed_messages_without_error_total Processed messages without error
# TYPE processed_messages_without_error_total counter
processed_messages_without_error_total{broker="rabbit"} 314559.0
# HELP processed_messages_without_error_created Processed messages without error
# TYPE processed_messages_without_error_created gauge
processed_messages_without_error_created{broker="rabbit"} 1.723374866372083e+09

It’s also a good idea to add a “handler” label in metrics, but I haven’t yet been able to figure out how to get meta information about the handler. Or how can I do this better by getting it from a msg object?
I would be glad to hear any comments and suggestions on this implementation.

I would be happy to implement this feature.

and what is the best type for metrics? counter or histogram?

and what is the best type for metrics? counter or histogram?

I think, it should be counter type

I'd be cool if there was also an option for publishing the metrics to a Prometheus Pushgateway. This would benefit short-lived and ephemeral jobs. It's already supported by the Prometheus Python client.
https://prometheus.github.io/client_python/exporting/pushgateway/

registry = CollectorRegistry()

The main idea, that you should pass CollectorRegistry() object to FastStream middlewares to write metrics into it. Then you still able to use this registry object any way, that prometheus python client allows. So, I don't think FastStream reqires any additional options to support Pushgateway

I'd be cool if there was also an option for publishing the metrics to a Prometheus Pushgateway. This would benefit short-lived and ephemeral jobs. It's already supported by the Prometheus Python client.
https://prometheus.github.io/client_python/exporting/pushgateway/

The main idea, that you should pass CollectorRegistry() object to FastStream middlewares to write metrics into it. Then you still able to use this registry object any way, that prometheus python client allows. So, I don't think FastStream reqires any additional options to support Pushgateway

I conducted a small research and am adding information about the planned metrics:
metrics that I plan to implement:

number of received messages (Counter);
size of received messages (Histogram);
duration of message processing (Histogram);
number of messages in processing (Gauge);
number of processed messages (Counter);
number of sent messages (Counter);
duration of sending messages (Histogram);
size of sent messages (Histogram);
number of errors.

each metric will have labels: broker, handler (for receiving metrics), destination (for sending metrics)
message processing metrics will also have a status label
status values for processing received messages:

ACK
NACK
REJECT

status values for processing sent messages:

SUCCESS
ERROR

the error number metric will have a label error_type, its values will be type(exc).__name__


Added # comments in code, and symbols decompilation in comments.

Hello,
I made this PR because I was frustrated with non-human readable code in replicated cells.
What it does, is trying to replace numbers with symbols where applicable, and write what could be a version of the original code in end-of-line # comments.
These comments have to pass the compilation process, so the code stays editable without burdens.
Due to the nature of the symbols map, it can't be perfect, but it's working pretty well most of the time.

您好，您的邮件已收到，我会尽快回复，谢谢。
蔡陈翼

东南大学建筑学院

南京市玄武区四牌楼2号Tel：18652966253

Hi,
this seems to be a useful extension! I'll have a closer look at it tomorrow.
In the upcoming version, however, the machine code will be replaced by neural networks (and the entire simulation model modified) because they are more amenable to evolutionary adaptation and learning methods. The development can be found on the branch https://github.com/chrxh/alien/tree/version4 and is not finished yet. You can find more extensions (e.g. genetic editor) and some code refactorings there.
I'll keep the current version, which allows cells to operate on the basis of machine code, on an extra branch in the future.

I've checked the version4, but failed to compile it on linux. Maybe I'll give it another try later, when I have time to fix compilation quirks.
I love the idea of neural networks! I was a bit disappointed when I saw a NN cell type... but disabled.
I wonder if the simulation will evolve as fast in the v4 as it does in the v3, if there is no code to bootstrap some "fixed" behaviors.
I personally see the code as DNA genes (genome), and the NN as the brain connections (connectome). Both can be subject to mutations, but one is static and the other is plastic (over the lifetime of an organism).
I could elaborate on this subject, but this is not the place :)
Anyway, I'm looking forward to check the version 4!

I've tested your feature. It works well and I find it very useful. I'll merge the PR.
The Ubuntu CI is complaining but it seems nothing to do with your changes. It has not been running for over a month. There were probably changes in the toolchain.
I'll have a look at it when I find time. If you have any idea of the reason, I'd be happy to know :)
Anyway, thanks a lot for your contribution!
I'm doing the development to version 4.0 on my Windows PC, so it's quite likely that it doesn't compile on other platforms yet. I will fix that at a later time.

Sorry, I forgot to answer your question:

I wonder if the simulation will evolve as fast in the v4 as it does in the v3, if there is no code to bootstrap some "fixed" behaviors.

Most of the "fixed behavior" or configurations will be set directly as properties in the cell functions. Most important, however, is that the constructor cells contains a genome. It encodes an entire cell cluster with configurations for all its cells. There will be a new editor for genomes.
I attach a screenshot:


add batch procedure and cache option

By the way, I have given you write access to the repo, so you can make new releases if you want.

@zielmicha Glad to hear, thank you for your trust!


Slack token analyzer

I ran the unit test but my pc thought that there wasn't a module named pastepwn.analyzers.slacktokenanalyzer. the same when I ran the steamkeyanalyzer. it did work when I ran the shahashanalyzer_test.py. I don't know why it's crashing. I copied and pasted it across the app to ensure it's the same. I tested the unit tests manually with regex and the pass ones passed and the failed ones failed. I couldn't find mock test keys but modeled them after the regex. Let me know what you think.
Thanks

Pull Request Test Coverage Report for Build 188

38 of 39 (97.44%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.6%) to 67.386%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pastepwn/analyzers/tests/slacktokenanalyzer_test.py
30
31
96.77%

Totals

Change from base Build 184:
0.6%

Covered Lines:
1312

Relevant Lines:
1947

💛  - Coveralls


Cubes don't bounce off of walls coated in Repulsion Gel

Cubes bounce on Repulsion Gel coated floors, which is to be expected, however, they do not bounce off of walls coated in Repulsion Gel. Extending this behaviour to walls would make the gel's effect on cubes more consistent, and open up new puzzle possibilities.

50d697335d11af3c7ca9abd522750598192580c5


[MACOS] Present HeadlessInAppWebView using InAppBrowser

I know that on iOS/Android you can convert a HeadlessInAppWebView to a normal InAppWebView which can then be presented to the user. However, is it possible do something similar on MacOS? We need to rely on InAppBrowser instances on MacOS, but I'm not seeing any constructors that accept an InAppWebView, and the webViewController field on InAppBrowser has no setter.
Is this even technically possible (right now) given https://github.com/flutter/flutter/issues/41722 ?

Currently, there is no such feature to convert a HeadlessInAppWebView to an InAppBrowser instance.
I think you can achieve the same result using InAppBrowser with InAppBrowserSettings.hidden: true and then when needed, you can call InAppBrowser.show method.

@pichillilorenzo So when I launch InAppBrowser with these settings, onExit is called immediately and it never does anything. Setting hidden: false runs as expected.
var settings = InAppBrowserClassSettings(
        browserSettings: InAppBrowserSettings(
            hideUrlBar: true, hideToolbarTop: true, hidden: true),
        webViewSettings: InAppWebViewSettings(
            javaScriptEnabled: true, isInspectable: kDebugMode));

However, if I set hidden: false, call openUrlRequest, and then immediately call webView.hide(), it runs in the background as expected, and I can show/hide it on demand. So there must be something with the initial request when hidden: true that causes it to believe the user closed the browser.


Hot loader not working correctly

The current configuration is causing the entire page to reload after editing / webpack update. Hot loader should only update changed nodes and styles. Is there something special about the use of TypeScript compiler that prevents targeted updates via the hot loader? As an example check out how RSB handles edits and hot loader page updates.

Hi @n8sabes ,
I have tried to emulate this issue, but I see when I edit a .ts file or .css file (using Chrome) then HMR is changing only the affected component as you can see in the image:

Just to make sure we both are in the same page, please could you provide more information about the issue?
Thanks!

Closing the issue, as I am not able to reproduce and there is no more information.

I confirm the issue.
This happens to me during the second change being applied.

Hi @egorshulga ,
Thanks for your comment.
Did you change index.ts second time?
Tkae into account that HMR is working under App.tsx but not in index.ts. The reason is because index.ts is where the code related to HMR configuration is placed.
Let me know if further questions / issues.

@jquintozamora, I have changed App.tsx both times.

@jquintozamora, besides, why have you used so complicated HMR configuration? Why simple "--hot --inline" switches are not suitable for the case?

So I switched from ts-loader to awesome-typescript-loader and the problem has gone.
I have no clue what kind of a bug it was (I'm not an expert or anything).
@jquintozamora, thanks for the quick reply.
Hope, it could help anyone with the issue.

Hi @egorshulga ,
Thank you.
I just solved the issue.
I just published the changes and I have published a new version (https://github.com/jquintozamora/react-typescript-webpack2-cssModules-postCSS/releases/tag/v2.1).
The problem was the way in which is referenced module.hot.accept. That is the right way:
// Hot Module Replacement API
if (module.hot) {
module.hot.accept(() => {
const NextApp = require<{default: typeof App}>("./containers/App/App").default;
render(

,
reactContainer
);
});
}
About ts-loader, I prefer using that one, because awesome-typescript-loader gave me few problems in the past when using declarations within TypeScript.
Thanks!


Include variable name in message if decode_cf_variable raises an error

[x] Closes #7145
[x] Tests added
[x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I'm not sure if there is a better way to do this, but this is one way to address #7145.  The error message for the example now looks like:
>>> xr.decode_cf(ds)
Traceback (most recent call last):
  File "/Users/spencer/software/xarray/xarray/coding/times.py", line 275, in decode_cf_datetime
    dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar)
  File "/Users/spencer/software/xarray/xarray/coding/times.py", line 210, in _decode_datetime_with_pandas
    raise OutOfBoundsDatetime(
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, 'noleap', using pandas.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/spencer/software/xarray/xarray/coding/times.py", line 180, in _decode_cf_datetime_dtype
    result = decode_cf_datetime(example_value, units, calendar, use_cftime)
  File "/Users/spencer/software/xarray/xarray/coding/times.py", line 277, in decode_cf_datetime
    dates = _decode_datetime_with_cftime(
  File "/Users/spencer/software/xarray/xarray/coding/times.py", line 202, in _decode_datetime_with_cftime
    cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True)
  File "src/cftime/_cftime.pyx", line 605, in cftime._cftime.num2date
  File "src/cftime/_cftime.pyx", line 404, in cftime._cftime.cast_to_int
OverflowError: time values outside range of 64 bit signed integers

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/spencer/software/xarray/xarray/conventions.py", line 523, in decode_cf_variables
    new_vars[k] = decode_cf_variable(
  File "/Users/spencer/software/xarray/xarray/conventions.py", line 369, in decode_cf_variable
    var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
  File "/Users/spencer/software/xarray/xarray/coding/times.py", line 688, in decode
    dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
  File "/Users/spencer/software/xarray/xarray/coding/times.py", line 190, in _decode_cf_datetime_dtype
    raise ValueError(msg)
ValueError: unable to decode time units 'days since 0001-01-01' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/spencer/software/xarray/xarray/conventions.py", line 659, in decode_cf
    vars, attrs, coord_names = decode_cf_variables(
  File "/Users/spencer/software/xarray/xarray/conventions.py", line 534, in decode_cf_variables
    raise type(e)(f"Failed to decode variable {k!r}: {e}")
ValueError: Failed to decode variable 'invalid_times': unable to decode time units 'days since 0001-01-01' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

Thanks for taking a look @dcherian.  This should at least handle any exceptions raised by decode_cf_variable.


Drop task to install dataplane services

Installing the dataplane services that come bundled with the
dataplane-operator under config/services is not necessary, they are are
created automatically by the NodeSet controller.
There was a similar change committed to install_yamls in
commit cb9e41ef12c79629a92cea1fd0a307a6b1d8f9a1.
Signed-off-by: James Slagle<EMAIL_ADDRESS>As a pull request owner and reviewers, I checked that:

[x] Appropriate testing is done and actually running
[x] Appropriate documentation exists and/or is up-to-date:

[x] README in the role
[x] Content of the docs/source is reflecting the changes

/approve
/lgtm


Comments: remove all copyright notices at the top of files (suggestion)

Could the Copyright notices at the top of files be removed?
I feel it is superfluous as it is available here: https://github.com/spf13/hugo/blob/master/LICENSE.md
Also, if needed, an update to the license will be easier to do as it'll require only one file to be changed.
It'll also make the "developer version" in line with the "user version" as the same file will be used.
I could do this change as a pull request if you wish for it to be done.

And please use http://discuss.gohugo.io/ for questions. Start a new thread there (not one per question) in the dev section.

Thanks. I'll do in the future.
I was unsure on how to make such suggestions as I felt the discuss forum was more geared towards "end-users".


how to change pth model to onnx model

It takes 50ms to use 3D mesh without 2D effect, which is very good. It can be driven by real-time 3D animation. However, I don't know how to turn onnx, so the application is popularized. As a matter of fact, he's already extensive

https://github.com/ykk648/AI_power/blob/main/face_lib/face_3d/face_3d_coeffs.py
converted facerecon model to onnx, GPU infer 22fps，tensorrt maybe faster ( after searched all github and found nothing


imaginary lighter description

the imaginary lighter can be used to add fire to imaginary things.

Lighter? What lighter?

description needs to say what it does

this description idea can be considered based on whether the scene's dialogue ends up describing what the item does

assigned to Crookie


Potential Bug: Non Overlapping Items are shown half

If one of the entries is overlapping only a bit this results dividing the width of others too.

Example Items:
[
            {
                "name": "",
                "weekday": 5,
                "color": "#682c8b",
                "building": "",
                "room": "",
                "location": "0",
                "start": "08:00",
                "end": "08:59"
            },
            {
                "name": "",
                "weekday": 5,
                "color": "#b02e7c",
                "building": "",
                "room": "",
                "location": "1",
                "start": "09:00",
                "end": "10:00"
            },
            {
                "name": "",
                "weekday": 5,
                "color": "#d60000",
                "building": "",
                "room": "",
                "location": "2",
                "start": "10:00",
                "end": "11:00"
            },
        ]

Workaround: By reducing all items for 1 minute fixes the problem


RossTalk

I'm new to Companion, but I seem to be having trouble to get the RossTalk key transitions to work. All the other commands for XPT and cut, auto trans, etc seam to work. But when i try to turn on and off a key for the Key transitions it doesn't do anything.
Is there an area where you can input custom commands?
Thanks

Seems to be tested OK


Errors when I run ./test_drive/run.sh

Hi,
I'm new on titanDB and I would like to use gremlin-python for my application need.
I follow the instructions but got some error :

Firstly when I execute ./test_drive/run.sh, this is the error I have in the terminal

An error occurred while processing the script for language [python]. All transactions across all graphs in the session have been concluded with failure: java.util.concurrent.ExecutionException: javax.script.ScriptException: AttributeError: 'NoneType' object has no attribute 'shutdown' in

@frank1 I have seen these errors before and can help out. I would like to try and reproduce this locally to get you in the right direction.  In detail, what steps did you do up to getting these errors?  Thank you! -Denise

Here are the steps I did

Firstly I ran ./test_drive/setup.sh && ./test_drive/run.sh successfully
Secondly I edit conf/rexster-cassandra-es.xml to add gremlin-python engine
Thirdly I ran ./bin/rexster-console.sh -l python and see the first error. But this error didn't stop the rexter-console.
4)I execute  g = rexster.getGraph("graph"). I was wonderring that the result is null so I checked the output of the instruction on the README.rst and realize that it's the same.
The second error appeared when I ran g.V.has('vertex_type','doctor').count()


Sema: Fix ASTPrinter and opened element environments to handle nested pack expansion types [5.9]

Description: In a couple of places we incorrectly decided that a same-type requirement T.A == G<repeat each U> (where T.A is a scalar type) was a pack requirement, as if we were expanding the entire requirement across the each U. The ASTPrinter would prefix the requirement with repeat, and creating element archetypes would do the wrong thing and crash. Fix these places to correctly handle nested pack expansion types so that pack references bound by those expansions are not considered.

Radar: rdar://112108253

Reviewed by: @hborla

@swift-ci Please test

@swift-ci Please test

@swift-ci Please test


chore: release main

:robot: I have created a release beep boop
dev-dependencies-common: 1.1.2
1.1.2 (2023-11-27)
Bug Fixes

bump non-breaking dependencies (207a66a)

dev-dependencies-client: 1.2.2
1.2.2 (2023-11-27)
Bug Fixes

bump non-breaking dependencies (207a66a)

dev-dependencies-server: 1.2.2
1.2.2 (2023-11-27)
Bug Fixes

bump non-breaking dependencies (207a66a)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/aversini/dev-dependencies/releases/tag/dev-dependencies-common-v1.1.2 :sunflower:

:robot: Release is at https://github.com/aversini/dev-dependencies/releases/tag/dev-dependencies-client-v1.2.2 :sunflower:

:robot: Release is at https://github.com/aversini/dev-dependencies/releases/tag/dev-dependencies-server-v1.2.2 :sunflower:


Oracle database <IP_ADDRESS> is not working

I have registered with the docker image of oracle database and got below link to pull the image.
docker pull store/oracle/database-enterprise:<IP_ADDRESS>
No other tags available to see on the page. But some how I got information about <IP_ADDRESS>
With <IP_ADDRESS>, I am able to run a database and connect to it.
But with <IP_ADDRESS>, the database is not connecting.
docker service create --with-registry-auth --network=oracle_net --name Oracledb_39 -p target=1521 store/oracle/database-enterprise:<IP_ADDRESS>
docker service create --with-registry-auth --network=oracle_net --name Oracledb_39 -p target=1521 store/oracle/database-enterprise:<IP_ADDRESS>
The <IP_ADDRESS> with dbeaver client to connect:

But able to connect with <IP_ADDRESS>

Please suggest how to connect to <IP_ADDRESS> or any of 12.1.x
And also please share how to see all the tags available for oracle database docker image.

The Docker Store images are no longer available, please use 19c onwards from container-registry.oracle.com


[commonWrapper] add hosteNodes function

Pull Request Test Coverage Report for Build 63

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.3%) to 61.871%

Totals

Change from base Build 61:
0.3%

Covered Lines:
77

Relevant Lines:
125

💛  - Coveralls

package bump


Compose sequences appear to open menu

Hi, Linux user here (Ubuntu 20.04, I compiled kmonad via stack from master).
In my own layout, I have some accented characters (ä ü ö) and I would like to add more.  The point is:

without cmp-seq at the beginning, the characters of the sequences are entered alone, such as  "a "u "o;
with cmp-seq compose, a strange thing happens:

Boom, context menu gets open.
The underlying layout (XKB one) is a plain American one.
Changing the Menu key to Multi_key in the underlying layout did not fix actually fixed the issue 😊
Also, ; in shifted layer returns : why???

Changing the Menu key to Multi_key in the underlying layout ~did not fix~ actually fixed the issue blush

Since this seems fixed, I'm closing

Also, ; in shifted layer returns : why???

This is the standard shift of ; on a US layout


SHAP not working with LSTM!

Discussed in https://github.com/shap/shap/discussions/3342

Originally posted by MohamedNedal October 16, 2023
Hello, I have a trained LSTM mode for timeseries foreasting and I cannot use SHAP with it.
I have checked the tutorials and the discussions here on similar problems and I tried those suggestions but still didn't work, unfortunately. ChatGPT couldn't help either as it's something specific to the SHAP routines.
I tried to run the code in another environment with a downgraded TF version, as some suggested, but still didn't work and had more compatibility problems.
My current versions of shap and tensorflow are
TF ver. 2.3.0
shap ver. 0.41.0

Here is a minimal example code to reproduce the error:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf
tf.compat.v1.disable_v2_behavior()
tf.compat.v1.enable_eager_execution()
import shap
from tqdm import tqdm
shap.initjs()

# Define the start and end datetime
start_datetime = pd.to_datetime('2020-01-01 00:00:00')
end_datetime = pd.to_datetime('2023-12-31 23:00:00')

# Generate a DatetimeIndex with hourly frequency
date_rng = pd.date_range(start=start_datetime, end=end_datetime, freq='H')

# Create a DataFrame with random data for 7 features
num_samples = len(date_rng)
num_features = 7

# Generate random data for the DataFrame
data = np.random.rand(num_samples, num_features)

# Create the DataFrame with a DatetimeIndex
df = pd.DataFrame(data, index=date_rng, columns=[f'X{i}' for i in range(1, num_features+1)])

def windowed_dataset(series=None, in_horizon=None, out_horizon=None, delay=None, batch_size=None):
    '''
    Convert multivariate data into input and output sequences.
    Convert NumPy arrays to TensorFlow tensors.
    Arguments:
    ===========
    series: a list or array of time-series data.
    total_horizon: an integer representing the size of the input window.
    out_horizon: an integer representing the size of the output window.
    delay: an integer representing the number of steps between each input window.
    batch_size: an integer representing the batch size.
    '''
    total_horizon = in_horizon + out_horizon
    dataset = tf.data.Dataset.from_tensor_slices(series)
    dataset = dataset.window(total_horizon, shift=delay, drop_remainder=True)
    dataset = dataset.flat_map(lambda window: window.batch(total_horizon))
    dataset = dataset.map(lambda window: (window[:-out_horizon,:], window[-out_horizon:,0]))
    dataset = dataset.batch(batch_size).prefetch(1)
    return dataset

# Define the proportions for the splits (70:20:10)%
train_size = 0.7
valid_size = 0.2
test_size = 0.1

# Calculate the split points
train_split = int(len(df)*train_size)
valid_split = int(len(df)*(train_size + valid_size))

# Split the DataFrame
df_train = df.iloc[:train_split]
df_valid = df.iloc[train_split:valid_split]
df_test = df.iloc[valid_split:]

# number of input features and output targets
n_features = df.shape[1]

# split the data into sliding sequential windows
train_dataset = windowed_dataset(series=df_train.values,
                                 in_horizon=100,
                                 out_horizon=3,
                                 delay=1,
                                 batch_size=32)

valid_dataset = windowed_dataset(series=df_valid.values,
                                 in_horizon=100,
                                 out_horizon=3,
                                 delay=1,
                                 batch_size=32)

test_dataset = windowed_dataset(series=df_test.values,
                                in_horizon=100,
                                out_horizon=3,
                                delay=1,
                                batch_size=32)

input_layer = tf.keras.layers.Input(shape=(100, n_features))
lstm_layer1 = tf.keras.layers.LSTM(5, return_sequences=True)(input_layer)
lstm_layer2 = tf.keras.layers.LSTM(5, return_sequences=True)(lstm_layer1)
lstm_layer3 = tf.keras.layers.LSTM(5)(lstm_layer2)
output_layer = tf.keras.layers.Dense(3)(lstm_layer3)
model = tf.keras.models.Model(inputs=input_layer, outputs=output_layer)

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.01), loss='mse', metrics=['mae'])
history = model.fit(train_dataset, epochs=5, validation_data=valid_dataset, verbose=1)

def tensor_to_arrays(input_obj=None):
    '''
    Convert a "tensorflow.python.data.ops.dataset_ops.PrefetchDataset" object into a numpy arrays.
    This function can be used to slice the tensor objects out of the `windowing` function.
    '''
    x = list(map(lambda x: x[0], input_obj))
    y = list(map(lambda x: x[1], input_obj))

    x_ = [xtmp.numpy() for xtmp in x]
    y_ = [ytmp.numpy() for ytmp in y]

    # Stack the arrays vertically
    x = np.vstack(x_)
    y = np.vstack(y_)

    return x, y

xarr, yarr = tensor_to_arrays(input_obj=train_dataset)

# Create an explainer object
explainer = shap.DeepExplainer(model, xarr)

# Calculate SHAP values for the data
#shap_values = explainer(xarr)
shap_values = explainer.shap_values(xarr)

The error message:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-19-ab3ee3b4955f> in <module>
      1 # Calculate SHAP values for the data portion
      2 #shap_values = explainer(xarr)
----> 3 shap_values = explainer.shap_values(xarr)

~/.conda/envs/lstm/lib/python3.6/site-packages/shap/explainers/_deep/__init__.py in shap_values(self, X, ranked_outputs, output_rank_order, check_additivity)
    122             were chosen as "top".
    123         """
--> 124         return self.explainer.shap_values(X, ranked_outputs, output_rank_order, check_additivity=check_additivity)

~/.conda/envs/lstm/lib/python3.6/site-packages/shap/explainers/_deep/deep_tf.py in shap_values(self, X, ranked_outputs, output_rank_order, check_additivity)
    310                 # run attribution computation graph
    311                 feature_ind = model_output_ranks[j,i]
--> 312                 sample_phis = self.run(self.phi_symbolic(feature_ind), self.model_inputs, joint_input)
    313
    314                 # assign the attributions to the right part of the output arrays

~/.conda/envs/lstm/lib/python3.6/site-packages/shap/explainers/_deep/deep_tf.py in run(self, out, model_inputs, X)
    370
    371                 return final_out
--> 372             return self.execute_with_overridden_gradients(anon)
    373
    374     def custom_grad(self, op, *grads):

~/.conda/envs/lstm/lib/python3.6/site-packages/shap/explainers/_deep/deep_tf.py in execute_with_overridden_gradients(self, f)
    406         # define the computation graph for the attribution values using a custom gradient-like computation
    407         try:
--> 408             out = f()
    409         finally:
    410             # reinstate the backpropagatable check

~/.conda/envs/lstm/lib/python3.6/site-packages/shap/explainers/_deep/deep_tf.py in anon()
    363                     v = tf.constant(data, dtype=self.model_inputs[i].dtype)
    364                     inputs.append(v)
--> 365                 final_out = out(inputs)
    366                 try:
    367                     tf_execute.record_gradient = tf_backprop._record_gradient

~/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    778       else:
    779         compiler = "nonXla"
--> 780         result = self._call(*args, **kwds)
    781
    782       new_tracing_count = self._get_tracing_count()

~/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    821       # This is the first call of __call__, so we have to initialize.
    822       initializers = []
--> 823       self._initialize(args, kwds, add_initializers_to=initializers)
    824     finally:
    825       # At this point we know that the initialization is complete (or less

~/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    695     self._concrete_stateful_fn = (
    696         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 697             *args, **kwds))
    698
    699     def invalid_creator_scope(*unused_args, **unused_kwds):

~/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2853       args, kwargs = None, None
   2854     with self._lock:
-> 2855       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2856     return graph_function
   2857

~/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3211
   3212       self._function_cache.missed.add(call_context_key)
-> 3213       graph_function = self._create_graph_function(args, kwargs)
   3214       self._function_cache.primary[cache_key] = graph_function
   3215       return graph_function, args, kwargs

~/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3073             arg_names=arg_names,
   3074             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3075             capture_by_value=self._capture_by_value),
   3076         self._function_attributes,
   3077         function_spec=self.function_spec,

~/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    984         _, original_func = tf_decorator.unwrap(python_func)
    985
--> 986       func_outputs = python_func(*func_args, **func_kwargs)
    987
    988       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

~/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    598         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    599         # the function a weak reference to itself to avoid a reference cycle.
--> 600         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    601     weak_wrapped_fn = weakref.ref(wrapped_fn)
    602

~/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

AttributeError: in user code:

    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/shap/explainers/_deep/deep_tf.py:247 grad_graph  *
        out = self.model(shap_rAnD)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py:985 __call__  **
        outputs = call_fn(inputs, *args, **kwargs)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py:386 call
        inputs, training=training, mask=mask)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py:508 _run_internal_graph
        outputs = node.layer(*args, **kwargs)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/keras/layers/recurrent.py:663 __call__
        return super(RNN, self).__call__(inputs, **kwargs)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py:985 __call__
        outputs = call_fn(inputs, *args, **kwargs)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/keras/layers/recurrent_v2.py:1183 call
        runtime) = lstm_with_backend_selection(**normal_lstm_kwargs)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/keras/layers/recurrent_v2.py:1559 lstm_with_backend_selection
        function.register(defun_gpu_lstm, **params)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/function.py:3241 register
        concrete_func.add_gradient_functions_to_graph()
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/function.py:2063 add_gradient_functions_to_graph
        self._delayed_rewrite_functions.forward_backward())
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/function.py:621 forward_backward
        forward, backward = self._construct_forward_backward(num_doutputs)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/function.py:669 _construct_forward_backward
        func_graph=backwards_graph)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py:986 func_graph_from_py_func
        func_outputs = python_func(*func_args, **func_kwargs)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/eager/function.py:659 _backprop_function
        src_graph=self._func_graph)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/ops/gradients_util.py:669 _GradientsHelper
        lambda: grad_fn(op, *out_grads))
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/ops/gradients_util.py:336 _MaybeCompile
        return grad_fn()  # Exit early
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/tensorflow/python/ops/gradients_util.py:669 <lambda>
        lambda: grad_fn(op, *out_grads))
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/shap/explainers/_deep/deep_tf.py:378 custom_grad
        out = op_handlers[type_name](self, op, *grads) # we cut off the shap_ prefex before the lookup
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/shap/explainers/_deep/deep_tf.py:667 handler
        return linearity_with_excluded_handler(input_inds, explainer, op, *grads)
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/shap/explainers/_deep/deep_tf.py:674 linearity_with_excluded_handler
        assert not explainer._variable_inputs(op)[i], str(i) + "th input to " + op.name + " cannot vary!"
    /home/mnedal/.conda/envs/lstm/lib/python3.6/site-packages/shap/explainers/_deep/deep_tf.py:224 _variable_inputs
        out[i] = t.name in self.between_tensors

    AttributeError: 'TFDeep' object has no attribute 'between_tensors'

I would really appreciate your inputs. Thanks in advance!

Thanks for the report and especially for the reproducing example. I know this is a longstanding issue and I'll look into this in the coming weeks.

So, I have looked into this and am working on a solution. The problem is that with tf version 2 it is not as easy to extract the graph of the model (since tensorflow is just building the graph with lazy execution). I found a solution to catch the graph if we do a forward pass through the model with some example data. But getting this data reliably (conversion of keras tensors to such a form that they can be used to run lazy tf models) is a bit tricky. Will keep you updated if I make further progress.

Hi @MohamedNedal. I have encountered the exact same issue in my project. Have you attempted to use any other feature importance algorithm that is compatible with an LSTM?

Hi @lordegeology , I'm actually not familiar with such tools. I used to find the feature importance via the correlation matrix, but it's not so reliable and limited as it doesn't show the feature importance over time.

Been having the same issue for a while, and a solution would be greatly appreciated. Thanks @CloseChoice !

I found that the LSTM of a lower version of pytorch can get results through shap, but there will still be a warning about an unrecognized model.pyTorch==1.11.0

I added a draft PR where the code at least runs through. I suspect that there is still something wrong in how we handle new operations (so basically we ignore them in the custom gradient calculation, which might be wrong), so use the results with caution. Any test cases for this are highly appreciated so feel free to start a review and post them there! Together we can do this!


Handle boards

Hi,
I just added APIs to handle Boards in Gitlab.
Thx.

Nice one! Thanks for this @thmatew!

@thmatew  I was a little too quick I guess... Updated your work in this commit f9ff9428caa3edd289d6b20cb29011ef2552e4b0 to make it fit in nicely.


Created an AUR package for installing fwup on Arch Linux

The package can be viewed here. I can take responsibility to maintain this package.
Any feedback is welcome. Thanks!

Closing because @fhunleth added a similar README line.

How did I not see this? I'm so sorry I missed it. Apparently I received an email with your comment closing the PR and not the one opening it.
Anyway, thank you so much for creating the AUR package!

No worries! Just happy to help.