Permit access to select environment where net-cap-bind support active

Here's a first version of my PR for issue #21770.
Hopefully the overall approach makes sense. Very much welcome feedback!
User Scenario
A user executing an application on linux that utilizes open ssl libraries cannot
make use of quantum-safe encryption algorithms, using OPENSSL_MODULES and OPENSSL_CONF, if that application needs the ability
to bind to a port below 1024 by enabling CAP_NET_BIND_SERVICE.
Setting this privilege is common in server environments to run on well known ports.
Technical Problem Summary
When using GLIBC > 2.17 (linux?), OpenSSL uses the secure_getenv call in glibc to retrieve all environment variables. If the process is setuid/setgid, OR has additional capabilities set, this call will return NULL.
This is intentional, since we do not want a non-priviliged user to inject malicious code, or otherwise compromise the configuration of the application.
Proposed change of behavior
The exact test as to whether to return NULL is slightly modified, in that it is only applied IF, and ONLY IF, the ONLY additional capability granted is CAP_NET_BIND_SERVICE.
Risks
The risks to evaluate for this solution include:

An error in the design such that the intended test allows for malicious configuration.
An error in the implementation such that the code does not implement the intended design.
Poor documentation, such that the feature is unknown, or unclear to use
Library/kernel changes affecting the implementation behaviour in future

Proposed Implementation in this PR
The intent is to keep checks as safe as possible, and make minimal changes to existing code.
The changes are protected by compile time flags, and the restriction is only lifted for SELECT environment variables, with ONLY the specified capability.

Add new compile-time define OPENSSL_NETCAP_ALLOW_ENV, so that the modified code is only compiled if specified at compile time ie using CFLAGS=-DOPENSSL_NETCAP_ALLOW_ENV (all)
If set, and we are on linux (linux defined) uses new function restricted_getenv() to retrieve specified environment variable.  Question - is the preprocessor check here appropriate? (getenv.c)
restricted_getenv``  uses a modified **OpenSSL_issetugid** (below) to check we aren't in a secure mode. This is a change in behaviour from previously, where this function was only used if glibc was too old to support secure_getenv` (getenv.c)
OpenSSL_issetugid is modified slightly so that it queries the processes's capabilities if we are in a secure mode. If the only capability set is for bind, we now return false (0) . This follows the intent of the function to indicate we need to act very securely in general ie for setuid etc., but allows for this proposed loosening of the check. (uid.c)
restricted_getenv then checks if the requested environment variable is a member of a configured (hardcoded) list (getenv.c)
If so, it is returned directly. If not it reverts to using secure_getenv.

Build
For a 'local' development build to support testing, making use of a pause check at startup (see below).
Install into a local directory (not overwriting system version)
./Configure --prefix=/home/jonesn/opt/openssl --openssldir=/home/jonesn/usr/local/ssl

Build with appropriate flags:
make CFLAGS=-DOPENSSL_NETCAP_ALLOW_ENV -g -Wl,-rpath,/home/jonesn/src/openssl

Alternative implementations considered
There are many ways of setting up the preprocessor checks, and exact code sequence  -- and the flow could likely be made simpler, but it does result in more changes, so more risk.
There are also some utility functions in libcap. However it is hard to check for only a select, permitted capability with the limited API. Additionally it would introduce an additional development and runtime dependency, albeit a fairly standard one.
Future enhancements
Making this feature available by default
OpenSSL libraries are used by many applications. We need to make it easier for users of these applications to enable quantum-safe cryptography schemes by deployment/app configuration, and not requiring a rebuild of openssl libraries.
To do this, I would like to suggest that if this PR is acceptable to the reviewers, that either within this PR, or in a later PR we can either

remove the build flag entirely
default the value to being enabled

Runtime configuration of this feature

Do we need to allow this feature to be disabled at runtime, in the configuration
Can we make the list of environment variables permitted configurable?

Both have one notable issue - where to store the config, especially if we already allow OPENSSL_CONF to be passed

Is it safe to permit all?

Testing
Recommendations on the best way to test welcomed.
So far testing has been done by using the 'openssl' command, and validating if environment variables are passed by inspection of command output, or by using a debugger (remotely attached so as not to affect permissions).
For example
setcap cap_net_bind_service=+ep  apps/openssl
export OPENSSL_CONF=/home/jonesn/etc/openssl.cnf
./apps/openssl engine -c qatengine

Capability was added to the apps/openssl executable where needed for the test using setcap cap_net_bind_service=+ep  apps/openssl. Other capabilities can also be set to verify these environment variables are no longer allowed to pass.
It was not possible to test (in Visual Studio Code) using gdb directly, as if run under debug no extra capabilities set set. Neither can the process be run as root, since then it has too many permissions. Instead the apps/openssl app was run, and then attached to remotely, using gdb running as root.
To do this the openssl command was paused at startup using this at the top of apps/openssl.c:
#ifdef OPENSSL_TEST_PAUSE_FOR_ATTACH
#include <signal.h>
#endif

then later, at the beginning of main:
#if defined(OPENSSL_TEST_PAUSE_FOR_ATTACH)
    if(getenv("OPENSSL_TEST_PAUSE_FOR_ATTACH"))
    {
        raise(SIGSTOP);
    }
#endif

This can be used by defining appropriate values at build/runtime.
If this is useful I am happy to submit further changes, but it was very much just
a mechanism to support my testing given the complexity of capabilities. It does not form part of the current PR. Similarly I can add more docs on visual studio code build/test if it helps.
I'll continue testing, to verify that the user scenario works, i.e. using the  liboqs provider & libraries:

https://github.com/open-quantum-safe/liboqs
https://github.com/open-quantum-safe/oqs-provider

Documentation
There should be some end-user documentation of this feature. NOTES-UNIX.md may be one location?
Should information be added elsewhere?

Here's a first version of my PR for issue #21770.
Hopefully the overall approach makes sense. Very much welcome feedback!
User Scenario
A user executing an application on linux that utilizes open ssl libraries cannot
make use of quantum-safe encryption algorithms, using OPENSSL_MODULES and OPENSSL_CONF, if that application needs the ability
to bind to a port below 1024 by enabling CAP_NET_BIND_SERVICE.
Setting this privilege is common in server environments to run on well known ports.
Technical Problem Summary
When using GLIBC > 2.17 (linux?), OpenSSL uses the secure_getenv call in glibc to retrieve all environment variables. If the process is setuid/setgid, OR has additional capabilities set, this call will return NULL.
This is intentional, since we do not want a non-priviliged user to inject malicious code, or otherwise compromise the configuration of the application.
Proposed change of behavior
The exact test as to whether to return NULL is slightly modified, in that it is only applied IF, and ONLY IF, the ONLY additional capability granted is CAP_NET_BIND_SERVICE.
Risks
The risks to evaluate for this solution include:

An error in the design such that the intended test allows for malicious configuration.
An error in the implementation such that the code does not implement the intended design.
Poor documentation, such that the feature is unknown, or unclear to use
Library/kernel changes affecting the implementation behaviour in future

Proposed Implementation in this PR
The intent is to keep checks as safe as possible, and make minimal changes to existing code.
The changes are protected by compile time flags, and the restriction is only lifted for SELECT environment variables, with ONLY the specified capability.

Add new compile-time define OPENSSL_NETCAP_ALLOW_ENV, so that the modified code is only compiled if specified at compile time ie using CFLAGS=-DOPENSSL_NETCAP_ALLOW_ENV (all)
If set, and we are on linux (linux defined) uses new function restricted_getenv() to retrieve specified environment variable.  Question - is the preprocessor check here appropriate? (getenv.c)
restricted_getenv``  uses a modified **OpenSSL_issetugid** (below) to check we aren't in a secure mode. This is a change in behaviour from previously, where this function was only used if glibc was too old to support secure_getenv` (getenv.c)
OpenSSL_issetugid is modified slightly so that it queries the processes's capabilities if we are in a secure mode. If the only capability set is for bind, we now return false (0) . This follows the intent of the function to indicate we need to act very securely in general ie for setuid etc., but allows for this proposed loosening of the check. (uid.c)
restricted_getenv then checks if the requested environment variable is a member of a configured (hardcoded) list (getenv.c)
If so, it is returned directly. If not it reverts to using secure_getenv.

Build
For a 'local' development build to support testing, making use of a pause check at startup (see below).
Install into a local directory (not overwriting system version)
./Configure --prefix=/home/jonesn/opt/openssl --openssldir=/home/jonesn/usr/local/ssl

Build with appropriate flags:
make CFLAGS=-DOPENSSL_NETCAP_ALLOW_ENV -g -Wl,-rpath,/home/jonesn/src/openssl

Alternative implementations considered
There are many ways of setting up the preprocessor checks, and exact code sequence  -- and the flow could likely be made simpler, but it does result in more changes, so more risk.
There are also some utility functions in libcap. However it is hard to check for only a select, permitted capability with the limited API. Additionally it would introduce an additional development and runtime dependency, albeit a fairly standard one.
Future enhancements
Making this feature available by default
OpenSSL libraries are used by many applications. We need to make it easier for users of these applications to enable quantum-safe cryptography schemes by deployment/app configuration, and not requiring a rebuild of openssl libraries.
To do this, I would like to suggest that if this PR is acceptable to the reviewers, that either within this PR, or in a later PR we can either

remove the build flag entirely
default the value to being enabled

Runtime configuration of this feature

Do we need to allow this feature to be disabled at runtime, in the configuration
Can we make the list of environment variables permitted configurable?

Both have one notable issue - where to store the config, especially if we already allow OPENSSL_CONF to be passed

Is it safe to permit all?

Testing
Recommendations on the best way to test welcomed.
So far testing has been done by using the 'openssl' command, and validating if environment variables are passed by inspection of command output, or by using a debugger (remotely attached so as not to affect permissions).
For example
setcap cap_net_bind_service=+ep  apps/openssl
export OPENSSL_CONF=/home/jonesn/etc/openssl.cnf
./apps/openssl engine -c qatengine

Capability was added to the apps/openssl executable where needed for the test using setcap cap_net_bind_service=+ep  apps/openssl. Other capabilities can also be set to verify these environment variables are no longer allowed to pass.
It was not possible to test (in Visual Studio Code) using gdb directly, as if run under debug no extra capabilities set set. Neither can the process be run as root, since then it has too many permissions. Instead the apps/openssl app was run, and then attached to remotely, using gdb running as root.
To do this the openssl command was paused at startup using this at the top of apps/openssl.c:
#ifdef OPENSSL_TEST_PAUSE_FOR_ATTACH
#include <signal.h>
#endif

then later, at the beginning of main:
#if defined(OPENSSL_TEST_PAUSE_FOR_ATTACH)
    if(getenv("OPENSSL_TEST_PAUSE_FOR_ATTACH"))
    {
        raise(SIGSTOP);
    }
#endif

This can be used by defining appropriate values at build/runtime.
If this is useful I am happy to submit further changes, but it was very much just
a mechanism to support my testing given the complexity of capabilities. It does not form part of the current PR. Similarly I can add more docs on visual studio code build/test if it helps.
I'll continue testing, to verify that the user scenario works, i.e. using the  liboqs provider & libraries:

https://github.com/open-quantum-safe/liboqs
https://github.com/open-quantum-safe/oqs-provider

Documentation
There should be some end-user documentation of this feature. NOTES-UNIX.md may be one location?
Should information be added elsewhere?

Close/reopen to trigger CLA check

OTC: We cannot accept this PR. We do not want to hardcode exemption for specific environment variables depending on Linux-specific syscalls or Linux capability APIs. We would consider a mechanism which allows an application to specify a list of environment variable names which should not be routed via secure getenv.

OTC: We cannot accept this PR. We do not want to hardcode exemption for specific environment variables depending on Linux-specific syscalls or Linux capability APIs. We would consider a mechanism which allows an application to specify a list of environment variable names which should not be routed via secure getenv.

Thanks for the clarification and feedback. I will work on a new/updated PR which allows for the environment variables to be configured at build time, and continue the discussion in #21770


Add word-wrap for article-body-container

This is what we used in Journal

@elifesciences/epp-tech-team Can someone test and merge this when the CI clears please?

Tested the branch and compared against prod, the fix looks good to me


resource management experiment in dev

This should only affect dev and let me look into why it's falling over.

Ha, this broken because of a bug in inheritance.


Issue in php artisan queue:work in updated framework. [v5.8.30]

As soon as i updated my laravel project to latest framework the supervisor stopped working.
For testing if i am sure about the issue i created a new project by
composer create-project --prefer-dist laravel/laravel testing

and run php artisan queue:work --sleep=60 -v and i got the same error.
  at /Users/hemant/Projects/laravel/testing/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:144
    140|         // process if it is running too long because it has frozen. This uses the async
    141|         // signals supported in recent versions of PHP to accomplish it conveniently.
    142|         pcntl_signal(SIGALRM, function () use ($job, $options) {
    143|             $this->markJobAsFailedIfWillExceedMaxAttempts(
  > 144|                 $job->getConnectionName(), $job, (int) $options->maxTries, $this->maxAttemptsExceededException($job)
    145|             );
    146|
    147|             $this->kill(1);
    148|         });

Hi there,
Thanks for reporting but it looks like this is a question which can be asked on a support channel. Please only use this issue tracker for reporting bugs with the library itself. If you have a question on how to use functionality provided by this repo you can try one of the following channels:

Laracasts Forums
Laravel.io Forums
StackOverflow
Discord
Larachat
IRC

However, this issue will not be locked and everyone is still free to discuss solutions to your problem!
Thanks.


OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 79.35 GiB total capacity; 77.18 GiB already allocated; 57.19 MiB free; 77.97 GiB reserved in total by PyTorch)

Ran into CUDA OOM issue during fine tuning
File "/opt/conda/lib/python3.9/site-packages/bitsandbytes/nn/modules.py", line 336, in _save_to_state_dict
self.weight.data = undo_layout(self.state.CxB, self.state.tile_indices)
File "/opt/conda/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 96, in undo_layout
outputs = torch.empty_like(tensor) # note: not using .index_copy because it was slower on cuda
torch.cuda.OutOfMemoryError: CUDA out of memory.
Any ideas to fix this?

This happened to me as well with the 40B model. This error only occurs when trying to save a checkpoint. Tried to save the model after all steps instead of every 50 steps - still got the error.

Any luck?

I am not sure if it is the same issue I had before, but please check this https://github.com/rhulha/lora/issues/1

Thanks @angelovAlex . Applied the patch in that issue. Now running into a similar issue at a different line
File "/root/.conda/envs/falcontune/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1815, in state_dict
self._save_to_state_dict(destination, prefix, keep_vars)
File "/root/.conda/envs/falcontune/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 330, in _save_to_state_dict
weight_clone = self.weight.data.clone()
^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 79.35 GiB total capacity; 75.73 GiB already allocated; 127.19 MiB free; 77.90 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF


Viewer fix: invalid html_widget resource paths

Resolves #665
Adds a mutation observer to the webview body, which watches for changes in the DOM. Specifically, the filescheme vscode-webview:// is tested for 'URLness' and (if true) is converted to an https:// request.
This PR also defines some (minimal) content security policies for the webview.
Screenshot

How can I check this pull request?
Prior to the PR, the following led to a blank viewer page, which should now be fixed:
library(leaflet)
leaflet() |> addTiles()

Dynamic loading of resources is supported too:
library(mapview)
mapview(breweries)

Thanks for working on this! Works nicely already.
When I test the mapview example, it actually works, but I found the following error in the log:
fgb.js:103 Refused to connect to 'vscode-webview://8fe576af-ef9f-4237-9417-cd8966ff7eef/lib/breweries-0.0.1/breweries_layer.fgb' because it violates the following Content Security Policy directive: "default-src https: data: filesystem:". Note that 'connect-src' was not explicitly set, so 'default-src' is used as a fallback.

LeafletWidget.methods.addFlatGeoBuf @ fgb.js:103
fgb.js:103 Refused to connect to 'vscode-webview://8fe576af-ef9f-4237-9417-cd8966ff7eef/lib/breweries-0.0.1/breweries_layer.fgb' because it violates the document's Content Security Policy.
LeafletWidget.methods.addFlatGeoBuf @ fgb.js:103
fgb.js:103 Uncaught (in promise) TypeError: Failed to fetch
    at e.LeafletWidget.methods.addFlatGeoBuf (fgb.js:103)
    at Object.doRenderValue (leaflet.js:812)
    at Object.renderValue (leaflet.js:714)
    at Object.renderValue (htmlwidgets.js:886)
    at htmlwidgets.js:653
    at Array.forEach (<anonymous>)
    at forEach (htmlwidgets.js:55)
    at htmlwidgets.js:576
    at Array.forEach (<anonymous>)
    at forEach (htmlwidgets.js:55)

Not sure if it has any impact on the functionality. Anything we should do about this or we could ignore it?

Thanks for working on this! Works nicely already.
When I test the mapview example, it actually works, but I found the following error in the log:
fgb.js:103 Refused to connect to 'vscode-webview://8fe576af-ef9f-4237-9417-cd8966ff7eef/lib/breweries-0.0.1/breweries_layer.fgb' because it violates the following Content Security Policy directive: "default-src https: data: filesystem:". Note that 'connect-src' was not explicitly set, so 'default-src' is used as a fallback.

LeafletWidget.methods.addFlatGeoBuf @ fgb.js:103
fgb.js:103 Refused to connect to 'vscode-webview://8fe576af-ef9f-4237-9417-cd8966ff7eef/lib/breweries-0.0.1/breweries_layer.fgb' because it violates the document's Content Security Policy.
LeafletWidget.methods.addFlatGeoBuf @ fgb.js:103
fgb.js:103 Uncaught (in promise) TypeError: Failed to fetch
    at e.LeafletWidget.methods.addFlatGeoBuf (fgb.js:103)
    at Object.doRenderValue (leaflet.js:812)
    at Object.renderValue (leaflet.js:714)
    at Object.renderValue (htmlwidgets.js:886)
    at htmlwidgets.js:653
    at Array.forEach (<anonymous>)
    at forEach (htmlwidgets.js:55)
    at htmlwidgets.js:576
    at Array.forEach (<anonymous>)
    at forEach (htmlwidgets.js:55)

Not sure if it has any impact on the functionality. Anything we should do about this or we could ignore it?

Very weird, this doesn't occur on WSL, but does on Linux. Removing the CSP policies shows a fetch attempt to a non-existent filepath for some reason... I think it does effect the blue dots on the map, but is fixed if you type mapviewOptions(fgb = FALSE). In looking into this I came across this thread which appears to suggest this isn't just a VS Code thing.

Very weird, this doesn't occur on WSL, but does on Linux. Removing the CSP policies shows a fetch attempt to a non-existent filepath for some reason... I think it does effect the blue dots on the map, but is fixed if you type mapviewOptions(fgb = FALSE). In looking into this I came across this thread which appears to suggest this isn't just a VS Code thing.

Do you think if there is anything we could do about it?

Do you think if there is anything we could do about it?

I found this, so I guess it might be something that can be fixed. I'll have a look


Cacheable logo's on API

Serve static assets with an efficient cache policy.
A long cache lifetime can speed up repeat visits to your page.

Logos loaded via cloufdfront (PSD) now instead of our image server.


pg 44 class RoutingTest should be ControllerTest

class ControllerTest extends AbstractHttpControllerTestCase
I don't think anyone will get stuck on this, just an fyi. I emailed you about this but I guess I forgot to log it here.
Thanks.

Got that sorted it. It'll be included in the next book release.


Ensure rule.resolve doesn't swallow errors

We're moving to graphql-shield from an @auth directive to take advantage of more complex per-field and per-type validations offered, however we've found runtime errors are swallowed inside a rule (they don't make it to options.fallbackError):
e.g.
const isAdmin = rule()((parent, args, context: Context) => {
    return context.thisDoesNotExist.bang; // TypeError: Cannot read property 'bang' of undefined
});

The above rule correctly returns an unauthorized error to the client which is great, but we lose the underlying runtime error so  our logging service never knows there was a runtime error, and instead thinks the user was merely unauthorised.
We're therefore having to effectively wrap all our rules in a catch like this which is boilerplate we'd like to avoid -- and obviously that boilerplate could error at which point it's lost again:
const isAdmin = rule()((parent, args, context: Context) => {
   try {
     return context.thisDoesNotExist.bang; // TypeError: Cannot read property 'bang' of undefined
   } catch(ex) {
     return ex;
   }
});

(I can fix the tests if you're happy for this change to go in)

Okay I actually just tested this and although the error is no longer swallowed, it still doesn't get passed into options.fallbackError so no mapping can be done in there...perhaps I'm missing something re: error handling; is there another way to distinguish between a bug / failure vs an authentication / authorisation error?

We ended up wrapping rule ourselves which holds the boilerplate / manages the errors we want to return, e.g.
import { rule as shieldRule, IRule } from 'graphql-shield';

// Like `rule` from graphql-shield but a) does not support caching because that's handled
// by dataloader / memo and b) does not swallow errors.
export const rule = (fn: IRuleFunction): IRule =>
  shieldRule({ cache: 'no_cache' })(async (...args) => {
    try {
      const result = await fn(...args);
      if (result === true) {
        return true;
      }
      return new AuthenticationError('Unauthorized');
    } catch (ex) {
      return new ApolloVError(ex);
    }
  });


RFC: Controllable task properties

Buckle up, it's a long one!
I'd like to suggest the use of a "properties" payload (or a Karton Task's class member) with some common properties one can define for a task in karton.
This properties payload is currently used in Checkpoint research in our production kartons and mwdb. We think it will be wise to use it across CERTPL's existing kartons.
*Note that we will use Payload as an implementation example, but it can also be anything else.
The idea
Encourage users to add a payload (if part of karton itself, than a dictionary item in KartonTask will be better) called properties to their tasks. These properties will instruct several Kartons (developed by CERT.PL) how to treat the task or payloads inside it. The properties we found useful are the following:

volatile: boolean, defaults to false
emulate: boolean, defaults to true
force_analyze: boolean, defaults to false
force_emulate: boolean, defaults to false

Usage
I think the best way to show why it might be a good idea is from examples of actual, running kartons.
volatile
The main usage of the volatile property is to instruct kartons to treat the sample of the task as volatile, meaning there's no need to report it to external databases (e.g MWDB, MISP). An easy implementation in karton-mwdb-reporter that will prevent the karotn from report a file to MWDB will look like this:
def process(self, task):
        # Checking if task is volatile.
        if task.has_payload("properties"):
            if task.get_payload("properties").get("volatile") == True:
                self.log.info("Task is volatile. Finishing here.")
                return
        ...

This way, samples won't be reported to MWDB in presence of a positive volatile property; either because it is some auxiliary sample made for testing, or a sample not interesting enough to have uploaded to the database. Consequently, another simple yet nice use of the volatile property is to create some internal testing/utility producers to be used by developers/analysts/researchers etc.
For instance, let's take a look at tester-producer.py:
import os, sys
from karton.core import Producer, Resource, Task

producer = Producer()

filename = sys.argv[1]
with open(filename, "rb") as f:
    content = f.read()

resource = Resource(os.path.basename(filename), content)

task = Task({"type": "sample", "kind": "raw"})
task.add_payload("sample", resource)
task.add_payload("properties", {"volatile": True})

producer.send_task(task)

It can help locally test staging or even production environments without the need to store the files in a datbase (MWDB), and still get the whole flow of the system. It can also be used to quickly analyze a file without having it show up in MWDB and sandboxed -- that's what the emulate keyword is for.
An example for something we've done recently using this approach, is that we sent millions of PDF files to our karton system, and we don't want them to be stored in MWDB unless we believe a PDF sample contains an exploit. So we produced these millions of tasks with the volatile flag turned on, and prevented them to flood mwdb.
emulate
Another useful usage is for controlling sandboxing and analysis of samples, in case you've set up some system to sandbox samples and report the results back into karton (which we all do).
Let's say you've set up feeders to get samples from different sources. It may happen that the same samples will come from different sources again and again. In such case we do not want to re-sandbox the sample, since emulating a file in a sandbox is costly and will probably pollute the database with redundant dumps and artifacts. A simple solution might be adding the following code to the kartons responsible for sending samples to be sandboxed:
# a previous check was done to ensure the sample
# was already emulated
if task.get_payload("properties").get("emulate") == False:
    self.log.info("Not going to emulate it.")
    ...
else:
    # logic that sends to a sandbox
    ...

force_analyze and force_emulate
These two, as the name suggests, force a sample to be analyzed/sandboxed. In the case of sandboxing, force_emulate will override emulate, whatever its value may be. The main usage of these two is for sample sources which heuristically produce "interesting" samples. For example, the upload button in MWDB, or the "reanalyze" button in MWDB. It is reasonable to assume that if an analyst chose to upload a sample, or re-analyze it, it is because they think it's an interesting one and would want it to be emulated.
Therefore, we suggest the following small change in mwdb-core/mwdb/core/karton.py:
task = Task(
            headers={"type": "sample", "kind": "raw", "quality": feed_quality},
            payload={
                "sample": Resource(file.file_name, path=path, sha256=file.sha256),
                "attributes": file.get_attributes(as_dict=True, check_permissions=False),
                "properties": {"force_analyze": True, "force_emulate": True}, # ensures what we wanted
            },
            priority=task_priority,
        )

And thus, when implementing a karton/system to dispatch samples from karton into a sandbox, this simple logic will work:
if task.get_payload("properties").get("force_emulate") == True:
    # send to a sandbox logic
    ...
elif task.get_payload("properties").get("emulate") == True:
    # send to a sandbox logic
    ...
else:
    # no emulation for you
    ...

These properties can be mixed and matched for different scenarios. For example:

force analyze a complete flow of an existing file (reanalyze on MWDB)
properties: {
    "force_analyze": True,
    "force_emulate": True,
}

new files to be analyzed by kartons without emulation and without saving
properties: {
    "emulate": False,
    "volatile": True
}

normal file analysis (default values):
Could be without properties

normal file analysis without sandbox:
properties: {
    "emulate": False,
}

Additional points
backwards compatibility(?)
The only case in which backwards compatibility is broken is if someone already implemented the same properties payload in their kartons, and wrote logic revolving around these exact properties that were mentioned. Then it might cause a conflict
However I believe this is not a problem because it is way too specific of a change to the logic. Also, if someone actually had these changes, they uses a forked version of the official kartons (reporter, for example). So there are two cases: either this addition helps them move to upstream version and not use forked repos, or they continue using a fork as they did up until now.
Must be a payload?
No, it does not. It can also be something integrated directly into the Task calss like "priority". We used apyload because it was the easiest to implement without branching from karton-core and other kartons.
Some other alternatives might be adding these properties to the header of tasks. It may be a personal preference but I think they should belong to a dictionary field (payload or not) called properties because these fields are ultimately user controllable, whereas thing like kind or type are internal values inferred as the task moves through different kartons.
Ok, so what actually needs to be changed?
The two main changes were mentioned above, in karton-mwdb-reporter and karton.py from mwdb-core. Other examples of places where it makes sense is, for example, this example drakvuf producer.
If there are other places where you think this might be useful, We'd be glad to discuss it and implement it if needed.
Another area what'll probably need to be changed is the docs; to let people know this exists. This issue can be used as a basis for the docs. In any case, we have no problem handling documentation :)

Of course, these are just a limited number of examples and anyone can extend this idea to whatever they might need. Let us know what you think about it.

Hey dears! :) @psrok1, @nazywam - we'd love your feedback about the suggestions. What do you think? do you find it as useful as we do? Do you see the needs? have other solutions, etc?

Hi! Welcome back. Thanks for making this detailed issue (I didn't notice it until today).
Since persistent headers are now implemented we can go back to thinking about this. I have to say this will be quite useful for us too.
The things I already know (I've asked people internally what do they think about it):

I think we all agree that we don't like force_emulate and emulate separation. If that's not critical, we plan to only support a single flag (emulate using this naming convention)
similarly, we don't understand  what force_analyze is for. We don't plan to support it, until we get a better understanding of it

So for now we're just interested in features provided by volatility and emulate.
Now some of my personal thoughts:

volatility is a more complex concept than it looks like IMO, because there's a lot of external services touched during the analysis in our pipeline. Of course we upload samples to mwdb, but there are many more moving parts: configs go to mtracker, iocs go to n6 and misp, samples themselves go to apkdetect and of course drakvuf. I think before jumping to implementation we should decide exactly what volatile means and maybe define "volatility levels"
emulate is a nice idea and solves our immediate problem. But I'm slightly concerned about duplicating concepts and I wonder if the same could be achieved with a hypothetical "ultralow" feed quality level?

benefits:

simpler implementation, less "special" headers
there's already a nice support in mwdb for quality, and it can even be set per user (exactly our use case)
it has a clearer semantics. Our real intention is to avoid running heavy tasks on samples from "mass" feeds and avoid overwhelming our infrastructure. There may be more services not fit for mass analysis (we used to have one more cost intensive karton service that did decompilation - it would also benefit from this)

downsides:

adding a new enum field may be not backward compatible, and we'll need to check every karton service we have
it's more work than just adding a persistent "emulate" header.

I'm still not decided and would appreciate some perspective.


Creating Registration form

I would like to create a registration form using HTML and CSS. It will contain the following:

User name
User contact information
User address
Submit button

You are not supposed to make issues now.

Okay! Thankyou so much for assigning it to me. I will make sure not to
dissapoint you!
On Mon, 31 Jan 2022, 10:01 pm Tamal Das, @.***> wrote:

You are not supposed to make issues now.
—
Reply to this email directly, view it on GitHub
https://github.com/IAmTamal/Milan/issues/7#issuecomment-1025968993, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AU2C4IKHOC35DNBSCDBD23TUY22MTANCNFSM5NGW7YGA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

--
CONFIDENTIALITY NOTICE
This
communication (including any
attachment/s) is intended only for the use of the
addressee(s) and contains
information that is PRIVILEGED AND CONFIDENTIAL.
Unauthorized reading,
dissemination, distribution, or copying of this
communication is
prohibited.
* DISCLAIMER*
Any views or
opinions presented in
this email are solely those of the author and do not
necessarily represent
those of the University.
WARNING:-
This
communication, including
any attachments, may not be free of viruses,
interceptions or interference,
and may not be compatible with your systems. You
should carry out your own
virus checks before opening any attachment to this
e-mail. The sender of
this e-mail and Galgotias University shall not be liable
for any damage
that you may sustain as a result of viruses, incompleteness of
this
message, a delay in receipt of this message or computer problems
experienced.


[fix] Comment _builtin_renames.h to avoid name mangling

#31

Not needed, this is a wrong PR

I don't know the reason for this closure.


Catch and display service errors

When the services fail due to misconfiguration we see the service failure but it doesn't display any useful error message, this makes it difficult to realize what's causing the issue in the CI.
It's debatable if this should be added to role runs outside of molecule as this issue usually doesn't come up outside of development.
I already added this to testinfra https://github.com/prometheus-community/ansible/pull/197/commits/083ff4ef361fdcc719a7338a14715620af45021d but that doesn't catch service errors which occur in ansible tasks.
I see two possible options:

Turn on debug logging in molecule
This would not be the preferred option as enabling debug output for all tasks and make the already long CI logs even longer and most likely slow them down as well.

Catch the error in a block.
This would add extra tasks in many places of the playbook and could be a bit messy, especially if we only want to capture the errors when running the molecule tests.

Related: https://github.com/prometheus-community/ansible/pull/213#issuecomment-1819135143

Can be helpful setting ANSIBLE_DIFF_ALWAYS=True in CI. With that, you can see the content of the service file, but otherwise it can be extremely spammy :sweat_smile:

Oh interesting, that seems like the ideal solution, thanks! I've hardly ever used that option so it didn't cross my mind for this use case.

@gardar It hugely depends on the quality of the diff mode. For #213 it helps a little, because you can see the content of the service file, but you still do not see what the service complains about.
It might be useful for CI to add additional tasks at the end to run systemctl status prometheus.service" and/or journalctl -xeu prometheus.service. I don't know how to implement this with molecule, maybe with post_tasks` or something like that which runs unconditionally at the end in the container.

@gardar What I ended up using for #213 was this (at the end of each respective tasks/main.yml):
# currently last task

- name: Ensure blackbox_exporter service is started and enabled
  ansible.builtin.systemd:
    ...

# my uncommitted additions to show the problems

- command: systemctl status blackbox_exporter.service
  changed_when: no
  failed_when: no
  register: status

- debug: var=status.stdout_lines

- command: journalctl -x -u blackbox_exporter.service
  changed_when: no
  failed_when: no
  register: journal

- debug: var=journal.stdout_lines

This could be added to each role setting up a systemd service and optionally to only run in CI. I decided against adding this to #213.
As for the diff mode, I wouldn't generally recommend using ANSIBLE_DIFF_ALWAYS=True in CI, as mentioned before, but we could enforce diff mode for the service template task by adding diff: yes.
I could draft a PR for both of these changes for all roles.

Yes these tasks are along the lines of what I had in mind, my problem with them is just that it doesn't feel natural including tasks like that in all the roles.
Ideally I would want something that we could have just when running the roles with molecule / in the CI.
I don't think it's necessary to display the output of failed services for users outside of what Ansible provides by default.
The chances of the user breaking the services are hopefully lower than the services breaking during development as the CI tests should catch most of that, and if the service breaks for the user he can probably just jump onto the server and look at the service output where as we can't diagnose anything other than the logs from the CI output and will have to run the CI again to debug which can take a log time.

What I do in some of my roles to limit tasks to CI / container is this:
# vars/main.yml
__locale_virtualized: >-
  {{
    ansible_facts.virtualization_role == "guest"
    and
    ansible_facts.virtualization_type == "docker"
  }}

# tasks/main.yml
- name: update package cache
  ansible.builtin.package:
    update_cache: yes
  become: yes
  when:
    - __locale_virtualized

This particular example is for containers that require package cache updates before you can use the package manager.

Taking a more general approach, there must be some GitHub workflow or molecule specific environment variables (or even define your own __PROMETHEUS_CI in the workflow vars) you could check on instead of just determining if you're running in a container. Then, you can put the service check commands from above inside a block and conditionally run that block based on these vars.

Taking a more general approach, there must be some GitHub workflow or molecule specific environment variables (or even define your own __PROMETHEUS_CI in the workflow vars) you could check on instead of just determining if you're running in a container. Then, you can put the service check commands from above inside a block and conditionally run that block based on these vars.

Yes indeed, there are Ansible tags that molecule looks for that could be used as well as several env vars that both molecule and the GitHub actions set at runtime, and in fact we are using the CI vars already here https://github.com/prometheus-community/ansible/commit/82d0c730ed23918f05f7952df6d22b55ed3cc395
I wish we could do something similar to that on a task level to get the service output rather than adding extra tasks. Unfortunately simply increasing the verbosity does not help as it won't print the service log output 😞

What's the harm in adding the extra tasks? We may even be able to DRY it up by using a role for these tasks with include_role.


Two-Factor Authentication (2FA/MFA)

two factor auth done, need to implement enable and disable 2fa, and show qrcode in UI

Use django-two-factor-auth package


Add semicolons at the end of locale files

I have no idea where files with pluralization rules in locale folder come from, but they don't have semicolons at the end and this may cause some really strange and hard to find bugs when these files are concatenated with other third-party library files. Here's an example from our app: http://jsfiddle.net/szimek/1xrzqgmm/3

In current master, and future releases all plural files are generated with make-plural which sources directly from CLDR. So I think this has more or less already been fixed.
(I would suggest smarter build tools than just "concat blindly" though.)

Thanks for the report!

For those who by any chance have the same issue and use gulp-concat - turns out it has newLine option, so calling it like concat("output.js", {newLine: "\n;"}) fixes this problem.


Add ja v100 (revised)

Fixed heading issues in #68 .
I want to review the Japanese translation, so I leave it draft until I do it.

I tidied up the translation!
@juggernautjp Take a look at it if you want.


CONFIGURATION-772: Exclude unused transitive deps

See CONFIGURATION-772
Hello. I noticed that poi-ooxml and postgresql, which are transitive dependencies via dbunit, are actually not used.
Looking at the Maven dependency tree of the project, we can see that these dependencies may create conflicts:
.  .  .
 |  +- org.apache.poi:poi-ooxml:jar:3.17:test
 |  |  +- org.apache.poi:poi:jar:3.17:test
 |  |  |  +- (commons-codec:commons-codec:jar:1.10:test - omitted for conflict with 1.13)
 |  |  |  \- org.apache.commons:commons-collections4:jar:4.1:test
 |  |  +- org.apache.poi:poi-ooxml-schemas:jar:3.17:test
 |  |  |  \- org.apache.xmlbeans:xmlbeans:jar:2.6.0:test
 |  |  |     \- stax:stax-api:jar:1.0.1:test
 |  |  \- com.github.virtuald:curvesapi:jar:1.04:test
 |  \- postgresql:postgresql:jar:8.4-701.jdbc3:test
.  .  .

Hence, It is a good practice to exclude these dependencies from the pom because it prevents from potential conflicts, it makes the library smaller and the dependency tree clearer.

Thank you for the PR. The excluded dependencies here are in the test scope and only affect the build, not the library artifact itself. Thus this change is unnecessary.


[Enhancement] Adding support for event emitters

I believe that Event Emitters are much more flexible than using a callback from listen function. That would be a nice addition to the library.

Honestly, rewriting the library to use EventEmitters and Promises/async functions instead of callbacks would make it so much easier to use, but I don't think Schmavery has interest in anything like that.

Honestly, rewriting the library to use EventEmitters and Promises/async functions instead of callbacks would make it so much easier to use, but I don't think Schmavery has interest in anything like that.

Yeah, though this lib definitely deserves a rewrite since the state of it has been bad for a long time. (not because of the maintainers, but rather facebook that keeps making changes)

IMO it deserves a rewrite, because currently there's quite a bit of left-over unused/commented code, a few different code styles and it lacks most of newer JS features, like Promises or BigInts (which could be used internally, like for generating OTIDs), current maintainers don't seem very active though :/
I was thinking about either making a principles fork of FCA by making a patchset or just abandoning this codebase and rewriting it from the ground, but it would take some time.

I'm actually working on a rewrite with new features. But I didn't have promises and event emitters in mind so I'll try to add them too it would be alot better.

I see that @lequanglam has started fca-unofficial, looks promising.

i'm working on it.

https://github.com/fca-unofficial/fca-unofficial/releases/tag/1.3.8-u
oh yeah took me fricking 4 MONTHS TO DO THIS.

https://github.com/fca-unofficial/fca-unofficial/releases/tag/1.3.8-u
oh yeah took me fricking 4 MONTHS TO DO THIS.


debug austin 3.5.0

relates to https://github.com/Homebrew/homebrew-core/pull/123851

ended up updating the test and rely on upstream tests moving forward.


Add telemetry around kernels

check whether a kernel fails to install
check whether we fail to detect any kernels

Edit: Also modify execute_native_cell to capture duration

This is likely the riskiest telemetry addition. We'd have to track failures of kernel install and kernel detection. Not sure we even know if that happens right now.

Not sure we even know if that happens right now.

I think we do, after we install a kernel, we list the kernels to find the kernel spec info.

Added check whether selected kernel is a kernel that was installed by extension (i.e. user selected pre-existing kernel vs an interpreter)

This helps us know how many are using custom kernels vs our standard ones.
Helps us figure out how many know & use kernels vs just using default interpreter kernel
I think its useful  @jmew /cc

Do we need time taken to install a kernel?

Do we need time taken to install a kernel?

I think thats good to have as well. Better to have extra telemetry than not enough.

Validated with other telemetry fixes


[OpenCL/GPU] Kernel binary caching

Changes for GPU kernel caching:

Added utility for saving kernel as binary files using clGetProgramInfo.
Added wrapper for clCreateProgramWithBinary.

Signed-off-by: Debadri Samaddar<EMAIL_ADDRESS>

:memo: TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2503. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.

:octocat: cibot: @s-debadri, nntrainer/opencl/opencl_loader.cpp includes bug(s). Please fix incorrect coding constructs in your commit before entering a review process.

:octocat: cibot: @s-debadri, nntrainer/opencl/opencl_program.cpp includes bug(s). Please fix incorrect coding constructs in your commit before entering a review process.

:octocat: cibot: @s-debadri, nntrainer/opencl/opencl_program.cpp includes bug(s). Please fix incorrect coding constructs in your commit before entering a review process.

:octocat: cibot: @s-debadri, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nntrainer/ci/repo-workers/pr-checker/2503-202403152117300.24992704391479-2d0ba968c8c5e44471a42361ff58b6b72a957738/.

:octocat: cibot: @s-debadri, nntrainer/opencl/opencl_program.cpp includes bug(s). Please fix incorrect coding constructs in your commit before entering a review process.

@EunjuYang @jihochu PTAL. This is taking too long.

could you check the CI?

:octocat: cibot: @s-debadri, nntrainer/opencl/opencl_program.cpp includes bug(s). Please fix incorrect coding constructs in your commit before entering a review process.

@s-debadri  @jijoongmoon
nofile:0:0: information: Too many #ifdef configurations - cppcheck only checks 12 configurations. Use --force to check all configurations. For more details, use --enable=information. [toomanyconfigs]

It means that there are too many conditional compilation directives (#ifdef and others) in the code being analyzed. Cppcheck can handle up to 12 such configurations at a time by default
and i give --force option and run in my local environment --> There is no cppcheck error on this PR


Works but maybe you could enhance README file

Hello and, at first, thanks for this nice plugin.
I spent a few time to understand why the plugin didn't work (no command and add to dictionary failed).
I agree that you mentioned to call the setup() function, but it's a bit confused with LazyVim.
Actually, to ensure that LazyVim calls the setup function, you need to pass the opts as non-null content (it could be empty).
So, on my side:
return {
  {
    "icewind/ltex-client.nvim",
    opts = {},
  },
}

To ensure that Mason includes ltex-ls it's also possible to give:
return {
  -- ensure mason istalls ltex-ls
  {
    "williamboman/mason.nvim",
    opts = function(_, opts)
      vim.list_extend(opts.ensure_installed, { "ltex-ls" })
    end,
  },
  {
    "icewind/ltex-client.nvim",
    opts = {},
  },
}

It works

Hello, thank you for your feedback. To my understanding, we can use config = true to make lazy.nvim automatically call setup function. At least it works in my setup that way.
I've updated the README, please feel free to comment if there is something I can add to make it more helpful.

Ha maybe I missed the "config" attribute. Right.
I will take a look. Thanks a lot, because having the "add to dictionary" is
a must have, and I didn't had the time to implement it on my side 😁
Le lun. 28 oct. 2024, 19:55, Nikolay Osaulenko @.***> a
écrit :

Hello, thank you for your feedback. To my understanding, we can use config
= true to make lazy.nvim automatically call setup function. At least it
works in my setup that way.
I've updated the README, please feel free to comment if there is something
I can add to make it more helpful.
—
Reply to this email directly, view it on GitHub
https://github.com/icewind/ltex-client.nvim/issues/3#issuecomment-2442380304,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAYN4BDA2JKTLNAAQNSZRTZ52CANAVCNFSM6AAAAABQXMS4PWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBSGM4DAMZQGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

Hi,
Here: https://lazy.folke.io/developers it is mentioned that we should use opts to an empty table.

Hello,
Oh, I see. Thank you. I updated the readme.


CUDA out of memory

After training with volsdf, I try to export the point cloud with the following command proposed by the viewer: ns-export pointcloud --load-config test_andres/train/test_andres/volsdf/2023-07-10_093029/config.yml --output-dir exports/pcd/ --num-points 100 --remove-outliers False --estimate-normals False --use-bounding-box False

It should be noted that when I try to export the mesh, the memory does NOT collapse

You can try lowering the --num-rays-per-batch (by default 32768)

Hi, you could also use ns-extract-mesh if you are trying sdf-based methods since sdf has a clear definition of surface and marching cubes works well (of course only if the model successfully trained).


fix compile error when tower not updated in Cargo.lock

Motivation
Fix the following compile error caused by the version of tower in Cargo.lock not having to be at least v0.4.6:
error[E0599]: no method named `layer_fn` found for struct `ServiceBuilder<tower::layer::util::Identity>` in the current scope
  --> tonic/src/transport/service/connection.rs:54:14
   |
54 |             .layer_fn(|s| AddOrigin::new(s, endpoint.uri.clone()))
   |              ^^^^^^^^ method not found in `ServiceBuilder<tower::layer::util::Identity>`

Since the existing version in tonic's Cargo.toml is v0.4.4, cargo did not know to update that dependency automatically if the Tonic developer has an older version in Cargo.lock. Since Tonic is a library, Cargo.lock is not checked in and can vary between developers.
Solution
Specify v0.4.6 for tower to ensure layer_fn is available.

This was fixed as part of https://github.com/hyperium/tonic/pull/606


Instruction for model training

I am very new for this work. While generating is able from python to swift, I could not find applicable way to make customized model by myself. Dreambooth of Diffusers from huggingface is not working for my 16GB M1 mini. - Actually dreambooth for 8GB was able but it took more than 10 hours for their dog example. I expect that there will be better way to be applied for Apple silicon.
If there is several instruction or example for tunning or training to be applied ml-stable-diffusion, it will be very helpful for persons as like me.

M1 MAX  is not support?

+1 for adding the ability to train models natively


Version equality not always correct

When tag is updated, dfetch does see this.  (Problem is in equality check in dfetch/project/version.py
Below scenario fails:
    Scenario: Tag is updated in manifest
        Given the manifest 'dfetch.yaml'
            """
            manifest:
              version: '0.0'

              projects:
                - name: ext/test-repo-tag
                  url: https://github.com/dfetch-org/test-repo
                  tag: v1

            """
        And all projects are updated
        When the manifest 'dfetch.yaml' is changed to
            """
            manifest:
              version: '0.0'

              projects:
                - name: ext/test-repo-tag
                  url: https://github.com/dfetch-org/test-repo
                  tag: v2.0

            """
        And I run "dfetch update"
        Then the output shows
            """
            Dfetch (0.0.6)
              ext/test-repo-tag   : Fetched v2.0
            """

See 96c2067


Support for 1D Convolutions?

I was trying to run visualize visualize_activation on a model trained on 1D data with 2 channels but ran into assertion error when it saw that it wasn't a 2D image it needed to generate:
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-144-899963b8ba25> in <module>()
      5 vis_images = []
      6 for idx in [0, 0, 0]:
----> 7     img = visualize_activation(model, layer_idx, filter_indices=idx, max_iter=500)
      8     vis_images.append(img)
      9

/usr/local/lib/python2.7/dist-packages/vis/visualization.pyc in visualize_activation(model, layer_idx, filter_indices, seed_img, text, act_max_weight, lp_norm_weight, tv_weight, **optimizer_params)
    108     ]
    109
--> 110     opt = Optimizer(model.input, losses, norm_grads=False)
    111     img = opt.minimize(**optimizer_params)[0]
    112     if text:

/usr/local/lib/python2.7/dist-packages/vis/optimizer.pyc in __init__(self, img_input, losses, wrt, norm_grads)
     33             # Perf optimization. Don't build loss function with 0 weight.
     34             if weight != 0:
---> 35                 loss_fn = weight * loss.build_loss()
     36                 overall_loss = loss_fn if overall_loss is None else overall_loss + loss_fn
     37                 self.loss_names.append(loss.name)

/usr/local/lib/python2.7/dist-packages/vis/regularizers.pyc in build_loss(self)
     63             ss = tuple(samples_channels_slice + start_slice)
     64             es = tuple(samples_channels_slice + end_slice)
---> 65             diff_square = K.square(self.img[utils.slicer[ss]] - self.img[utils.slicer[es]])
     66             tv = diff_square if tv is None else tv + diff_square
     67

/usr/local/lib/python2.7/dist-packages/vis/utils/utils.pyc in __getitem__(self, item_slice)
    247         """Assuming a slice for shape `(samples, channels, image_dims...)`
    248         """
--> 249         assert len(item_slice) == 4
    250         if K.image_data_format() == 'channels_first':
    251             return item_slice

AssertionError:

Would it be difficult to modify this to support 1D input data? Then we can plot using plt.plot instead of plt.imshow?

I removed that assert a while ago. Can you test your change on latest? AFAIK, it should work for any N-dim convolutions

Closing due to inactivity. Feel free to reopen if the issue persists

I think it still not supports the 1D Convolution.
This is the error when I try to Visualize all filters in this layer using following code
filters = np.arange(get_num_filters(model.layers[layer_idx]))
Tensor("convolution1d_3/add:0", shape=(?, 1, 64), dtype=float32) (None, 1, 64) Traceback (most recent call last):   File "1.py", line 29, in <module>     filters = np.arange(get_num_filters(model.layers[layer_idx]))   File "/Users/zhishanzhang/code/ke2/lib/python2.7/site-packages/vis/visualization.py", line 38, in get_num_filters     return K.int_shape(layer.output)[3] IndexError: tuple index out of range
So I tried to change 3 to 2, and get an error
Traceback (most recent call last):   File "1.py", line 35, in <module>     img = visualize_activation(model, layer_idx, filter_indices=idx)   File "/Users/zhishanzhang/code/ke2/lib/python2.7/site-packages/vis/visualization.py", line 110, in visualize_activation     opt = Optimizer(model.input, losses, norm_grads=False)   File "/Users/zhishanzhang/code/ke2/lib/python2.7/site-packages/vis/optimizer.py", line 35, in __init__     loss_fn = weight * loss.build_loss()   File "/Users/zhishanzhang/code/ke2/lib/python2.7/site-packages/vis/losses.py", line 87, in build_loss     loss += -K.mean(layer_output[utils.slicer[:, idx, ...]])   File "/Users/zhishanzhang/code/ke2/lib/python2.7/site-packages/vis/utils/utils.py", line 249, in __getitem__     assert len(item_slice) == 4 AssertionError

This line len(item_slice) == 4 does not exist in latest source. Can you install from source? The pip package is out of date.

OK，I will try it. Thank you.
On Fri, Jun 16, 2017, 9:31 AM Raghavendra Kotikalapudi <
<EMAIL_ADDRESS>wrote:

This line len(item_slice) == 4 does not exist in latest source. Can you
install from source? The pip package is out of date.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/raghakot/keras-vis/issues/28#issuecomment-308908516,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFcwiRSmQ4GxTF9BjM69W5QSLlASQBknks5sEdrygaJpZM4NmIlt
.

--
祝您身体健康，工作顺利
张治山
18392383167


cannot import ToolNode from Langgraph

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangGraph/LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangGraph/LangChain rather than my code.
[X] I am sure this is better as an issue rather than a GitHub discussion, since this is a LangGraph bug and not a design question.

Example Code
from langchain_core.runnables import RunnableLambda, RunnableWithFallbacks
from langgraph.prebuilt import ToolNode

def create_tool_node_with_fallback(tools: list) -> RunnableWithFallbacks[Any, dict]:
    """
    Create a ToolNode with a fallback to handle errors and surface them to the agent.
    """
    return ToolNode(tools).with_fallbacks(
        [RunnableLambda(handle_tool_error)], exception_key="error"
    )

Error Message and Stack Trace (if applicable)
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[66], line 2
      1 from langchain_core.runnables import RunnableLambda, RunnableWithFallbacks
----> 2 from langgraph.prebuilt import ToolNode
      5 def create_tool_node_with_fallback(tools: list) -> RunnableWithFallbacks[Any, dict]:
      6     """
      7     Create a ToolNode with a fallback to handle errors and surface them to the agent.
      8     """

ImportError: cannot import name 'ToolNode' from 'langgraph.prebuilt' (/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/langgraph/prebuilt/__init__.py)

Description
I am getting this error today, when I was able to execute it before
System Info
from langgraph.prebuilt import ToolNode

@rajidhamon31 what version of langgraph are you using?

+1, please fill out the system info as required by the issue template.


Add statefulset aliases to kubectl

Adds aliases for managing statefulsets

Thanks for this -- could you update the README, too?

@robbyrussell Done!


Move sample data to zivid data

This PR is the equivalent of https://github.com/zivid/cpp-extra-samples/pull/102
We've agreed to stop duplicating sample data in each repository. Instead we will host all sample data on webpage/downloads/ZividSampleData.zip.

https://github.com/zivid/python-samples/pull/65/commits/361eaf7d78a398bab992c4e8ae512064a0e40a5d should not be required after rebase.

361eaf7 should not be required after rebase.

It seems it was still needed for wrong-import-order. I left it and reword-ed the commit message.

Ready for the next round.


Update private-endpoint-overview.md

App Service Slots with Private Link functionality is now available.

@geoffop : Thanks for your contribution! The author(s) have been notified to review your proposed change.

MSFT contact is Naveed Zaheer

@naveedzaheer

@ktoliver not seeing movement on this from reviewer

@malopMSFT @naveedzaheer @kumudd Would you review the proposed update to the article?
Thanks.

@asudbring Please can you review this PR. Thanks!

Looks correct.  Verified in portal
#sign-off

@asudbring: I'm sorry - only the author of this article, @malopMSFT, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.


Who wants to test Sieve Filters?

Good news, Everyone!
I finally have finished this functionality.
If you have the opportunity to test the filters, I'll be grateful to you.
link: http://www.rainloop.net/downloads/
screenshots:

Hi, sorry for the delay, but I finally got time to test this. I created a test install with your supplied zip and here are the results:
When I create a Sieve rule, I'm stuck on "These changes need to be saved to the server":

I created some logs.
When I enter Settings > Filters, I see this in theRainLoop log:
[08:55:36.006][215dc864] INFO[DATA]: [DATE:30.01.15][RL:<IP_ADDRESS>][PHP:5.4.36-0+deb7u3][IP:<IP_ADDRESS>][PID:7542][Apache][apache2handler]
[08:55:36.007][215dc864] INFO[DATA]: [Suhosin:off][APC:on][MB:on][PDO:mysql,pgsql,sqlite][Streams:tcp,udp,unix,udg,ssl,sslv3,tls]
[08:55:36.007][215dc864] REQUEST[NOTE]: [POST] https://#MYFQDN#/rainloop-sieve-test/?/Ajax/&s=/_#IMSENCORINGTHISJUSTINCASE#/
[08:55:36.007][215dc864] AJAX[NOTE]: Action: DoFilters
[08:55:36.008][215dc864] POST[DATA]: {"Action":"Filters","XToken":"#IMSENCORINGTHISJUSTINCASE#"}
[08:55:36.013][215dc864] SIEVE[NOTE]: Start connection to "tcp://localhost:4190"
[08:55:36.014][215dc864] SIEVE[NOTE]: Connected (success)
[08:55:36.034][215dc864] SIEVE[DATA]: < "IMPLEMENTATION" "Dovecot Pigeonhole"\r\n
[08:55:36.034][215dc864] SIEVE[DATA]: < "SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave"\r\n
[08:55:36.034][215dc864] SIEVE[DATA]: < "NOTIFY" "mailto"\r\n
[08:55:36.035][215dc864] SIEVE[DATA]: < "SASL" "PLAIN"\r\n
[08:55:36.035][215dc864] SIEVE[DATA]: < "STARTTLS"\r\n
[08:55:36.035][215dc864] SIEVE[DATA]: < "VERSION" "1.0"\r\n
[08:55:36.035][215dc864] SIEVE[DATA]: < OK "Dovecot ready."\r\n
[08:55:36.037][215dc864] SIEVE[DATA]: > AUTHENTICATE "PLAIN" {24+}\r\n
[08:55:36.037][215dc864] SIEVE[DATA]: > #IMSENCORINGTHISJUSTINCASE#\r\n
[08:55:36.083][215dc864] SIEVE[DATA]: < OK "Logged in."\r\n
[08:55:36.083][215dc864] SIEVE[DATA]: > LISTSCRIPTS\r\n
[08:55:36.083][215dc864] SIEVE[DATA]: < OK "Listscripts completed."\r\n
[08:55:36.084][215dc864] SIEVE[DATA]: > LOGOUT\r\n
[08:55:36.088][215dc864] SIEVE[DATA]: < OK "Logout completed."\r\n
[08:55:36.088][215dc864] SIEVE[NOTE]: Disconnected from "tcp://localhost:4190" (success)
[08:55:36.088][215dc864] AJAX[DATA]: {"Action":"Filters","Result":{"RawIsAllow":true,"RawIsActive":false,"Raw":"","Filters":[],"Capa":["fileinto","reject","envelope","encoded-character","vacation","subaddress","comparator-i;ascii-numeric","relational","regex","imap4flags","copy","include","variables","body","enotify","environment","mai...
[08:55:36.088][215dc864] INFO[MEMORY]: Memory peak usage: 1.25MB
[08:55:36.089][215dc864] INFO[TIME]: Time delta: 0.086724996566772

And my syslog says:
Jan 30 09:55:36 webandmail dovecot: managesieve-login: Login: user=<#USERNAME#>, method=PLAIN, rip=<IP_ADDRESS>, lip=<IP_ADDRESS>, mpid=8318, secured, session=<9WSaxNoNHQB/AAAB>
Jan 30 09:55:36 webandmail dovecot: managesieve(#USERNAME#): Disconnected: Logged out bytes=19/70

Then I add a filter, click done and end up with the message "These changes need to be saved to the server".
When I press Save, nothing happens and nothing gets logged...
Ralf.

Thanks!
Can you create some logs then you press Save button?

What do you mean? When I press Save, nothing happens and nothing gets logged...

Any js errors?

You mean in Apache logs? I get also nothing in Apache logs when pressing Save.

Same here. Not able to save new Filterscripts. Using this testinstallation on Nginx.
Tested in Firefox and Chrome. Nothing appears in any logs

Hi, sorry for not replying in time.
@ervee No, I mean browser console.
Ok, there is a new version on RainLoop Webmail with a lot of changes and fixes.
link:
http://repository.rainloop.net/v2/webmail/1.7.3/rainloop-<IP_ADDRESS>-932d36047cdaa2426e97645f5f212d89.zip

Still the same behaviour with the new version.

Can you give me access credentials for tests?

wrote an mail to<EMAIL_ADDRESS>

@andre-dierker Thank you very much. You really helped.
new version:
http://repository.rainloop.net/v2/webmail/1.7.3/rainloop-<IP_ADDRESS>-edae8ff8823f5fbf684fd076c6e9446f.zip

Works fine now :)

Saving scripts works here too. Saved script also looks okay. I'll try to test as much rule types as possible. If I find anything wrong I'll let you know.

Ok, thanks!

I do already have a little feedback:

I think it should me more clear to the user that the rules are processed from top to bottom and when there is a match, processing stops there. Al least, you insert a "stop" in every Sieve filter.

The filters are movable. So perhaps tell the user or make it visualy noticable like in most Android movable lists there are sort of a visual handle at the beginning of each movable item.

Perhaps an (advanced) option to not stop rule processing after the rule you are creating/editing?

Ralf.

@ervee  Thanks.

Agree.

Perhaps an (advanced) option to not stop rule processing after the rule you are creating/editing?

I thought about that. But I can not understand the reason why this is necessary.
If only to use it for "Forward to" and "Vacation message" actions.

The only thing I can quickly think of is indeed vacation messages. Perhaps:

Make "don't stop processing rules" an option when creating a vacation message?
Also have the user decide the reply interval (fixed set at 1 day now)

The only thing I can quickly think of is indeed vacation messages. Perhaps:

Make "don't stop processing rules" an option when creating a vacation message?
Also have the user decide the reply interval (fixed set at 1 day now)

Done.

Managesieve running on port 2000 is very outdated, maybe change it to 4190 in sieve configuration?

Ok

new version:
http://repository.rainloop.net/v2/webmail/1.7.3/rainloop-<IP_ADDRESS>-a2510466f4c3fa9adeea110e005a220d.zip

The filters I created with the previous version (filename rainloop.sieve.user.sieve) are now not visible anymore (but still there) and a new file rainloop.user.sieve is created and the symlink .dovecot.sieve is updated to use that new script. I assume this was a deliberate change :)
Also RainLoop does not use an already existing script? I don't realy know how this works though...
All looks good. But I do get a weird problem in this version (and the previous) if I open an email to read. I can't close it (with the X button of ESC). The message just pops back up again!

I assume this was a deliberate change

Yes

Also RainLoop does not use an already existing script?
I don't realy know how this works though...

RainLoop knows how to build script only.

The message just pops back up again!

I can not reproduce it.
Probably a browser/cookie/whatever. It's a test setup so I don't mind.

Ok, now I see the problem.
Thanks for reporting!

a new version: http://repository.rainloop.net/v2/webmail/1.7.3/rainloop-<IP_ADDRESS>-214b8f8384bb4830fac7e47680b5371d.zip

This version works great so far.

Sieve is working fine now. Now there is a related problem in Inbox view. Reloading the message list, doesn't reloading subdirectories too. New mails in folders aren't displayed.

@ervee Good.
@andre-dierker There is other reloading mechanism for that (automatically).
Reload button is for current folder message list only.

Hi Rainloop Team!
I couldn't test this feature yet, but I have one feedback:
Have to remove the Sieve configs of the validation (Test button) when "Allow sieve scripts" is not checked
Nice feature!

@arturbonnett Thanks for reporting!

I think it is a last beta version before release (RC1):
http://repository.rainloop.net/v2/webmail/1.8.0/rainloop-<IP_ADDRESS>-e1eb4ba4d60756c560ef7a9e4f406746.zip

Just tested this version and it looks okay to me.

I have released a new version (1.8.0)
http://www.rainloop.net/downloads/
Thanks all!

Updated my acceptance instance and still looks good :)

Ahhhhhh I'm so happy to see that sieve filters are in :)
But I did not understand why imap & smtp settings are there and what I have to put ?
My settings:

My error:

If i try to put <IP_ADDRESS> : 2000, it says that the server is not accessible.
Thanks a lot

Why your sieve port is 993?
Did you mail server support sieve scripts?

That's good question.. I'll try to find the answer... I hope so !
Ty for ur reply

please allow it to support already existing filters which were created by sogo.

Suggestion to add a link on the sieve page to the sieve manual. This is probably the best page that I've seen: https://support.tigertech.net/sieve but also of course the rfc https://www.ietf.org/rfc/rfc3028.txt and http://sieve.info/

So I finally got the time to test the sieve filters and I remember I had some set up .. but I don't see them .. found out that rainloop has its own "commented headers" that don't allow reading the filters I had written myself OR the ones I had created with squirrelmail .. why ?

why ?

A quick answer is: "It's too complicated"

But when you manage Sieve filters from multiple applications, the problem rises quickly to the surface. When you create rules in RainLoop (and you fail to notice your existing rules don't show in RainLoop) and save them, you wipe out your existing filters...
I don't have this problem because I only use RainLoop to edit Sieve filters, but I can imagine people running IE RoundCube and RainLoop together. Or when migrating from RoundCube to RainLoop, this needs admin intervention.

you wipe out your existing filters...

I do not delete them, I created my own script and sets it as default.
I understand that the best decision is to write a complete Sieve scripts parsing library,
but it is very complex and time consuming task.

im getting the message "Cannot connect to Server" when i click on the filters link.

nevermind - left off sieve in dovecot.conf

Works fine! thanks a lots!

i try to use with gmail
but not working
please check below screen
1 .

@dineshrabara why do you expect this to work? You need a sieve server for sieve to work (duh!)

how can i?
please help me

@dineshrabara you can't.

is this possible to add vacation email script ??

@dineshrabara https://github.com/RainLoop/rainloop-webmail/issues/43

Thanks

working well here, using managesieve on standard port.

Sorry add a new comment :)
but last year you comment about subfolder refresh :

RainLoop commented on 2 Feb 2015
@ervee Good.
@andre-dierker There is other reloading mechanism for that (automatically).
Reload button is for current folder message list only.

is it possible to know how it's working and is there a way to speed up notification for unread message moved in folder ?
btw  thanks for this fresh webmail :)

Hello,
Is it possible to enable the "body" condition for the sieve filter?
Thanks.

Hello,
Sorry to reply on an old/closed subject but I really would like to know this:
I have added the sieve support to create rules, but I would LOVE an option to copy a message to another folder without removing it from the inbox folder.
Is there already an option like this or can this be done?

Copy: in theory should work like this: https://tools.ietf.org/html/rfc3894 Did you try it?
Saving them - this best be troubleshoot from server logs. What sieve server are you using?

Is it possible to implement the parsing using existing projects? The inability of RainLoop to modify filters created by other programs makes this feature rather incomplete and in some cases even useless. Maybe to add an "Advance" mode in which parsing/interpretation is not needed, just pure editor which would edit all sieve scripts served by the server?

@adundovi as far as parsing goes, it does not belong on the mail client such as rainloop. As far as ability to edit sieve scripts, as far as I remember it has been supported from the first release that included sieve support.

Hi @AndrewSav, thank you for your quick reply.
I was referring to this comment of @RainLoop above. As I already saw it in practice: RainLoop creates a custom base64 encoded comment inside a sieve script which saves the content of it in RainLoop's recognizable format therefore it doesn't need to parse the sieve language to interpret those rules - reading that comment is enough, if I understand the mechanism correctly (or if I correctly read the code: RainLoop/Providers/Filters/SieveStorage.php). If the other mail client doesn't update the comment in the sieve script created by RainLoop, RainLoop will not display it correctly.

OK, now by reading the code I understand that there is an option for "raw" sieve scripts like I suggest in my first comment. Great! Thus, if one uses the "raw" mode, there is no graphical editing of the scripts, but at least other programs can interfere with those rules. This is fair enough for me.

Is there some filter limit? After 59 filter I get a ”Invalid token” error, when I try to save filters.

Same issue for me.
59th filter cannot be saved with "Invalid Token" issue.
Removing one filter and adding a new one works correctly.

Hi, today I've tested Rainloop filters (or course, installed and configured Sieve with Dovecot first) and works. But I have a problem, which I think is related of Rainloop works: I needed to implement filters most of all cause I need to move emails to folders (since We are leaving Google apps, missing Gmail labels, at least I need to move emails to folders), but does not appears as new emails, only if i navigate to that folder.
There is no way to check for new emails in any folder and not only in the current folder I am browsing?
Thanks in advance.

In settings, folders you can toggle a checkmark for that:

@ervee: thanks! I didn't saw that.
Cheers.

Hello!
Currently I use mail server with postfix+cyrus-imap+roundcubemail.
It work fine and a can manage sieve rules via roundcubemail web console.
I installed rainloop app in nextcloud and see that sieve filters not working.
I can't see my current sieve rules and can't create new rules.
Nextcloud version- 12.0.2
Rainloop app version - 5.0.1
Please. fix this issue.

Have you thought about using a good library for parsing the true sieve rules? I understand that this is probably complicated to implement, but there seems to be some library around, which supports sieve + extensions:
https://github.com/ProtonMail/libsieve-php
This would mean no more rules restricted to an own format with a converter to sieve, but being able to implement all possible sieve rules and parse them.
I did not test it myself, but you could give it a try. Would make your life easier in maintaining the sieve-function and our life easier in using every sieve client we can find and still being able to edit the rules in rainloop.

±1

So what's the condition for rainloop to show a script ? I have a "Main" script in use currently which was created from thunderbird, but rainloop doesn't show anything even with the custom script ticked.

this may be obvious to most folks here, but i just want to link more and add redundancy because i had lots of issues to find a link with good documents and examples on how to use sieve script.
imho, this should be a link into rainloop on that very page from op in which there's a list of supported capabilities:
https://tools.ietf.org/html/rfc5228
cheers!

For me Sieve does not work on Rainloop (and Sieve is running in my server, Ubuntu 18.04).

Any updates?
I still don't see existing sieve script under Rainloop interface.
I'm afraid of adding new filters, as I'm not sure will it override my existing script with lot of filters in it or not.


PowerBi does not list ALA as a supported data source

See https://docs.microsoft.com/en-us/power-bi/connect-data/power-bi-data-sources
ALA is not a supported data source explicitly absent

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: daf84287-d3e6-38f3-e67d-41b7aaf350ce
Version Independent ID: b03d56d0-1fba-bea4-6f02-97fbc5832651
Content: Import Azure Log Analytics data into Power BI - Azure Monitor
Content Source: articles/azure-monitor/platform/powerbi.md
Service: azure-monitor
Sub-service: logs
GitHub Login: @bwren
Microsoft Alias: bwren

@vboyev-MSFT Thank you for bringing this to our notice, we will look into it and get back to you shortly.

@vboyev-MSFT Apologies for delayed response. Post validation with our internal product team it was called out that Log Analytics was not listed explicitly as the data source with in PowerBI because it integrate as a generic data source via mquery. That's the reason we don't have it listed.
Hope the above information helps. At this point it appears that there's no further edits or updates needed to the documentation. We will go ahead and close this issue, however if you have any further queries please tag us in your reply and will be happy to reopen the thread and continue the conversation. Thank you


[Question] How to incrementally calculate indicators

In the live trading setting instead of backtesting one, how can I incrementally calculate the indicators whenever I receive new rows of data?

Hello @pinkfrog9,
You will need to install the development version since it has fixes and updates across the board as well as handling incremental calculations.
Here is the incremental test code demonstrated in tests/test_studies.py
@pytest.mark.parametrize("talib", [False, True])
def test_study_all_incremental_rows(df, all_study, talib):
    MAX_ROWS = 90
    df = df.iloc[:MAX_ROWS]   # Trim for this test

    for i in range(0, MAX_ROWS):
        _df = df.iloc[:i]
        _df.ta.study(all_study, cores=0, talib=talib)
        # Break when max columns reached
        if _df.shape[1] - df.shape[1] == ALL_COLUMNS:
            assert _df.shape[1] > df.shape[1]
            break

Essentially put your indicators in a Study, then when the DataFrame receives a new row(s) call df.ta.study(your_study, cores=0, **kwargs) to update the DataFrame with the new calculations.
Kind Regards,
KJ

Thanks for the detailed reply.


Rename WebpushFcmOptions to WebpushFCMOptions

We should do this rename in order to be consistent with Go conventions. This is unfortunately a breaking change, and therefore cannot be implemented until we plan a major release.

Yes this is a breaking change, But i think the option currently not working because the param name is  fcmOptions instead of fcm_options (this is fixed in #256  )

Backend API accepts both fcmOptions and fcm_options. This was confirmed with the FCM team a while ago.

Fixed in the v4 branch


[circleci] Build swagger reference - this is an automatic commit [skip ci]

A later version exists.


message timestamp tests

This pull request adds tests for message timestamps in the protocol.

hmmm. saw the large message issue with you. I'm on the trail.

Situation: the issue is in OwnedPartition.enqueue_messages. It does not know that the topic is compressed (maybe it isn't...) and so it expects meaningful message offsets, which it is not getting in the compressed message case. I suspect the solution is to ensure that the message set  knows its messages were compressed.

BTW from the protocol text:

When the producer is sending compressed messages, to avoid server side recompression, each compressed message should have offset starting from 0 and increasing by one for each inner message in the compressed message.

So I added a flag to Messages so the message_offset is not taken into account in ordering, and this test passes.
This probably has more far-reaching impact. I have no clue why this test passed earlier; I suspect the change of protocol changed the distribution of messages between brokers.

I don't expect this to pass as such, please add https://github.com/maparent/pykafka/commit/2af4662dde7e326b9ebade42116f2c318d4d24b7

(Sorry, I have to log off for the evening. Best of luck.)

I saw tests failed, your commit is different from mine; offsets start at 1, mine still start at 0.
BTW: I'll be off next week, so best if we can resolve this today.

Codecov Report

Merging #693 into master will increase coverage by 1.03%.
The diff coverage is 94.44%.

@@            Coverage Diff             @@
##           master     #693      +/-   ##
==========================================
+ Coverage   83.34%   84.37%   +1.03%
==========================================
  Files          25       25
  Lines        3320     3348      +28
  Branches      527      532       +5
==========================================
+ Hits         2767     2825      +58
+ Misses        408      379      -29
+ Partials      145      144       -1

Impacted Files
Coverage Δ

pykafka/cluster.py
70% <ø> (-0.44%)
:arrow_down:

pykafka/broker.py
90.9% <100%> (+0.77%)
:arrow_up:

pykafka/protocol.py
96.67% <93.93%> (+0.25%)
:arrow_up:

pykafka/rdkafka/producer.py
87.22% <0%> (-0.57%)
:arrow_down:

pykafka/balancedconsumer.py
89.67% <0%> (+1.47%)
:arrow_up:

pykafka/simpleconsumer.py
82.59% <0%> (+1.96%)
:arrow_up:

pykafka/managedbalancedconsumer.py
79.16% <0%> (+2.08%)
:arrow_up:

pykafka/handlers.py
97.76% <0%> (+2.23%)
:arrow_up:

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 83d31dd...02a38c2. Read the comment docs.

@maparent This change is looking good to go, so I'm merging it via this PR. Thanks again for the contribution - I'll tag you in any additional changes that need to happen around message timestamps.

Thank you very much!


Fix date for release period

Signed-off-by: Georg J.P. Link<EMAIL_ADDRESS>

Sorry, missed this in the review of the original PR


Add Black and reformat

This PR is quite large in that it touches probably every file in the code base. The main contributions are as follows

Added make init which will setup a virtual environment and install all of the dependencies in one step.
Add black formatting. This has formatted all of the code.
Added a make format option which will run black on the repo. So this will do formatting automatically.
Modified the make lint option that will check for black or flake8 errors but does not correct them.
Added flake8 and black commit hooks so it will automatically check for black or flake8 errors before allowing commits. This way all commits will be compliant.

The main reason for doing all of this is so that the code has one standard format and that format is automatic (i.e. make format). This will help out with CI as you don't have to wait for all of tests to run just to find out that you have a linting error.
This PR is only python3 compatible. It can probably be modified to cover python2 as well but I figured you all will be dropping python2 support soon. Also this PR sets the max line length to 120 instead of 80 which I think is more readable but this option can be set in the config files.

I would vote for reducing the max line length.  For instance this 118 character line seems too long:
        ans = (1.0 / np.sqrt(2)) * (ss.sph_harm(mm, ll, phi, theta) + ((-1) ** mm) * ss.sph_harm(-mm, ll, phi, theta))

There are a few simultaneous reformatting projects going on.  #200 which is a bit stale, but was meant to drop python 2.7 support.  #211 which moves some enterprise_extensions stuff to enterprise base as part of a larger enterprise_extensions reformatting.
We should combine these reformatting projects.  I think @Hazboun6's work is the most "disruptive", so maybe that should be the one PR to start from.
What do we think of at @jellis18 and @paulthebaker making PR's of their changes in to that branch?
Otherwise we could coordinate merges and or rebases to get all of this into master in separate PRs is some planned order.

@paulthebaker, I've merged my changes with @Hazboun6's here.
In terms of the line length. That is totally up to you guys.

This all sounds good to me. I am guessing I should have merged before my last set of changes though? My new tests are running, then I can work on merging with @jellis18 's branch.

@Hazboun6, I submitted a PR to your branch


Remove Rewind and FastForward capabilities from Netcast service

It fixes this https://github.com/ConnectSDK/Connect-SDK-Android/issues/291

There are a few extra changes related to this in the iOS fix: https://github.com/ConnectSDK/Connect-SDK-iOS-Core/pull/33/files  They should also be implemented on Android.


Covers with different opening times

My covers in Dobiss have different opening times.
2 covers need approx. 25seconds
1 cover in the living room (high window) needs 50seconds
Is there a way to configure different opening times for different cover entities ?
And if not, could this be implemented one day ?
many thanks,

This is already implemented, but not well documented indeed :).
See https://community.home-assistant.io/t/dobiss-nxt-support/245369/137 as well.
for examlpe, what I have in my customize.yaml file is this:
cover.bureel:
  travelling_time_up: 34
  travelling_time_down: 32
cover.bibliotheek:
  travelling_time_up: 55
  travelling_time_down: 50
cover.dressing:
  travelling_time_up: 33
  travelling_time_down: 33
cover.slk_ouders:
  travelling_time_up: 49
  travelling_time_down: 49
cover.duplex:
  travelling_time_up: 40
  travelling_time_down: 37

You might need to restart home assistant after modifying this file...

Thank you very much, that worked fine for the curtains, but I forgot to mention that I also have 2 pulse operator covers : zwembad en garage.
As the pulse only lasts 1sec, these timed covers obviously don't work.
So, I thought I could disable the timed covers features on the garage and zwembad by setting the up and down time to zero, but apparently that does not work. I have :
customize:
    cover.gordijnen_keuken:
      travelling_time_up: 37
      travelling_time_down: 35
    cover.gordijnen_groot_raam:
      travelling_time_up: 90
      travelling_time_down: 73
    cover.gordijnen_zithoek:
      travelling_time_up: 37
      travelling_time_down: 35
    cover.garagepoort:
      travelling_time_up: 0
      travelling_time_down: 0
    cover.zwembad:
      travelling_time_up: 0
      travelling_time_down: 0

Is there another way to disable the timed covers only for garagepoort & zwembad ?
I could of course also set the pulse of both covers in Dobiss to the time it takes to open/close, but then the stop button still does not work the way it's supposed to be. Stop is supposed to send another pulse command (either the same or reverse direction) to stop.

They are indeed all configured as timed, or all as non-timed.
Could you have a look if you can easily create your cover from the buttons itself using https://www.home-assistant.io/integrations/cover.template/? I think that will be much easier to do than trying to add this to the dobiss integration itself. If it doesn't work, I can have a look...

Well, I already started implementing it differently by setting a delayed off of 3'30" on the Dobiss outputs and using the following automation to send a short trigger to stop the pool cover when HASS expects it to be in a stopped state 👍
alias: Zwembad - Stop Cover
description: ""
trigger:

platform: state
entity_id:

cover.zwembad
from: opening
to: unknown

platform: state
entity_id:

cover.zwembad
from: closing
to: unknown
condition: []
action:

choose:

conditions:

condition: state
entity_id: binary_sensor.zwembad_inp1_cover_opening
state: "on"
sequence:
service: cover.open_cover
data: {}
target:
entity_id: cover.zwembad
delay:
hours: 0
minutes: 0
seconds: 0
milliseconds: 500
service: cover.stop_cover
data: {}
target:
entity_id: cover.zwembad

conditions:

condition: state
entity_id: binary_sensor.zwembad_inp2_cover_closing
state: "on"
sequence:
service: cover.close_cover
data: {}
target:
entity_id: cover.zwembad
delay:
hours: 0
minutes: 0
seconds: 0
milliseconds: 500
service: cover.stop_cover
data: {}
target:
entity_id: cover.zwembad
mode: single

I dropped the whole automation thing above (dodgy behaviour and far from elegant) and created a proper cover template as you suggested.
Entity cover.zwembad is the Dobiss cover which is controller by pulses. I only use this as an input to this new template:
cover.pool_cover which the new cover I use all my HASS cards
- platform: template
  covers:
    pool_cover:
      device_class: shutter
      friendly_name: "Zwembad Cover"
      unique_id: pool_cover

      value_template: >
        {% if is_state('binary_sensor.zwembad_inp1_cover_opening', 'on') %}
          opening
        {% elif is_state('binary_sensor.zwembad_inp2_cover_closing', 'on') %}
          closing
        {% elif is_state('binary_sensor.zwembad_inp3_cover_open', 'on') %}
          open
        {% elif is_state('binary_sensor.zwembad_inp4_cover_closed', 'on') %}
          closed
        {% else %}
          stopped
        {% endif %}

      position_template: >
        {% if is_state('binary_sensor.zwembad_inp1_cover_opening', 'on') %}
          50
        {% elif is_state('binary_sensor.zwembad_inp2_cover_closing', 'on') %}
          50
        {% elif is_state('binary_sensor.zwembad_inp3_cover_open', 'on') %}
          100
        {% elif is_state('binary_sensor.zwembad_inp4_cover_closed', 'on') %}
          0
        {% else %}
          50
        {% endif %}

      open_cover:
        if:
          - condition: state
            entity_id: binary_sensor.zwembad_inp2_cover_closing
            state: "on"
        then:
          - service: cover.close_cover
            data:
              entity_id: cover.zwembad
          - delay:
              milliseconds: 500
          - service: cover.open_cover
            data:
              entity_id: cover.zwembad
        else:
          - service: cover.open_cover
            data:
              entity_id: cover.zwembad

      close_cover:
        if:
          - condition: state
            entity_id: binary_sensor.zwembad_inp1_cover_opening
            state: "on"
        then:
          - service: cover.open_cover
            data:
              entity_id: cover.zwembad
          - delay:
              milliseconds: 500
          - service: cover.close_cover
            data:
              entity_id: cover.zwembad
        else:
          - service: cover.close_cover
            data:
              entity_id: cover.zwembad

      stop_cover:
        choose:
          - conditions:
              - condition: state
                entity_id: binary_sensor.zwembad_inp1_cover_opening
                state: "on"
            sequence:
              - service: cover.open_cover
                data:
                  entity_id: cover.zwembad
          - conditions:
              - condition: state
                entity_id: binary_sensor.zwembad_inp2_cover_closing
                state: "on"
            sequence:
              - service: cover.close_cover
                data:
                  entity_id: cover.zwembad

      icon_template: mdi:pool

The challenge is now to make them timed. I know the time to fully open or close the pool covers is 185secs
I know how to create a timer in automations and use that timer to set the cover template position, but now I am getting more familiar with the cover template, it would be more elegant to implement that timer into the cover template.
Any ideas or suggestions ?

Would this be something you can use? https://community.home-assistant.io/t/custom-component-cover-time-based/187654


Datagrid examples don't work

Hi,
I'm running an app that was created with "shiny create" -> "dashboard example".  That app works seems to start up nicely, but it does not show the datagrid.
The reason behind this, is most likely this: there are errors in the Javascript console (Google Chrome) telling that the mime type is not correct.  I attached a screenshot.

I have tested with both Shiny core as Shiny express.  Both samples of the datagrid (core / express) seem to have the same problem.

I'm unable to replicate this -- would you mind including another screenshot of what you see when going to the "Network" tab and select the "dataframe.js" file? I'm mainly interested in the response headers. This is what I see:

Hi, sorry for the late response.  Long weekend here in Belgium ...

Further information:
I'm running on Windows, with a conda environment created as local admin because I need to write in c:\ProgramData\Miniconda3, which is by default protected by our company.
Strangely, I have also a WSL environment on the same machine.  When I run this in a conda environment that is rebuild with the same packages, and I run it from WSL, I do get to see the data-view.
Here's the same info as above, but when started from WSL


truvari: disable some tests on darwin

Description of changes
add patches to darwin in order to fix truvari anno grm/trf commands.
The upstream is tracking it here. https://github.com/ACEnglish/truvari/issues/154
ZHF: #230712

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[x] x86_64-darwin
[ ] aarch64-darwin

[x] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Looks like we could apply https://github.com/ACEnglish/truvari/commit/31416552008a506204ed4e2add55474f10392357 now.

Result of nixpkgs-review pr 231598 run on x86_64-darwin 1

2 packages built:

truvari
truvari.dist

Result of nixpkgs-review pr 231598 run on aarch64-darwin 1

2 packages marked as broken and skipped:

truvari
truvari.dist

Is this expected to work on aarch64-darwin or is it still broken? If still broken, this is ready to go.

@RaitoBezarius san, the nixpkgs-review results are correct because the "bwa" on which it depends only supports x86_64.
I submitted a PR for aarch64 support in #231635, but it is out of the scope of the ZHF.

@RaitoBezarius san, the nixpkgs-review results are correct because the "bwa" on which it depends only supports x86_64. I submitted a PR for aarch64 support in #231635, but it is out of the scope of the ZHF.

@natsukium san, thank you for your answer. This is good to go :).