Add studyStopReasonCategory to the ChEMBL evidence

As a developer I want to categorise the reasons why a Clinical Trial has stopped in order to analyse ChEMBL evidence in a new dimension and score the evidence accordingly.
Background
This ticket belongs to the epic where we want to integrate @LesyaR's NLP model onto the Platform's pipelines.
The NLP model can be downloaded here: gs://ot-team/olesya/bert_trials
Olesya has put together some scripts to run predictions:

common_classes.py: boilerplate code to initialise the BERT model
predictions.py: code to run prediction on a Pandas DataFrame

Tasks
The idea is to add a new column called studyStopReasonCategory that is a result of applying the model on studyStopReason. We want to achieve this in a computationally efficient way.

[ ] Run predictions.py by importing ChEMBL's data in a Pandas DF. Evaluate performance.
[ ] Move the logic to PySpark. The most straightforward way is to encapsulate the create_predictions method in an UDF. Evaluate performance.

Acceptance tests
How do we know the task is complete?

When I check the CT NCT05075902, reason to stop is COVID19 and the assigned category is Covid19.
When I check the CT NCT04958967, reason to stop is The protocol need to review and the assigned category is Study_Design.
When I check the CT NCT05067322, reason to stop is Low enrollment at the site. and the assigned category is Insufficient_Enrollment.
When I check the CT NCT05048511, reason to stop is The project was abandoned because of a lot of publications on the subject in the meantime and   it was not considered relevant to continue. and the assigned category is Another_Study.

At the moment I am stuck at task #1 because I cannot reproduce the env to successfully load the model. This is the error I am having:
ModuleNotFoundError: No module named 'transformers.modeling_bert'

Keep in mind there are 2 levels of categories. For example, the classified stop reason Insufficient enrollment is also catalogued as Neutral reason as the problem is considered to be independent of the study design.
The high-level classes are the result of a one-2-one mapping from the low-level classes.

I committed some changes in common_classes.py and predict.py scripts. The common_files now has the mapping to superclass function that is called when outputting the data. The hardcoded path names are now parametrised, so the script should be callable as: python3 predict.py input_file output_stopped_file output_nonstopped_file
The commit comment contains more details on the changes.

Data analysis
The enrichment of the categorisation for the reasons why a clinical trial ends on the evidence of ChEMBL 22.02 is as follows:

The vast majority (~89%) of the evidence will not contain this information because it does not have a reason to terminate. These are mostly 'Completed' studies and therefore do not enter into the analysis.
Among those that do enter the analysis, the model is not able to assign a class to 3213 records.
For those with results, most are assigned to only one class.

This is a break down of how the different subclasses are distributed:

Categories with the most interest are Negative and Safety_Sideeffects.
By taking a look at examples of Possibly_Negative, I don't think these should be relevant for the scoring, as they are indeed related to administrative reasons.

Workflow
This is what I followed to extract these results.
On a separate branch I am working to do the prediction part in the ChEMBL.py module. This is still WIP and will track the discussion in another ticket.
1. Prepare input data from ChEMBL evidence
evd = spark.read.json('data/cttv008-20-01-2022.json.gz')

studies = (
    evd

    # Extract studies with their reasons to stop
    .filter(F.col('studyStopReason').isNotNull())
    .withColumn('urls', F.explode('urls'))
    .filter(F.col('urls.niceName').contains('ClinicalTrials'))
    .withColumn('nct_id', F.element_at(F.split(F.col('urls.url'), '%22'), -2))
    .select('nct_id', F.col('studyStopReason').alias('why_stopped'))
    .distinct()
)

studies.coalesce(1).write.csv('data/studies.tsv', sep='\t', header=True)

2. Apply  [[BERT]] model
By calling the predict.py  module written by Olesya and edited by me.
Code and instructions on how to run it can be found here: https://github.com/ireneisdoomed/stopReasons
This script loads the model, instantiates the BERT Tokenizer and classifier and returns the predictions in a TSV file.
3. Build prediction on top of the ChEMBL evidence
Build the output from step 2 onto the latest ChEMBL submission.
This is handled by the new module ChEMBL.py
Final evidence schema:
root
 |-- clinicalPhase: long (nullable = true)
 |-- clinicalStatus: string (nullable = true)
 |-- datasourceId: string (nullable = true)
 |-- datatypeId: string (nullable = true)
 |-- diseaseFromSource: string (nullable = true)
 |-- diseaseFromSourceMappedId: string (nullable = true)
 |-- drugId: string (nullable = true)
 |-- studyStartDate: string (nullable = true)
 |-- studyStopReason: string (nullable = true)
 |-- targetFromSource: string (nullable = true)
 |-- targetFromSourceId: string (nullable = true)
 |-- urls: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- niceName: string (nullable = true)
 |    |    |-- url: string (nullable = true)
 |-- studyStopReasonCategory: array (nullable = true)  <--- New field
 |    |-- element: string (containsNull = true)


Take Me There button in notification window does not change map depth

When a notification occurs that is centered on a tile at a different depth, pressing take me there does not change depth, just centers the screen on the current depth.
The Signal used to trigger the take me there event does not have a field currently to pass the depth of the event. I didn't look into how to update a signal to take 2 parameters.
class NotificationWindow : public Window
{
public:
	using Signal = NAS2D::Signal<NAS2D::Point<int>>;

We should probably have a struct representing a 3D map coordinate. That's something that's been on my mind for a long time now.

That's probably a better option than what I was thinking. Could just make a 3-value struct, something like Coordinate in the TileMap, e.g., TileMap::Coordinate and just have it effectively be a 3-value point. Can either use the NAS2D::Point for the position and a third int Depth or just use three int values: int x, y, depth;


Datasets for deepbio

Hi folks, this issue is to brainstorm what kind of datasets can be useful to deepbio projects. Some good starting points will be datasets on genomics, proteins but this broad. More specific datasets and their use cases will be helpful.

Hi! I've been looking at some popular datasets that join the scope of biology and deep learning. The main topics are genomics, epigenomics, proteins, and clinical & healthcare. Here is the reference repo of the datasets.
Fascinating projects gather data such as 1000 Genomes, Encode Project, or NCBI Proteins. I'm excited to receive comments about these topics and datasets. Finally, which topics can we include in this brainstorming, to start discussing and looking for data?

Data from Bone Marrow Mononuclear Cells (BMMCs) would be a great candidate due to the reasons outlined in this video (time stamp 14:20) outlined below and in the picture attached:

The system is well understood
We know cell types present in the bone marrow and what surface markers are expressed
Mixture/Variety of different cell types (both spectrum and clusters)
Commercially available

The data is also openly available here.


Performance loss in NIeR Automata since 0.54

This is the only title I have available, so I don't know if this would affect the performance of other titles under my system. from 0.53 to 0.54 the game went from (looking to a wall) from ~38fps to 27fps, and when moving around  from ~32fps to ~18fps as well as not running at full speed anymore.
System information

GPU: AMD HD 7770 (1GB)
Driver: radv (tried git and 18.1.4 18.1.5 and same result, will be using stable for this test)
Wine version: 3.13 staging
DXVK version: 0.53/0.54

Log files
Might not be useful for this, but uploading just in case. Will update when I know what's the last good commit for me.
NieRAutomata_d3d11.log
NieRAutomata_dxgi.log
-Edit-
my distro's packages were updated so I'll be using 18.1.5 now.
I'm installing some dependencies for building dxvk.

Can't reproduce, 0.63 is actually faster than 0.53 on my RX 480.

I'm aware that my gpu is underpowered, which is the reason why I run it at the lowest setting (and resolution), and as the difference is quite pronounced for me between 0.53 and 0.54 (and beyond) and that someone else also seemed to have performance issues after 0.54 with an nvidia card and a different game, I thought that there might be something from that point that might be causing some bottleneck issue with underpowered cards. So, who knows, maybe this can be improved, or maybe it's something that can't be changed, but to try to find what is causing the problem here I thought it'd be a good idea to know what commit introduced the problem for me.
I'm still downloading some dependencies, but I'd like to know if, after starting git bisect, and when running
./package-release.sh master /your/target/directory --no-package
we should replace "master" with the hash of the commit in question.

if we should replace "master" with the hash of the commit in question.

Not necessary, as this parameter merely sets the name of the output directory.
Anyway, there's nothing new in 0.54 that should reduce performance, and for me it definitely doesn't. Let's see what your bisect uncovers, although I doubt that anything can be done about it.

Sorry for the delay, first time I used git bisect, I'm a bit confused why it seemingly skips some commits but... I found where there performance changed for me
commit f77392a

commit a9eff13

(I understand if this is the sort of thing that just can't be fixed I imagine this could be the sort of change that might tax my gpu while providing benefits to basically everyone else).

I guess a9eff13b920c65ec020fbf170514731980efefc1 could be causing that since it prevents DXVK from allocating more device memory than your GPU actually has. Can you test this patch:
dxvk-allow-overcommit.patch.txt

yep, it's working like before with that patch:

Confirming the issue. There is about 2x-3x performance improvement on 0.53 vs 0.64 on Everspace.
Maybe DXVK can check on startup if there is enough free RAM and enable overcommitting for GPUs with less than 3GB RAM? This way, GPUs that can use RAM would be able to do so.

Of course the proper fix would be to dynamically move textures between host memory and video memory in low-memory conditions, but given that pretty much everyone has 4GB or more VRAM I don't think it's worth the development time, or the additional overhead that would also affect GPUs that do have enough VRAM.
I'll consider adding a knob to enable the old behaviour with an environment variable.

@doitsujin I tested the patch on my GTX970 w/4GB vram, and i am not sure if its a placebo effect or not, but it seems as there is slightly less stuttering when playing World of Warcraft.
WoW uses 1.3-1.5GB vram on my settings, so im nowhere near any vram cap
Is this a "ongoing check" that is being done, and thus maybe creating a slight overhead each time new textures is loaded or shuffled around in vram? WoW i guess does this a lot every time you move around and new stuff pops into view :)

Is this a "ongoing check" that is being done

No, this check is done when allocating a new slice of device memory, which actually happens very rarely (usually just a few dozen times in total when running a game).

With latest master, you can create a configuration file dxvk.conf next to your NieRAutomata.exe with the following contents:
dxvk.allowMemoryOvercommit = True
See https://github.com/doitsujin/dxvk/wiki/Configuration for more details.


Pend example and generalize eepos size

added Brian's pend example, which necessitated sqp to take in any eepos size (was originally hardcoded to match the iiwa and indy robots). Also fixed some memory allocation bugs that were uncaught.

Looks good to me!


[Temporary] Run Namada action with custom Github runner

Closes: #XXX
Description

This PR is used to test the custom Github runner

PR author checklist:

[ ] Added changelog entry, using unclog.
[ ] Added tests: integration (for Hermes) or unit/mock tests (for modules).
[ ] Linked to GitHub issue.
[ ] Updated code comments and documentation (e.g., docs/).
[ ] Tagged one reviewer who will be the one responsible for shepherding this PR.

Reviewer checklist:

[ ] Reviewed Files changed in the GitHub PR explorer.
[ ] Manually tested (in case integration/unit/mock tests are absent).

Issue was solved, this PR can be closed


Fix error modal in user merge frontend

As noted in #168, the issue is that things thrown asynchronously (e.g. in a callback) get handled in some other context, which doesn't appear to get picked up anywhere within Nuxt/Vue.
The main way this manifests is withLoading, which one has to wait on to have the error thrown within the app context, though it appears in other places (like the confirm dialog, as fixed here)
To sanity check this theory, I found another un-awaited withLoading, on RunAnalysis. I intentionally broke the endpoint and confirmed that no error dialog appears before the changes I made in this PR.
LMK if you want to comb the codebase for these, or if you'd like me to.
Fixes #168, in two cases

Closing in favor of #177


Velox does not install folly in centos8 by default, break compile in centos8.

Describe the bug
Velox remove install folly in centos 4f528, when compile gluten, find_package(folly) will throw error.
To Reproduce
./tools/one_step_veloxbackend.sh or separate build velox then build gluten can reproduce.
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Velox still install folly in setup_ubuntu.
So, if i execute cd /path-to-gluten/ep/build-velox/build/velox_ep/_build/release/_deps/ & cmake --install folly-build, everything would be ok.
What's community's options about this? I could submit a PR if needed.

How about using resolve_dependency macro as velox does in 4f528? @FelixYBW

This just should be added to the centos8 deployment
(
  git clone https://github.com/facebook/folly &&
  cd folly &&
  git checkout $FB_OS_VERSION &&
  cmake_install -DBUILD_TESTS=OFF -DBUILD_SHARED_LIBS=ON
)

@zhejiangxiaomai

I already fix this issue here https://github.com/oap-project/gluten/pull/994/files#diff-e2b6a0ac0f3384d96902fbc545a697d0aa305c7ed711493df5e1bddf6c6d401b. we can close this isssue.


docs(main): fix change owner

https://docs.github.com/zh/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsif


sendshortcut binds broken in certain applications

Bug or Regression?
Bug
System Info and Version
 System/Version info
Hyprland, built from branch main at commit 91fe58f8f278d126852877eadc87c50ca7b9b78d  (window: improve swallowing functionality).
Date: Sat Jun 15 11:20:41 2024
Tag: v0.41.1-18-g91fe58f8, commits: 4836

flags: (if any)

System Information:
System name: Linux
Node name: arch
Release: 6.9.4-arch1-1
Version: #1 SMP PREEMPT_DYNAMIC Wed, 12 Jun 2024 20:17:17 +0000

GPU information:
2b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1) (prog-if 00 [VGA controller])

os-release: NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

Description
sendshortcut binds do not working in certain applications like minecraft whilst in other applications it works flawlessly ex: firefox
bindl = , F8, sendshortcut,CTRL SHIFT, m, class:^(vesktop)
How to reproduce

load the keybind into your configuration
focus in on certain application ex: minecraft (from prismlauncher AUR)
activate bind
check if bind succeeded in my use case vesktop mute/deafen

Crash reports, logs, images, videos
No response

It might have something to do with XWayland as per #6174.

This diff removes some redundant comments, cleans up code, and makes wl -> xwl work, leaving xwl -> xwl in a kinda broken state (will send to current)

the sendshortcut and pass dispatchers only work when the window is focused for electron and chromium applications running in wayland mode ( --enable-features=UseOzonePlatform --ozone-platform=wayland).
Ive tested:
Vesktop (wayland mode)
Spotify (wayland mode)
Chromium (wayland mode)
Results
All of them had the exact same behaviour.
The bind and sendshortcut dispatchers only worked when the window was being focused.
I tested this with binds and the hyprctl dispatch method.

I tested the keybinds on slack, brave, & firefox and it only works on firefox.
# works fine and opens a new tab in firefox
hyprctl dispatch sendshortcut 'CTRL, T, class:^(firefox)$'
# same command doesn't work in brave
hyprctl dispatch sendshortcut 'CTRL, T, class:^(brave-browser)$'

I also tried it in slack with wayland flags and it doesn't work
slack --enable-features=WebRTCPipeWireCapturer --enable-features=UseOzonePlatform --ozone-platform=wayland
hyprctl dispatch sendshortcut 'CTRL, K, class:^(Slack)$'

I've been running into the same issue, and I'm quite sure that the issue seems to occur when trying to use sendshortcut while the current window is an xwayland window.
Interestingly, hyprctl dispatch sendshortcut ... reports "ok" even in this situation, despite the shortcut not actually being sent.

I run into the same issue and @JonathanSteininger seems to be on a good track: it only happens with chromium / electron applications running under wayland. You need to focus the window to make it work.
I tested with other, non electron / chromium applications working on wayland (like wezterm) and the dispatcher works as intended with those.

With the v0.46.2 rebuild with the new hyprutils, sendshortcut is only working when the target window is focused for me. It doesn't work when native wayland applications are focused anymore.


Add file toggling

@ThePrimeagen this PR adds file toggling and also updates rm_file to allow passing in the same arguments that add_file accepts.

@brandoncc this looks good.  We should extract the file is marked check from add_file into a function and just use that instead of add_file to call to see if it adds.

I was thinking the same thing, but didn't do it because it just works 😂 I'll make that change.


TMDB API key

Great walkthrough!
One recommendation is to replace your actual TMDB API key with a placeholder. That way no one can abuse your account via your API key.
P.S. Super nitpicky, but in that same block, I think the Jupyter step should read In [5]:

Thanks for pointing that out @bwobst!
@AnshulBasia I suggest deleting your TMDB account. Even if you change the version on github now, anyone can still access your key from previous commits. Also, act on this fast I would suggest.


Add packaging for hooks/README.md

Signed-off-by: baude<EMAIL_ADDRESS>

LGTM

@rh-atomic-bot r+


Fix client SDK build

After some of the code changes for Snapps, the client SDK was not building.
Fixed by moving some code inside consensus_mechanism blocks, opening Snark_params_nonconsensus where needed.

Subsumed by #9470.


Fix a test failure on autopilot cluster

The gcenode-askpass-sidecar container's resource requirements will be adjusted on the autopilot cluster, so the generation should bumpt by 2.

/lgtm
/approve


Unable to set up integration - No module stream_unzip

HA is not loading the dwd integration with the following error log:
Logger: homeassistant.setup
Quelle: setup.py:334
Erstmals aufgetreten: 14:20:44 (1 Vorkommnisse)
Zuletzt protokolliert: 14:20:44
Setup failed for custom integration 'dwd_weather': Unable to import component: No module named 'stream_unzip'
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component
component = await integration.async_get_component()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/loader.py", line 1027, in async_get_component
self._component_future.result()
File "/usr/src/homeassistant/homeassistant/loader.py", line 1007, in async_get_component
comp = await self.hass.async_add_import_executor_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/loader.py", line 1067, in _get_component
ComponentProtocol, importlib.import_module(self.pkg_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 995, in exec_module
File "", line 488, in _call_with_frames_removed
File "/config/custom_components/dwd_weather/init.py", line 12, in
from simple_dwd_weatherforecast import dwdforecast, dwdmap
File "/usr/local/lib/python3.12/site-packages/simple_dwd_weatherforecast/dwdforecast.py", line 11, in
from stream_unzip import stream_unzip
ModuleNotFoundError: No module named 'stream_unzip'
I am using HAOS.
HA Core
2024.10.3
Supervisor
2024.10.2
Operating System
13.2
Frontend
20241002.3

Same here ...
Logger: homeassistant.setup
Source: setup.py:334
First occurred: 15:42:46 (1 occurrences)
Last logged: 15:42:46

Setup failed for custom integration 'dwd_weather': Unable to import component: No module named 'stream_unzip'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component
    component = await integration.async_get_component()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1027, in async_get_component
    self._component_future.result()
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1007, in async_get_component
    comp = await self.hass.async_add_import_executor_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1067, in _get_component
    ComponentProtocol, importlib.import_module(self.pkg_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/config/custom_components/dwd_weather/__init__.py", line 12, in <module>
    from simple_dwd_weatherforecast import dwdforecast, dwdmap
  File "/usr/local/lib/python3.12/site-packages/simple_dwd_weatherforecast/dwdforecast.py", line 11, in <module>
    from stream_unzip import stream_unzip
ModuleNotFoundError: No module named 'stream_unzip'

same here after update core to 2024.10.3
Logger: homeassistant.setup
Source: setup.py:334
First occurred: 14:46:17 (1 occurrences)
Last logged: 14:46:17
Setup failed for custom integration 'dwd_weather': Unable to import component: No module named 'stream_unzip'
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component
component = await integration.async_get_component()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/loader.py", line 1027, in async_get_component
self._component_future.result()
File "/usr/src/homeassistant/homeassistant/loader.py", line 1007, in async_get_component
comp = await self.hass.async_add_import_executor_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/loader.py", line 1067, in _get_component
ComponentProtocol, importlib.import_module(self.pkg_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 995, in exec_module
File "", line 488, in _call_with_frames_removed
File "/config/custom_components/dwd_weather/init.py", line 12, in
from simple_dwd_weatherforecast import dwdforecast, dwdmap
File "/usr/local/lib/python3.12/site-packages/simple_dwd_weatherforecast/dwdforecast.py", line 11, in
from stream_unzip import stream_unzip
ModuleNotFoundError: No module named 'stream_unzip'

i second that. Came with the update of the core to 2024.10.3

i run HA in docker with 2024.10.3 and see no such error.
are you all using HAOS like PottiMc?

Yes, I'm running HAOS

Same here. HAOS

As nothing was changed from my side, I guess this might be an error in HAOS not pulling all the requirements correctly.
stream_unzip is a requirement of my underlying library for data retrieval and is there declared as a dependency. What I could try, is to define these also as a requirement for the main integration. However I am not sure if this fixes this error.

@FL550 Thanks, would be great if you could give it a try. If it is not working, I'll open a issue for HAOS Core as it has been working with 2024.10.2.

If 2.1.18 is the pre-release, error is still present.

Can someone check if the requirements are now at least installed during boot?

See https://github.com/home-assistant/core/issues/128921 and https://github.com/home-assistant/core/issues/127966

@FL550 not sure whether this is what you are looking for, but it looks that stream_unzip is still "missing" with 2.1.18.
`Logger: homeassistant.setup
Source: setup.py:334
First occurred: 22:56:45 (1 occurrences)
Last logged: 22:56:45
Setup failed for custom integration 'dwd_weather': Unable to import component: No module named 'stream_unzip'
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component
component = await integration.async_get_component()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/loader.py", line 1027, in async_get_component
self._component_future.result()
File "/usr/src/homeassistant/homeassistant/loader.py", line 1007, in async_get_component
comp = await self.hass.async_add_import_executor_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/loader.py", line 1067, in _get_component
ComponentProtocol, importlib.import_module(self.pkg_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 995, in exec_module
File "", line 488, in _call_with_frames_removed
File "/config/custom_components/dwd_weather/init.py", line 12, in
from simple_dwd_weatherforecast import dwdforecast, dwdmap
File "/usr/local/lib/python3.12/site-packages/simple_dwd_weatherforecast/dwdforecast.py", line 11, in
from stream_unzip import stream_unzip
ModuleNotFoundError: No module named 'stream_unzip'`

I just went back to Core 2024.10.2
Issue persists with the same error DWD Weather 2.1.18

I'll post the issue which was filed on HA here for reference: https://github.com/home-assistant/core/issues/128921

For now you can open a shell inside your container and install the modules by hand:
python3 -m pip install simple_dwd_weatherforecast==2.1.10 markdownify==0.6.5 suntimes==1.1.2 lxml requests Pillow arrow stream-inflate==0.0.14 stream-unzip httpx

The problem also seems to be the same cause as in #155.
It looks like HA has changed how dependencies are installed in 2024.10 and this has broken many custom integrations.
So basically this error only occurs to new installations or upgrades where the package cache was deleted. When you did an upgrade without deleting the HA config folder, everything should still work.

Thanks, @FL550 , but I don't think you've got any influence on HA config folder unless you upgrade the HAOS core manually step by step. Most people (including myself 😅 won't do that). So crossing fingers that it will be corrected with the core soonest. Thanks for your efforts.

That's not true I think. I just clicked "Update" in my HA settings menu. Deleted nothing at all, especially not the config folder where all my stuff is.

I'm also not sure on this, as I am running HA in a container on 2024.10.1 in my production environment and everything is working fine for now.

It was working fine until 2024.10.2

@Teonyr Thank you for providing the logs. This means that the packages have been installed previously which is correct behavior. Therefore we have to wait until this is fixed in HA itself.
Please see #155 in the meantime as it might help here as well.

problem turned up after a restrat in my HA, too. couldn’t fix it with rolling back to 2024.10.2 or eveb rolling back to 2024.9.3.
what fixed it for me was „rolling back“ one minor version of stream-unzip (in manifest.json):
{
  "domain": "dwd_weather",
  "version": "v2.1.18",
  "name": "Deutscher Wetterdienst (DWD)",
  "documentation": "https://github.com/FL550/dwd_weather",
  "issue_tracker": "https://github.com/FL550/dwd_weather/issues",
  "dependencies": [],
  "config_flow": true,
  "codeowners": [
    "@FL550"
  ],
  "requirements": [
    "simple_dwd_weatherforecast==2.1.10",
    "markdownify==0.6.5",
    "suntimes==1.1.2",
    "lxml",
    "requests",
    "Pillow",
    "arrow",
    "stream-inflate==0.0.14",
    "stream-unzip==0.0.95",
    "httpx"
  ],
  "iot_class": "cloud_polling"
}

Wow, thanks for sharing. How did you figure that out?🤔

poking around … :)
since all other imports/dependencies worked fine, i figured that it must have something to do with stream-unzip and even before i saw the filed issue, noticed that the 0.0.96 release was from yesterday. so i pinned the version to 0.0.95 and it worked. actually i first ran this inside my docker:
pip3 install stream-unzip==0.0.95 --upgrade
seems to work with 2024.10.3, too — which is kind of strange.

I'm not sure if this fixes this for everyone as I am guessing this is due to the package has veen already in cache.
However I relaxed a new version which pins the version of stream_unzip to resolve it at least for the moment.

I can confirm, that the problem is gone with release v2.1.19.

Quick feedback also from me. Solved it here as well. Thanks!

Yeah. Thanks!

It seems you are all long time users then 😉

@FL550 Well, that really doesn't matter. The community lives up based on different roles. You develop and open it to the community (many thanks for that 🙏), we provide feedback to help improving quality. This is how it works 🙂, isn't it? Everybody is doing the best possible.

Thank you for your fast help. I am gonna close this now since the new release fixed it.

Workaround mentioned above worked for me to get DWD 2.1.24 running on HA 2024.11.03:
Executed within my docker container: pip3 install stream-unzip==0.0.95 --upgrade


Can't install chumpy

uv version
uv 0.4.7 (a178051e8 2024-09-07)

Reproduce:
uv add chumpy

It's similar to #1551 but the solution don't work:

Use uv init --lib to create a project that allows [build-system]
Modify the [build-system] to

Use uv add chumpy, and get an error

uv add pip/uv pip install pip then uv add chumpy/uv pip install chumpy don't work.
uv add pip/uv pip install pip then uv run pip install chumpy/uv run python -m pip install chumpy don't work. (NOTE: they work when there is a cached wheel.)

But pixi can handle this.
pixi add python pip
pixi run pip install chumpy

Sadly this is an issue with chumpy itself. The package is not properly structured -- it's documented as such here: https://github.com/astral-sh/uv/issues/2252#issuecomment-1981978204. For example, it also won't work if you run pip install chumpy --use-pep517, which will eventually be the default:
❯ pip install chumpy --use-pep517
Collecting chumpy
  Downloading chumpy-0.70.tar.gz (50 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [26 lines of output]
      Traceback (most recent call last):
        File "<string>", line 9, in <module>
      ModuleNotFoundError: No module named 'pip'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/Users/crmarsh/.local/share/rtx/installs/python/3.12.3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/crmarsh/.local/share/rtx/installs/python/3.12.3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       ...

(In the pixi example you included, that's just pixi running pip within an environment, there's no special handling there that differs from pip install.)
You can either use uv pip install chumpy --no-build-isolation, or see the docs on using build isolation with uv add and related APIs.

Thanks! I should google "uv can't install chumpy" instead of the error message! Hope this pointer can help others who google the error message!
Here are verified solutions following build isolation:
Solution 1:
[project]
name = "smpl-preprocess"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.10,<3.11"
dependencies = [
    "numpy<1.24",
    "scipy<2",
    "chumpy",
]

[tool.uv]
no-build-isolation-package = ["chumpy"]

Run
uv venv
uv pip install pip setuptools
uv sync

Solution 2:
[project]
name = "smpl-preprocess"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.10,<3.11"
dependencies = [
    "numpy<1.24",
    "scipy<2",
]

[project.optional-dependencies]
build = ["setuptools", "pip"]
run = ["chumpy"]

[tool.uv]
no-build-isolation-package = ["chumpy"]

It won't work if we follow build isolation
uv sync --extra build
uv sync --extra run

But I found that it will work if we run uv lock first: (@charliermarsh do you know why?)
uv lock
uv sync --extra build
uv sync --extra run

Does it work if you use uv sync --extra run --extra build for the second command, instead of uv sync --extra run?

the screen showed it failed at the first command, i.e. uv sync --extra build.

It looks like chumpy doesn't declare any static metadata so you have to build it even just to resolve the project dependencies. In that case, you can't use the strategy you described above (with separate optional groups). You have to pre-install the package with uv pip install unfortunately.

but separate optional groups do work, if I run uv lock first, as shown in another screenshot.
that's why I asked you🧐.

My best guess is that the package is cached at some point, and so the build isn't recurring. If you run uv cache clean before each set of commands, do you see more consistent behavior? For example, I still see this locally:
❯ uv lock
⠙ chumpy==0.70                                                                                error: Failed to download and build `chumpy==0.70`
  Caused by: Build backend failed to determine metadata through `prepare_metadata_for_build_wheel` with exit status: 1
--- stdout:

--- stderr:
Traceback (most recent call last):
  File "<string>", line 8, in <module>
ModuleNotFoundError: No module named 'setuptools'
---

But if I run uv venv && uv pip install pip setuptools && uv sync first, the package is now cached, so running uv lock works.

I suspected the cache too.
but I did run uv cache clean before(sorry I should include it in the screenshot), such that uv lock took 2.15s.

but "built chumpy" only took 2ms so there should be something cached…
I would try to reproduce it in a "fresher" environment…

Yeah I think to explore further I'd need a consistent reproduction (I'd suggest running uv cache clean and rm -f uv.lock before any sequence of commands).

I have written a GitHub Action to create a consistent reproduction.
https://github.com/YouJiacheng/chumpy-uv-test/actions/runs/10855047631/job/30126795524

Local Tests:
Windows:

Ubuntu:

Thank you, I'll take a look.

Can you rerun the Actions with --verbose on every command?

Done.
https://github.com/YouJiacheng/chumpy-uv-test/actions/runs/10856035081/job/30129895137
https://github.com/YouJiacheng/chumpy-uv-test/actions/runs/10856035837/job/30129897646

I think the difference is that uv lock runs in the system Python interpreter (since we're not doing any install operations), which happens to have setuptools installed in the environment (this isn't true of later versions of Python, like Python 3.12), whereas with uv sync, we run in the project virtual environment, which doesn't include setuptools by default. If the project had a build-time dependency other than setuptools and pip, both versions would fail. So it kind of works by accident.
You can probably get the uv lock version to fail (I assume) by running uv venv first?

Run uv venv --python 3.12 first:
https://github.com/YouJiacheng/chumpy-uv-test/actions/runs/10860520657
Run uv venv --python 3.12 first and use uv lock --python-preference only-managed instead of uv lock:
https://github.com/YouJiacheng/chumpy-uv-test/actions/runs/10860542823

I think it's better to have a more consistent behavior. Let uv lock be "polluted" by an arbitrary system Python Interpreter sounds undesirable.

Moreover, maybe we can add an extra option to skip "Preparing metadata" with a user provided metadata.
IIUC, only
Requires-Dist: scipy >=0.13.0
Requires-Dist: six >=1.11.0

in the METADATA is required by uv lock?

Closing in favor of the other issue.


Implemented VI mode change inside and delete inside functionality

This implements the change in (ci) and delete inside vi mode functionality:

ci" - Change in quotes
di" - Delete in quotes
ci[ - change in brackets

supports [ , ( , ", ', `, {

I was waiting for this feature for long. It made me curious about the future plans for nushell repl parser/highlighter, are we going to use treesitter here in reedline in near future? Since without AST info, it's hard to deal with nasty cases where brackets get nested like (foo(bar "(" ) baz)

Another thing that should be addressed at some point.. Since this is basically supplying two actions, it requires two undo commands to get back to the previous state.

I was waiting for this feature for long. It made me curious about the future plans for nushell repl parser/highlighter, are we going to use treesitter here in reedline in near future? Since without AST info, it's hard to deal with nasty cases where brackets get nested like (foo(bar "(" ) baz)

+1 to that — I didn't realize someone was already working on this and had started looking into how one might implement the full set of vim text object operations...

The basic pattern is  where:

Operation is something like d (delete), c (change), y (yank/copy), v (select)
Scope is either i (inside) or a (around)
Text object can be things like w (word), p (paragraph), ( (parentheses), " (quotes), } (curly braces)

Anyways — it seems like the way to be able to implement all of this stuff in a consistent way would be to use tree sitter.

reedline could be more useful to other shells since support for other languages
what @blindFS mentioned about handling complex nesting — actually it seems like maybe you've already started on improving https://github.com/nushell/tree-sitter-nu in support of this?

@bishopmatthew tree-sitter-nu's accuracy has been improved recently on its own. Feel free to report any parsing error or WA. As for the integration to reedline, I think it's not planned yet, and a general set of api to process ast/cst would be a better option on second thought. Using treesitter may not be the only option since it needs extra overhead compared to the native parser.
Actually there's a hacking way (in case you want the functionality right now) for limited textobj related ops with current reedline implementation: mapping to user defined command with internal ast and commandline for the dirty job.


Log rejected requests?

Is there a way to know when a request has exceeded a throttle and then trigger a logging event for that? I know we can track but I would like to know only when rack attack is blocking/rejecting a request.

@aurcioli-handy yup, that's totally possible. :smile:
It works by using ActiveSupport Notifications (see this example).
Here's an example subscriber that logs when a request is blacklisted or throttled:
LOGGER = Logger.new("log/rack-attack.log")
ActiveSupport::Notifications.subscribe('rack.attack') do |name, start, finish, request_id, req|
  if [:throttle, :blacklist].include? req.env['rack.attack.match_type']
    LOGGER.info [match, req.ip, req.request_method, req.fullpath, ('"' + req.user_agent.to_s + '"')].join(' ')
  end
end

Awesome, thankyou!

Fixed a couple of errors: match not defined, and :blacklist should be :blocklist.  Also generalized it to track all requests.
LOGGER = Logger.new("log/rack-attack.log")
ActiveSupport::Notifications.subscribe('rack.attack') do |name, start, finish, request_id, req|
  msg = [req.env['rack.attack.match_type'], req.ip, req.request_method, req.fullpath, ('"' + req.user_agent.to_s + '"')].join(' ')
  if [:throttle, :blocklist].include? req.env['rack.attack.match_type']
    LOGGER.error(msg)
  else
    LOGGER.info(msg)
  end
end

@jageschiff
I'm wondering where would you put this code? Thanks.

this is the content the file in my application config\initializers\rack_attack_logger.rb

After using the above code I was getting the following error:

NoMethodError (undefined method `env' for #Hash:0x005569f95f02b8)

so posting updated code for others who may encounter the same.
LOGGER = Logger.new('log/rack-attack.log')
ActiveSupport::Notifications.subscribe('rack.attack') do |_name, _start, _finish, _request_id, req|
  req = req[:request]
  msg = [req.env['rack.attack.match_type'], req.ip, req.request_method, req.fullpath, ('"' + req.user_agent.to_s + '"')].join(' ')
  if %i[throttle blocklist].include?(req.env['rack.attack.match_type'])
    LOGGER.error(msg)
  else
    LOGGER.info(msg)
  end
end

Hi @md-farhan-memon,
Thanks for the comment.
Indeed, that's because of the change in https://github.com/kickstarter/rack-attack/pull/256, which was first released in rack-attack v6.0.

A small tweak that I might suggest for the next person who happens on to this thread:
LOGGER = Logger.new("log/#{Rails.env}-rack-attack.log")

This gets your log file names in line with Rails logs: development-rack-attack.log, test-rack-attack.log, production-rack-attack.log, etc.

@bjhess et al -- thanks for the improvements. I updated the example above.


Fix BUG: can not create two hqps_actor_autogen target in one CMakeList

Only add_custom_target on installed_libclang on the first time.
Related Issue: #30

@nlighting Hi, could you review my PR?


Retriever and lvm update for multimodal rag on videos

Description
This PR introduces multimodal retriever microservice and provides update on LVM microservice. They are required for Multimodal RAG on Videos application. This allow users to retrieve text and its associated metadata as multimodal data from Redis vector store. The update on LVM microservice allows LVM accepts as input SearchedMultimodalDoc (additionally to LVMDoc). Such SearchedMultimodalDoc is the output of multimodal retriever microservice.
Issues
RFC: https://github.com/opea-project/docs/pull/49/files
Issue: https://github.com/opea-project/GenAIExamples/issues/358
Type of change
List the type of change like below. Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x ] New feature (non-breaking change which adds new functionality) Introducting multimodal retriever microservice
[ ] Breaking change (fix or feature that would break existing design and interface)
[ x] Others (enhancement, documentation, validation, etc.) Enhancing LVM microservice

Dependencies
docarray[full]
fastapi
langchain_community
opentelemetry-api
opentelemetry-exporter-otlp
opentelemetry-sdk
prometheus-fastapi-instrumentator
redis
shortuuid
uvicorn
transformers
llava-hf/llava-1.5-7b-hf
BridgeTower
langchain-core
Tests
We have provided 2 tests for these microservice.

tests/test_lvms_llava.sh. We provided 2 additional tests within validate_microservice() for new request type of LVMSearchedMultimodalDoc.
tests/test_retrievers_multimodal_langchain_redis.sh. This tests the introduced multimodal retriever microservice.

@lvliang-intel @XuhuiRen This is the last PR for GenAIComps on Multimodal RAG on Videos. Would you please review this? We will need this PR for GenAIExamples.
Thanks

hi, @tileintel @lvliang-intel I am confused about the directory structure,  what about using this directory structure comps/retrievers/langchain/redis and putting the multi-modal implementations into comps/retrievers/langchain/redis?
@XuhuiRen  do you have some comments for this?

hi, @tileintel @lvliang-intel I am confused about the directory structure, what about using this directory structure comps/retrievers/langchain/redis and putting the multi-modal implementations into comps/retrievers/langchain/redis?
@XuhuiRen do you have some comments for this?

Dear @lkk12014402.
Thank you for your feedback. At first, I totally agree that we can put this implementation in langchain/redis. However, we prefer having separate folder as multimodal_langchain/redis due to the following reasons:

Langchain as you know is developed mainly for language modality only. Thus, in order to extend langchain to accept multimodalities, we actually had to extend some langchain original classes to new multimodal corresponding class. We developed those classes but not yet submit PR to langchain open-source repo yet. Thus, having separate folders helps us to switch back to original langchain implementation easily in future if langchain community decides to accept multimodalities and have different implementation to what we have implemented.
You might not see any new implementation for langchain in this retriever microservice. However, there are quite a bit of them in another microservice like multimodal_embedding. And in other microservice implementation, we already have a folder structure like this, so we propose the similar folder structure here for consistency.
imho, langchain that accepts secondary modality like image is not original langchain anymore. It should be multimodal_langchain. And redis class here is an interface between langchain and Redis database. Similarly, new redis class (if existing) for multimodal_langchain should not be in langchain/multimodal_redis.

Given our observations above, would you please reconsider this? I hope these can address your concern.
Again, thank a lot for your insight feedback/question.

comps/retrievers/langchain/redis

I think it would be better to put the retriever under "comps/retrievers/langchain/". Currently, we category the type of the retriever according to their implementation framework. it is not reasonable to create a parallel folder with langchain and llamaindex.

Please revise the folder structure

@XuhuiRen: Sure, I can revise the folder structure as you suggested. However, if we leave these new files in langchain/redis/, what files' names should I use for README, Dockerfile and docker_compose yml files? There will be two README files: one for retriever microservice and one for multimodal retriever microservice. Also, should I name dockerfile as Dockerfile_multimodal? Please advise.
Given this, langchain/multimodal_redis is also not a good solution as well.
Thanks a lot

Please revise the folder structure

@XuhuiRen: Sure, I can revise the folder structure as you suggested. However, if we leave these new files in langchain/redis/, what files' names should I use for README, Dockerfile and docker_compose yml files? There will be two README files: one for retriever microservice and one for multimodal retriever microservice. Also, should I name dockerfile as Dockerfile_multimodal? Please advise. Given this, langchain/multimodal_redis is also not a good solution as well. Thanks a lot

you can temperately create a redis-multimodal folder in the retriever/langchain folder. we can schedule a movement to rename the folder structure like base_redis, multimodal_redis in the next release

Please revise the folder structure

@XuhuiRen: Sure, I can revise the folder structure as you suggested. However, if we leave these new files in langchain/redis/, what files' names should I use for README, Dockerfile and docker_compose yml files? There will be two README files: one for retriever microservice and one for multimodal retriever microservice. Also, should I name dockerfile as Dockerfile_multimodal? Please advise. Given this, langchain/multimodal_redis is also not a good solution as well. Thanks a lot

you can temperately create a redis-multimodal folder in the retriever/langchain folder. we can schedule a movement to rename the folder structure like base_redis, multimodal_redis in the next release

@XuhuiRen Thank for suggestions. I have revised folder structure as you suggested. In specific, all files are moved to comps/retrievers/langchain/redis_multimodal.
I have updated README and test_retrievers_langchain_multimodal_redis.sh accordingly.

Please revise the folder structure

@XuhuiRen: Sure, I can revise the folder structure as you suggested. However, if we leave these new files in langchain/redis/, what files' names should I use for README, Dockerfile and docker_compose yml files? There will be two README files: one for retriever microservice and one for multimodal retriever microservice. Also, should I name dockerfile as Dockerfile_multimodal? Please advise. Given this, langchain/multimodal_redis is also not a good solution as well. Thanks a lot

you can temperately create a redis-multimodal folder in the retriever/langchain folder. we can schedule a movement to rename the folder structure like base_redis, multimodal_redis in the next release

@XuhuiRen Thank for suggestions. I have revised folder structure as you suggested. In specific, all files are moved to comps/retrievers/langchain/redis_multimodal. I have updated README and test_retrievers_langchain_multimodal_redis.sh accordingly. Would you please review this PR again and merge if there is no new feedback?
Thanks a lot

some uts are not passed, please check~

Please revise the folder structure

@XuhuiRen: Sure, I can revise the folder structure as you suggested. However, if we leave these new files in langchain/redis/, what files' names should I use for README, Dockerfile and docker_compose yml files? There will be two README files: one for retriever microservice and one for multimodal retriever microservice. Also, should I name dockerfile as Dockerfile_multimodal? Please advise. Given this, langchain/multimodal_redis is also not a good solution as well. Thanks a lot

you can temperately create a redis-multimodal folder in the retriever/langchain folder. we can schedule a movement to rename the folder structure like base_redis, multimodal_redis in the next release

@XuhuiRen Thank for suggestions. I have revised folder structure as you suggested. In specific, all files are moved to comps/retrievers/langchain/redis_multimodal. I have updated README and test_retrievers_langchain_multimodal_redis.sh accordingly. Would you please review this PR again and merge if there is no new feedback?
Thanks a lot

some uts are not passed, please check~

One UT failed because of something happened in Circle CI in which it cannot clear the previously started dockers and some ports were not available for start some microservice for that UT. I have changed the port number in the tests. It seems working now.
@lkk12014402 @XuhuiRen @lvliang-intel Since all feedbacks are resolved and all the tests work, can you please merge this PR?
Thanks a lot

@XuhuiRen @lkk12014402 @lvliang-intel  Since all tests have passed and there is no more feedbacks, can anyone please merge this PR? Thanks a lot for helping this.


Context menu contribution does not warn when activation event is missing

Generate Yeoman sample extension.
Add a context menu contribution

    "activationEvents": [
        "onCommand:extension.sayHello"
    ],
    "main": "./out/extension",
    "contributes": {
        "menus": {
            "explorer/context": [
                {
                    "command": "extension.foobar",
                    "group": "1_modification"
                }
            ]
        },
        "commands": [
            {
                "command": "extension.foobarCon",
                "title": "Hello World"
            }
        ]
    },

'use strict';

import * as vscode from 'vscode';

export function activate(context: vscode.ExtensionContext) {
    let disposable = vscode.commands.registerCommand('extension.foobar', () => {
        vscode.window.showInformationMessage('Hello World!');
    });

    context.subscriptions.push(disposable);
}

Run the extension, open the context menu in the explorer, select Hello World
Nothing happens, no error message.
Run the hello world command from the command menu, error appears as expected.

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.
If you wonder what we are up to, please see our roadmap and issue reporting guidelines.
Thanks for your understanding, and happy coding!


pdf.js viewer Large pdf file Generate issue

i have to use my website pdf.js all project....
but i have 30mb pdf file view than create error
Version 2.0
please suggest me..

Please read https://github.com/mozilla/pdf.js/blob/master/CONTRIBUTING.md and provide the mentioned information to reopen this issue. Thanks.


Add an ASE Relax job for ONETEP

Summary of Changes
Implemented an ase relax job and associated test for the ONETEP calculator.
Requirements
Checklist

[x] I have read the Contributing Guide. Don't lie! 😉
[x] My PR is on a custom branch and is not named main.
[x] I have used black, isort, and ruff as described in the style guide.
[x] I have added relevant, comprehensive unit tests.

Notes

Your PR will likely not be merged without proper and thorough tests.
If you are an external contributor, you will see a comment from @buildbot-princeton. This is solely for the maintainers.
When your code is ready for review, ping one of the active maintainers.

Can one of the admins verify this patch?

In theory you are right. Practically we have seen that in some cases external optimisers do work better, it also allows the user to have flexibility of switching optimisers much more easily

Thanks for the contribution, @Nekkrad! Looks like the tests are likely failing because of https://gitlab.com/ase/ase/-/issues/1411. Any idea how to fix that issue upstream? If not, I'll have to check it out myself or implement a workaround here.
Otherwise, this PR is good to go!

Feel free to merge!

Looks like the failing test was unrelated to the aforementioned ASE issue. I fixed it and will merge momentarily!


Detections and hunting queries- Aug 2020

Fixes #
Proposed Changes

Hunting query for Suspicious Login outside normal hours
Detection for Expensive Computes based on Azure Activity logs

Made the changes per PR review feedback


[stable/prometheus-blackbox-exporter] Fix upstream URL

What this PR does / why we need it:
Fixes broken URL in README and install command

/assign @unguiculus

Please bump the chart version.

Sure, though I'm surprised to do that for a README-only change.

Is this a duplicate of https://github.com/kubernetes/charts/pull/6399?

Yeah looks like the README changes are already part of that one. Feel free to close this if that one will get merged.

Closed in favor of #6399.


adaptor.js should set classVarName for Api

classVarName on an api object is always "default". At least the python template expects it to be a tag name. It may also need casing consideration.

Thanks again!

v1.4.1 includes the setting of classVarName per 'api' / tag. Still outstanding is the possible case options you mentioned.


New top bar in workspace view

New design of a top bar which will be visible for users on workspace view. It contains (from the right):

Icons which allows user to go to the welcome screen, dashboard and currently open project (active on workspace)
Project name and node breadcrumbs
Help chat and user profile / settings related stuff

1 & 3 have to be implemented in react and 2nd is already done in Rust. To meet these requirements we (react team) need to render a fixed div and tell rust width of left and right box (so you can move your parts X pixels to the right).
This task assumes:

Remove current back to dashboard icon.
Allow us to inject the 1&3 boxes width during the EnsoApp constructing next to e.g loader / engine / startup configuration options.
Ensure that events are being properly propagated between the rust / react layers

@somebody1234 anything you would like to add?

i can't think of anything missing right now, this should be enough


Are there similar plugin that support worklet?

If you use webpack v5, you can use this https://github.com/webpack/webpack/issues/11543#issuecomment-745529334, no need plugins or loaders


Smaller geos missing in several cities

I've found this in Calgary, Toronto, and more.

Can you check if these are missing from OpenStreetMap as well?

Can you check if these are missing from OpenStreetMap as well?

@dtreich-itdp


can't create-release from existing dev-release .yml

Did this in the bosh-repo after creating a new dev release with bundle exec rake release:create_dev_release
$ bosh create-release /Users/<user>/workspace/bosh/release/dev_releases/bosh/bosh-257.9+dev.3.yml --tarball
Internal inconsistency: version 'b26048fd1490b32e64a74c8014df6cfa27300d68' vs '3'

Exit code 1

yeah. currently we raise an error explicitly since we dont expect to see numeric versions (fairly old). planning to remove this error.

Fixed recently.

For those who like details... https://github.com/cloudfoundry/bosh-cli/commit/57771c2fb964071c816587d4885498770b5ec52d

Thanks for the details, @dpb587-pivotal :)


Ambiguity in NIST-UNIFAC subgroup 309 allocation

Thanks for such a great project, Caleb.
Not a problem with the software as such, but the NIST-UNIFAC (2014) method). Whilst looking at fragmentation method in the NIST paper, I noticed that two components have been allocated subgroup number 309: acetal (CH2(O)2) in main-group 74; and oxime (CH=NOH) in main-group 82. These two components have different van der Waals surface area and volume and the main-groups have different binary interaction parameters; however it will always be interpreted as oxime because it is referred to later in the NIST-UNIFAC parameter allocation in the thermo module.
Suggest that maybe a warning is issued when using NIST-UNIFAC subgroups as a group contribution method? As I said, not an issue in the software, but ambiguity in the method.
UPDATE: NIST-Unifac parameters defined in the SI of https://www.sciencedirect.com/science/article/pii/S0378381214007353?ref=pdf_download&fr=RR-2&rr=8e508110f8779553

Hi,
You are dead on correct. This issue wasn't caught by any of the reviews of the NIST UNIFAC method, although many mistakes were. Sure enough their error is in thermo:
NISTUFSG[309] = UNIFAC_subgroup(309, 'CH2(O)2', 74, 'Acetal', 0.9347, 0.708)
NISTUFSG[309] = UNIFAC_subgroup(309, 'CH=NOH', 82, 'Oxime', 1.499, 1.46)
The second one, Oxime, overwrites the first. There is no way to use the acetal group. A different ID would need to be assigned to one of the groups, probably the acetal one. Someone could email the authors of the paper and ask for a correction and a correct ID. Without that, I think it makes sense to create a new ID for acetal that will be sure not to conflict with one in an updated NIST paper - maybe 10309 or 10000.
Would you like to prepare a PR? I believe just changing the ID and making a small note of it in the "Data for NIST UNIFAC (2015)" section of the documentation would be sufficient.
Sincerely,
Caleb


Adding tools.get_makeblastdb helper function

Will need this for #13

I'm ambivalent about recording makeblastdb version/parameters in the database. I don't think it hurts but - as discussed - it adds a wee bit of complexity (maybe an additional field for "other programs" might be useful, populated with a JSON entry?). We might revisit that.


hitting "Create" with a blank category field returns a proxy error

w/in an activity, in the category list, hit the Create button without entering anything in the field.

Same for subcategory btw


Schedule is showing as all free periods

<EMAIL_ADDRESS>

Bug is now fixed. Turns out it only affected students with free periods in both D and F period, meaning that it affected two people.


Bump Go and dependencies

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 94.932%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
562

Relevant Lines:
592

💛  - Coveralls


Allow Free Roaming observers

When "checkpoint_respawn" is true, it may be really boring to be limited to watch a player, instead allow for a third spectate mode for roaming around the map

Where does it switch the spectator mode? I think that is missing.


For Profile-collection and Actual Equipment APs , the response keeps changing.

When we try the below API's , the response is varying everytime the API is tried
/core-model-1-4:network-control-domain=cache/control-construct={mountName}/equipment={uuid}/actual-equipment
/core-model-1-4:network-control-domain=cache/control-construct={mountName}/profile-collection
Responses observed :

500 Internal server error
Response with details of forwarding-domain
Control construct
ltp-augment-1-0:ltp-augment-pac

curl -X GET "http://xxx:xxx/core-model-1-4:network-control-domain=cache/control-construct=xxx/profile-collection" -H "accept: application/json" -H "user: User Name" -H "originator: Resolver" -H "x-correlator: 550e8400-e29b-11d4-a716-446655440000" -H "trace-indicator: 1.3.1" -H "customer-journey: Unknown value" -H "operation-key: Operation key not yet provided."

curl -X GET "http://XXX:XX/core-model-1-4:network-control-domain=cache/control-construct=XXXX/equipment=CTRL%20IduBoard%20Xpic%2032E1/actual-equipment" -H "accept: application/json" -H "user: User Name" -H "originator: Resolver" -H "x-correlator: 550e8400-e29b-11d4-a716-446655440000" -H "trace-indicator: 1.3.1" -H "customer-journey: Unknown value" -H "operation-key: Operation key not yet provided."

 Issue is no longer observed.
Proper response observed for both actual equipment and Profile-equipment APIs
200OK


Identify stdlib dep and update dep functions

Prior to this commit pecdm did not state stdlib as a dependency, this commit fixes that and updates the deprecated stdlib to_json_pretty and merge functions to their named space equivalents.
Also added Puppet 8 Support and updated the PDK template

Maybe it will be also good to bump peadm in this PR?

Maybe it will be also good to bump peadm in this PR?

We can probably do it in a separate one, just to divide and conquer.
I do have some worries about lack of testing here... Anyone has any ideas on how to properly unit/acceptance test this?

The existing tests should cover this because in theory no functionality has changed.

@davidsandilands looks good


Chapter not found 🔞 لا يوجد فصول(ليس في الكل)

Source information
هنتاي العرب 1.4.2
Source language
العربية
Steps to reproduce
1- Open Add (Arab Hentai)
2-Open a manga or manhwa
3-The chapters do not appear for me and only the oneshot appears
Expected behavior
It should show me all chapters without a problem

Mihon/Tachiyomi version
Beta r6768
Android version
12
Other details
لا يحدث هذا مع كل المانجا و المانهوا
و ايضا لا تظهر الفصول الجديدة بعد اضافتها
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] I have updated the app to version 0.15.3.
[X] I have updated all installed extensions.
[X] I have tried the troubleshooting guide.
[X] If this is an issue with the app itself, I should be opening an issue in the app repository.
[x] I will fill out all of the requested information in this form.

@Smol-Ame Sorry, I didn't mean anything bad
!!Done

@Smol-Ame When will this problem be solved?

@Smol-Ame When will this problem be solved?


proposal: chore(package.json): upgrade typescript to 2.0

BREAKING CHANGE

This does not cause the breaking change directly, but after merging this change, our codes will get a potentially breaking changes for all change.

As similar to what we did in module augmentation in 1.8, I'd like to defer to get public release. There are number of dependencies (like ng2), better to align typescript compilers as well.

@kwonoj okay. I'll close this at this moment.


feat: display author images

Changes

fixes #174
displays author image if one is provided, otherwise display a default D_D logo
to prove the concept, this includes raza and my avatar images. remaining author images need to be uploaded in the authors directory and follow the patterns within this PR.

@elPiablo i just pulled raza's image by saving his twitter avatar. would suggest that as mvp route. also for mvp, i do think any lesson with multiple authors is well-served by the default D_D logo. even if someone deserves the primary author credit, its a bit smelly UX if only one author image appears. ill see where your instincts take you though.


installing sharp on electron-react-boilerplate

Possible install-time or require-time problem

[x] I have read the documentation relating to installation.
[x] I have ensured that the architecture and platform of Node.js used for npm install is the same as the architecture and platform of Node.js used at runtime.

Are you using the latest version of sharp?

[x] I am using the latest version of sharp as reported by npm view sharp dist-tags.latest.

If you cannot confirm this, please upgrade to the latest version and try again before opening an issue.
If you are using another package which depends on a version of sharp that is not the latest, please open an issue against that package instead.
Is this a problem with filesystem permissions?
If you are using npm v6 or earlier and installing as a root or sudo user, have you tried with the npm install --unsafe-perm flag?
If you are using npm v7 or later, does the user running npm install own the directory it is run in?
If you are using the ignore-scripts feature of npm, have you tried with the npm install --ignore-scripts=false flag?
no its not an filesystem issue
What is the complete output of running npm install --verbose sharp?

✗ npm install --verbose sharp
npm verb cli [
npm verb cli   '/usr/local/bin/node',
npm verb cli   '/usr/local/bin/npm',
npm verb cli   'install',
npm verb cli   '--verbose',
npm verb cli   'sharp'
npm verb cli ]
npm info using<EMAIL_ADDRESS>npm info using<EMAIL_ADDRESS>npm timing npm:load:whichnode Completed in 0ms
npm timing config:load:defaults Completed in 2ms
npm timing config:load:file:/usr/local/lib/node_modules/npm/npmrc Completed in 0ms
npm timing config:load:builtin Completed in 0ms
npm timing config:load:cli Completed in 2ms
npm timing config:load:env Completed in 0ms
npm timing config:load:file:/Users/user/app/release/app/.npmrc Completed in 0ms
npm timing config:load:project Completed in 1ms
npm timing config:load:file:/Users/user/app/release/.npmrc Completed in 2ms
npm timing config:load:user Completed in 2ms
npm timing config:load:file:/usr/local/etc/npmrc Completed in 0ms
npm timing config:load:global Completed in 0ms
npm timing config:load:validate Completed in 0ms
npm timing config:load:credentials Completed in 1ms
npm timing config:load:setEnvs Completed in 1ms
npm timing config:load Completed in 9ms
npm timing npm:load:configload Completed in 9ms
npm timing npm:load:setTitle Completed in 19ms
npm timing npm:load:setupLog Completed in 1ms
npm timing config:load:flatten Completed in 2ms
npm timing npm:load:cleanupLog Completed in 2ms
npm timing npm:load:configScope Completed in 0ms
npm timing npm:load:projectScope Completed in 1ms
npm timing npm:load Completed in 40ms
npm timing arborist:ctor Completed in 0ms
npm timing idealTree:init Completed in 46ms
npm timing idealTree:userRequests Completed in 1ms
npm timing arborist:ctor Completed in 1ms
npm http fetch GET 200 https://registry.npmjs.org/sharp 37ms (cache hit)
npm timing idealTree:#root Completed in 46ms
npm timing idealTree:node_modules/sharp Completed in 1ms
npm timing idealTree:buildDeps Completed in 47ms
npm timing idealTree:fixDepFlags Completed in 1ms
npm timing idealTree Completed in 97ms
npm timing reify:loadTrees Completed in 97ms
npm timing reify:diffTrees Completed in 3ms
npm timing reify:retireShallow Completed in 1ms
npm timing reify:createSparse Completed in 0ms
npm timing reify:loadBundles Completed in 0ms
npm timing reify:unpack Completed in 1ms
npm timing reify:unretire Completed in 1ms
npm timing build:queue Completed in 0ms
npm timing build:deps Completed in 1ms
npm timing build Completed in 1ms
npm timing reify:build Completed in 1ms
npm timing reify:trash Completed in 0ms
npm timing reify:save Completed in 14ms
npm http fetch POST 200 https://registry.npmjs.org/-/npm/v1/security/advisories/bulk 1440ms
npm timing auditReport:getReport Completed in 1442ms
npm timing auditReport:init Completed in 0ms
npm timing reify:audit Completed in 1443ms
npm timing reify Completed in 1557ms
up to date, audited 68 packages in 2s
8 packages are looking for funding
run npm fund for details
found 0 vulnerabilities
npm timing command:install Completed in 1565ms
npm verb exit 0
npm timing npm Completed in 1784ms
npm info ok

What is the output of running npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp?
System:
OS: macOS 11.6
CPU: (8) x64 Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
Memory: 460.45 MB / 16.00 GB
Shell: 5.8 - /bin/zsh
Binaries:
Node: 16.13.0 - /usr/local/bin/node
Yarn: 1.22.4 - /usr/local/bin/yarn
npm: 8.1.0 - /usr/local/bin/npm
npmPackages:
sharp: ^0.30.1 => 0.30.1
I have installed sharp on https://github.com/electron-react-boilerplate/electron-react-boilerplate
but when using it I get
(0 , sharp_1.default) is not a function
I think its tscoing.ts issue or webpack

Please see https://sharp.pixelplumbing.com/install#webpack and https://github.com/lovell/sharp/issues/2909

I tried both solutions ..
sharp is not a function

I found out its working but only on main.ts


Node.js payload

@piotr-rojek tried the new version and no longer getting the timeout issue described in #9 🥳
However a new issue cropped up. We're sending messages with a JSON payload with nodejs and receiving them with nodejs via the Azure SDK.
Example code:
const message: ServiceBusMessage = {
        body: { questionId: "fbf1ccfe-c544-4125-9e42-2e09b0d26a64", answerId: "51abac65-e201-402b-8b5a-09186a0fbdea", value: "", userId: "290c1ead-c95b-4559-969e-851be85689a6", version: "1.0", createdOn: "2023-04-19T13:20:27.718Z" }
        partitionKey:'questionnaire-V1'
      }

sender.sendMessages(message)

const serviceBusTrigger: AzureFunction = async function (
  context: Context,
  // eslint-disable-next-line @typescript-eslint/no-explicit-any
  req: any
): Promise<void> {
   console.log(req)
}

However, using the emulator the body is in a different format, Sp�@@@Sr�+�x-opt-partition-key�questionnaire-V1SsESu��{"questionId":"fbf1ccfe-c544-4125-9e42-2e09b0d26a64","answerId":"51abac65-e201-402b-8b5a-09186a0fbdea","value":"","userId":"290c1ead-c95b-4559-969e-851be85689a6","version":"1.0","createdOn":"2023-04-19T13:20:27.718Z"} as an example.
Seems to be the exact problem described in https://www.bfcamara.com/post/84113031238/send-a-message-to-an-azure-service-bus-queue-with

There is in fact some weirdness with the content (using node azure-sdk)
example:

the message being published is

(...)

const msgBody =  [{
        "SuspiciousNumber":"87777788870009",
        "TestingEptyName":"",
        "StatusId":5,
        "IsProcessing":true,
        "StartingDate":null,
        "ExpiryDate":null,
        "LastModifiedDate":"2023-03-13T21:28:04.8387347Z"
      }]

const msg = {
        body: msgBody,
        subject: 'test pci processing',
        contentType: 'application/json',
      }

      await sender.sendMessages([msg])

what lands in rabbitmq is:

payload:
AFNw0AAAAAcAAAADQEBAAFNy0QAAAAQAAAAAAFNz0AAAADAAAAAHQEBAoRN0ZXN0IHBjaSBwcm9jZXNzaW5nQECjEGFwcGxpY2F0aW9uL2pzb24AU3WgtVt7
IlN1c3BpY2lvdXNOdW1iZXIiOiI4Nzc3Nzc4ODg3MDAwOSIsIlRlc3RpbmdFcHR5TmFtZSI6IiIsIlN0YXR1c0lkIjo1LCJJc1Byb2Nlc3NpbmciOnRydWUs
IlN0YXJ0aW5nRGF0ZSI6bnVsbCwiRXhwaXJ5RGF0ZSI6bnVsbCwiTGFzdE1vZGlmaWVkRGF0ZSI6IjIwMjMtMDMtMTNUMjE6Mjg6MDQuODM4NzM0N1oifV0=

decoded payload:
echo "AFNw0AAAAAcAAAADQEBAAFNy0QAAAAQAAAAAAFNz0AAAADAAAAAHQEBAoRN0ZXN0IHBjaSBwcm9jZXNzaW5nQECjEGFwcGxpY2F0aW9uL2pzb24AU3WgtVt7
> IlN1c3BpY2lvdXNOdW1iZXIiOiI4Nzc3Nzc4ODg3MDAwOSIsIlRlc3RpbmdFcHR5TmFtZSI6IiIsIlN0YXR1c0lkIjo1LCJJc1Byb2Nlc3NpbmciOnRydWUs
> IlN0YXJ0aW5nRGF0ZSI6bnVsbCwiRXhwaXJ5RGF0ZSI6bnVsbCwiTGFzdE1vZGlmaWVkRGF0ZSI6IjIwMjMtMDMtMTNUMjE6Mjg6MDQuODM4NzM0N1oifV0=" | base64 --decode
Sp▒@@@Sr▒Ss▒0@@@▒test pci processing@@▒application/jsonSu▒▒[{"SuspiciousNumber":"87777788870009","TestingEptyName":"","StatusId":5,"IsProcessing":true,"StartingDate":null,"ExpiryDate":null,"LastModifiedDate":"2023-03-13T21:28:04.8387347Z"}]

i am guessing it's a problem with this method:
https://github.com/piotr-rojek/devopsifyme-sbemulator/blob/5d7f338b880ac3940a3759157dfe1205bac16d12/src/ServiceBusEmulator.RabbitMq/RabbitMqMapper.cs#L69
but i might be wrong.
the thing is: rabbit already has the extra content at the beginning of the payload
and it looks like some "encoded" version of the subject and contentType  and body keys
if i were to take a guess:

Sp▒@@@Sr▒Ss▒0@@@▒ - subject
@@▒ - contentType
Su▒▒ - body

but that is pure speculation. maybe you could try to run the sbemu-node docker projects and POST to the /publish endpoint  and perhaps set a trap and inspect the raw payload coming in from azure-sdk - perhaps it's in a different format?
cheers
r.

Hey guys,
The problem seems to be related to the message batching not being supported by the emulator atm. When batching is enabled, multiple messages are serialized and wrapped by an outer message. Emulator recognizes only the outer message but does not understand its inner body containing the actual message, and forwards as-is to rabbitmq.
A quick workaround is to send a single message, not an array, by changing
await sender.sendMessages([msg])

to
await sender.sendMessages(msg)

Note that the example given by @confuser seems to already pass a single message, but since it is typed, might be following a "batched" code path on nodejs sdk side. On the other hand the code snippet given by @rafak works fine after removing [].
Going forward, I will try to find a way to handle this in the emulator. I just need to figure out how to access a MessageFormat field, which indicates that we are dealing with a batch. TBC
/Piotr

ok i will try to work with single messages, thanks

i can confirm that single messages work and all is good,
i found some info in ms docs, but you probably already got there:
i will put in the links in case they become somehow useful.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messages-payloads#payload-serialization
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Azure.Messaging.ServiceBus/samples/Sample01_SendReceive.md#send-and-receive-a-batch-of-messages


Demo server out of memory

The scratchpad demo server isn't working anymore:

Service Unavailable Not enough resources: no_memory In: [13] thread_create(wrap_spawned((0x7f5c4c86ea90),...),69,[...]) [12] thread_httpd:http_spawn(http_dispatch: ...,[]) at /usr/lib/swi-prolog/library/http/thread_httpd.pl:796 [8]

Thanks.  Restarted


[Feature]: Allow configuring keep-alive settings in the agent as gRPC client

Requirement
From: https://cloud-native.slack.com/archives/CGG7NFUJ3/p1659627880482739
question on using the jaeger-agent in combination with a collector that is hosted/run and load-balanced in k8s: I found the various jaeger-agent configuration settings for load-balanced collector endpoints, but from what I read (and from experiments), the endpoints that the agent chooses are selected at start and do not changed afterwards.
In our scenario, we use the jaeger-agent as k8s daemonset (so, 1 instance per k8s worker node) that sends data to a collector deployed as regular k8s daemonset which is auto-scaled to handle the volume increasing/decreasing over time. In our specific scenario we are using the opentelemetry collector, but not sure that changes conceptually what I'm asking here.
The issue that we see is that since the jaeger-agents are very long-lived services (the worker nodes rarely come or go), choosing the collector endpoint/IP using the collector service hostname means that if the collector scales up, the scaled-up instances will not be used in practice. This is because of the long-lived TCP connections between the jaeger-agent and the chosen collector, whose IP(s) are resolved on start and used essentially forever. Thus, only if the jaeger-agent were to restart would we ever see traffic to a collector instance started (scaled up) after the agent had started.
As a result, even if the collector scales up and new endpoint IPs are now available behind the k8s service, only the old collector IPs continue to receive traffic and the scale-up had no effect.
Side-node: of course, if a collector instance is removed, it would cause a re-balance as the IP becomes unreachable and the jaeger-agent needs to re-select the upstream IPs to talk to. But on collector scale-up, the old IPs remain valid.
FYI, could be interesting for others to know: the idea of a purely load-balanced solution is not that easy in k8s, as in most situations the pod IPs of the collector will be abstracted away behind a k8s service. Thus, the jaeger-agent will only see one IP, and periodically forcing a DNS-re-resolution doesn't yield a change (we'll simply lookup the very same IP again).
However, the gRPC server can help here: for example, in golang grpc (grpc-go ), which is used by most tools including jaeger-agent and opentelemetry-collector, allows enforcing a maximum connection age. This can be configured for idle connections but also for any connection:
keepalive:
  server_parameters:
    max_connection_age: "<max-age>s"
    max_connection_age_grace: "<a few>s"

It's not overly clean, as the client is told to disconnect/reconnect periodically, even if there is no need (i.e., even if no new IPs have been added to the service). But, it's an effective solution to the problem if periodic reconnects are acceptable. The newly established connection will use a new TCP source port, and the k8s service loadbalancer will choose a new pod IP (at random). Thus, over time, the clients balance out their connections to the available collector pods
Problem
We already support configuring keepalive on the server (e.g. --collector.grpc-server.max-connection-age), but it would be good to support the same in the agent as the client. If server closes the connection it manifests as an error to the client.
Proposal
There doesn't seem to be a client-side gRPC option to control keep-alive connection age, but maybe it is possible to close the connections manually.
Open questions
No response

Hi, can I take this?

Yes

@yurishkuro I   have gone through the slack discussion and client-side there is no Connection timeout support. Could you eloborate on the comment  "may be possible to close the connections manually "  .

Could you elaborate on the comment "may be possible to close the connections manually".

We could have the logic in the agent that would try to reconnect to the server(s) every N seconds (N being configurable, e.g. 30sec by default). I don't know how exactly to do that w/ gRPC, maybe via custom load balancer, or by forcing a redial.

@yurishkuro I am interested in working on it, can guide me from where I can start?

Hi! Is this worked on? may I pick this up?

Please assign me.

You're right, we can close it, not worth the investment at this point.


[Feature]: Make the library embedabble

Description about the feature
Hello,
I just came across Xline when on the hunt for a KV store that I can embed in my own Rust binaries. I don't really want to run a container/node for a persistent KV store, but I want to be able to share state between multiple instances of my own binaries.
Is this something you'd consider adding? Essentially, like RocksDB et al but the ability to throw it run in a tokio runtime alongside my own code and interact with it just like the existing client does, but under the hood it's just calling methods instead of communicating with a remote node.
SurrelDB is embeddedable as a library, but the license and corporate interest puts me off. However, it relies on database backends for persistence (e.g. TiKV). I'd rather it just handle the persistence in RocksDB and do the distribution itself but this about as close as I've seen:
https://crates.io/crates/surrealdb
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi, @AshleySchaeffer.  From your description, it seems like what you need is a local KV database. Xline adopts CURP as the consensus protocol, with a focus on applications in WAN with high latency. We may not consider adding such a feature in the short term.

Hey @Phoenix500526. While I totally appreciate that you may not wish to add the feature, I just wanted to clarify what my original request meant.
Today, if you want to build a distributed service and have shared state amongst all instances of it, you might choose to write a REST API using Rust's Axum framework and couple it with a database of some kind (e.g. xline-kv, mongodb, postgres, etc.) which will maintain the state. All the instances of the Axum service you have written will communicate across the network with the database to get and update state(s) from the database, and you pay the cost of those network communications. In addition, now you have two components to manage operationally; your Rust Axum service and the database. They must be monitored, scaled, managed, and maintained separately. In most modern day cases, you'll do all that within a container orchestration framework that may run these systems on same nodes (hosts or servers) or on multiple nodes.
SurrealDB takes a slightly different approach. If you used that in the system described above, instead of having two systems you would have one. The REST API would be hosted in the SurrealDB itself (they provide a way to load code into an instance) and your code would interact with it locally (I'm not sure how this works exactly, I'd guess via communicating with a localhost interface but I could be wrong). The latency costs of these communications is drastically reduced over the system above and SurrealDB is still doing all the things it would have been doing before (e.g. syncing state via consensus amongst its nodes) it is just faster to communicate with it.
The downsides of using SurrealDB's approach is that you are limited to what libraries and frameworks you can use. It appears you may also be restricted WASM (could be wrong). This could be a complete deal breaker for many.
What I propose is that instead of running an app in the database ecosystem, the database runs in the apps ecosystem. To stick with my example above; if the database could be executed in the same context as the Axum based REST API (i.e. within one or more tokio/OS threads but in the same process), communicating with it would be as cheap as dispatching messages via channels (extremely low latency) or potentially direct memory access. You could compile the entire stack (the REST API and the database) into the same binary and only one system to manage. The database would scale up and down with your app. You'd be able to write code as you do today and manage the database from a code-level like you manage database connections or any other in process state today. The database library would simply spawn some threads that it executes on, and do what it does today with regards to the consensus process.
Does that make sense? I believe it aligns with the goals of xline (high performance in high latency scenarios) but again, I completely understand if you'd prefer to focus elsewhere or disagree completely :smile:
If you disagree, feel free to close the issue. No hard feelings!


URL parameters in v4 express.Router()

This isn't working for me:
app.js
var router = require('router');
app.use('/user/:id', router);

router.js
var router = require('express').Router();
router.get('/user/:id', function(req, res) {
    console.log(req.params.id);
});

curl http://localhost:3000/user/123
Cannot GET /user/123

@wulfgarpro Not sure what exactly is an error or whatever but why would you want to mount the router to the /user/:id endpoint? It seems to already handle that on its own.
Also, not sure what are you requireing by require('router') - should probably be a require('./router') as long as they are in a single folder?
Would the following changes to app.js work?
app.js
var router = require('./router');
app.use(router);

@VasilioRuzanni
Sorry, my example was contrived, my code is actually:
var user = require('./routes/user');
So no, the problem has nothing to do with the require clause.
I've split my routes up into different route files and I'm using a router implementation for each.  It's working fine for any route without a parameter.

@wulfgarpro
In your code above you are doing it like this (combined into a single script for simplicity):
var router = require('express').Router();
router.get('/user/:id', function(req, res) {
  console.log(req.params.id);
});

app.use('/user/:id', router);

With that, you're effectively telling the app that it should expose the /user/:id/user/:id endpoint (not sure how Express handles duplicate params but I'm pretty sure this is isn't something you want).
Try mounting your router without specifying path, like this:
app.use(router);

@VasilioRuzanni, thanks.
I'm not sure where I got the example showing my method above, but you're correct.


Multiple Model or ViewModel

Hello, Your m2vm project is awesome! Thank you very much.
I have a question. What if I have multiple model or viewmodel?
How can I manage the view that has more than one model or viewmodel?

Thanks, glad you like it!
Multiple viewModels are not managed right now, however you can create a "facade" viewModel that delegates to other viewModels. Every fragment/activity uses a single viewModel and save/restore it in a retained fragment.
Another easy solution is to create multiple fragments, every fragment has his own viewModel.
Can you tell me a use case where you need multiple viewModels? Is it just for a better code organization or is there another reason?
In next days I'll document a way to manage recycler view, maybe is this the reason?

Thank you very much for your very fast response. and yes. it is just for a better code organization right now. I will try follow your suggestion. Thanks!.
How about multiple model? (not viewmodel). Should I put into HashMap? (just for a better code organization as well).
Thank you again and hope you keep update this project. this is useful for everyone. I  have seen your video presentation in youtube it will be great if there is english subtitle.

The model is automatically saved in Activity/Fragment instance state, so the simplest solution is the same I said for ViewModels: create a facade model that contains all the "submodels". Usually you don't need an HashMap, you can just use some fields in the model.


Vimeo player not working in Firefox

Error occurs when attempting to play a Vimeo video in Firefox 31.0 on OS X 10.11.2
TypeError: Argument 1 of Window.getComputedStyle is not an object.
Error: Permission denied to access property 'toString'

A quick search of the error gives several results but not much in the way of a solution:

https://github.com/vimeo/player-api/issues/53
https://vimeo.com/forums/topic:99956
https://vimeo.com/forums/topic:103865

Wow, I hadn't updated Firefox in a while.. No longer a problem in Firefox 43.0.4


Consider hosting project under pytest-dev

Have you considered hosting this plugin under @pytest-dev organization? The process can be initiated with a ticket on https://github.com/pytest-dev/meta/issues
I am asking this because I am not keen to add dependency on plugins that can easily become unmaintained. (happened multiple times in the past).
I am also a python plugin author and I did that to make the future of the plugin safer.

:+1: That's a good idea. I'm going to prepare for requirements of transferring repository to pytest-dev (there are no tox configuration or changelog file in this repository now).

@ssbarnea I prepared changelog and tox configuration (I think this plugin is ready to be transferred). Can I submit an issue of migrating this plugin to pytest-dev/meta (or should I do something else more)?

Please create a ticket, I will try to help with it. Is better if owner is proposing it, for obvious reasons.

I see.

I created a ticket https://github.com/pytest-dev/meta/issues/9. Thank you for your help!


Remove old deprecated code

I'm not sure what the exact policy we decided on for deprecating is but we have code deprecated since v0.24.0 and v.026.2, both of these seem reasonable safe to remove.

Patch 1: Remove MerkleBlock methods deprecated in 0.26.2
Patch 2: Remove SighashComponents struct deprecated in 0.24.0

Draft while I actually do it properly.

what the exact policy

AFAIK there's none but anything is good enough IMO. We do unannounced changes too. :man_shrugging: .

I'll ACK this but in the interest of getting the release out I think we'll wait til after 0.29.0 to merge

Oh yes, I expected that. I'll add 0.30 milestone and try to remember to add labels/milestones to my PRs. Thanks.


[Snyk] Security upgrade nginx from mainline-alpine to 1.25.3-alpine3.18

This PR was automatically created by Snyk using the credentials of a real user.Keeping your Docker base image up-to-date means you’ll benefit from security fixes in the latest version of your chosen image.
Changes included in this PR

Dockerfile.ui

We recommend upgrading to nginx:1.25.3-alpine3.18, as this image has only 0 known vulnerabilities. To do this, merge this pull request, then verify your application still works as expected.
Some of the most important vulnerabilities in your base image include:

Severity
Issue
Exploit Maturity

Missing Encryption of Sensitive Data SNYK-ALPINE318-CURL-6104720
No Known Exploit

Missing Encryption of Sensitive Data SNYK-ALPINE318-CURL-6104720
No Known Exploit

CVE-2023-46218 SNYK-ALPINE318-CURL-6104721
No Known Exploit

CVE-2023-46218 SNYK-ALPINE318-CURL-6104721
No Known Exploit

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
🛠 Adjust project settings

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

mainline-alpine has less vulnerabilities according to dockerhub.


Resource git actions checkout and sync abend in a Dokken environment: no /dev/tty

Description

Resource git actions checkout and sync abend in a Dokken environment: no /dev/tty
Chef Version

Chef Development Kit Version: 3.11.3
chef-client version: 14.13.11
delivery version: master (9d07501a3b347cc687c902319d23dc32dd5fa621)
berks version: 7.0.8
kitchen version: 1.25.0
inspec version: 3.9.3

Platform Version

Ubuntu 18.04
Replication Case

When Dokken testing the git resource as follows:
33:   git "Install #{new_resource.name} plugin" do
34:     destination ::File.join(root_path, 'plugins', new_resource.name)
35:     repository new_resource.git_url
36:     reference new_resource.git_ref
37:     user new_resource.user if new_resource.user
38:     action :checkout
39:   end
40: end

Client Output

I encounter an issue as described on SO.
The error is:
* git[Install julia-build plugin] action checkout

================================================================================
Error executing action `checkout` on resource 'git[Install julia-build plugin]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '128'
---- Begin output of git ls-remote "https://github.com/jlenv/jlenv-build.git" "master*" ----
STDOUT:
STDERR: fatal: could not read Username for 'https://github.com': No such device or address
---- End output of git ls-remote "https://github.com/jlenv/jlenv-build.git" "master*" ----
Ran git ls-remote "https://github.com/jlenv/jlenv-build.git" "master*" returned 128

I was able to test and confirm that in the Dokken environment:
echo ahoj > /dev/tty
bash: /dev/tty: No such device or address

It appears the issue is here?
This appears to be confirmed by the fact that the git unit specs require (stub) tty? to return true.
If this is not going to be fixed, then at least the resource should abend with an error message that a tty is required.
Stacktrace

As Above.

I believe the correct way to accomplish this is git show-ref --quiet --verify <ref> >/dev/null 2>&1, redirecting stderr and stdout to /dev/null and returning the result depending on the exit code?
Before I sink more of my time. Would you accept a pull request implementing this approach?

yeah although 2>&1 is a bash-ism and it needs to work if the shell is tcsh, and probably needs to work on windows as well.
if there's a way to pass the moral equivalent of ssh's -n argument to git that would be better rather than trying to convince mixlib-shellout to wire up /dev/null in a shel-portable-fashion and do <something that i couldn't guess> on windows, which is probably going to be a lot harder.

@lamont-granquist thanks for looking at this.
Agreed.  I've been thinking about this a little more.
The absence of a TTY is the default in many of my chef-automated use cases i.e. cloud server setup's, containers, etc.
From my perspective (and reading around), issues such as this one are circumvented by using vagrant, passing docker a --device or avoiding such stdout/stderror parsing recipes.
In my case I'm making more use of dokken, and here I can't sidestep the git resource.
Rather than piping to stdout and stderr it seems the more generic approach is to send the data to files and read it from there.
Agreed?
Is there an example of the best-practice Chef way to write/read temp files in a recipe?
In this case the process would be git show-ref --quiet --verify to get the result/return code.
If the return indicates verified then read in the output file generated.
This (no tty) seems to be an issue more widespread than the git resource?


chore: support the extended readiness check

Following https://github.com/oauth2-proxy/oauth2-proxy/pull/1839, this will add the non-breaking support for the extended healthcheck as a readinessProbe.

@kobim this is not enough, you need to bump also the chart version.

@kobim if this PR is ready please mark it "Ready for review"

@kobim could you please take a look?

@pierluigilenoci I'm still waiting to complete and merge https://github.com/oauth2-proxy/oauth2-proxy/pull/1839 before marking this as ready.

@kobim the other PR is merged, can you please follow up on this?