IterableDataset with wrong length causes validation loop to be skipped.

🐛 Bug
When IterableDataset has a wrong length defined, specifically a higher than the actual number of iterations, the validation epoch is skipped.
To Reproduce
import os

import torch
from torch.utils.data import DataLoader, IterableDataset, get_worker_info

from pytorch_lightning import LightningModule, Trainer

class RandomDataset(IterableDataset):
    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def sample_queue(self, indices):
        for index in indices:
            yield self.data[index]

    def __len__(self):
        return self.len * 2

    def __iter__(self):
        indices = list(range(len(self.data)))
        # Get worker info.
        worker_info = get_worker_info()
        if worker_info is None:
            # Only a single process, so it gets all the data.
            return self.sample_queue(indices)
        else:
            # Divide indices to workers.
            worker_indices = indices[worker_info.id::worker_info.num_workers]
            return self.sample_queue(worker_indices)

class BoringModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("train_loss", loss)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("valid_loss", loss)

    def test_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("test_loss", loss)

    def configure_optimizers(self):
        return torch.optim.SGD(self.layer.parameters(), lr=0.1)

def run():
    train_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    val_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    test_data = DataLoader(RandomDataset(32, 64), batch_size=2)

    model = BoringModel()
    trainer = Trainer(
        default_root_dir=os.getcwd(),
        max_epochs=2,
    )
    trainer.fit(model, train_dataloaders=train_data, val_dataloaders=val_data)
    trainer.test(model, dataloaders=test_data)

if __name__ == "__main__":
    run()

Expected behavior
Run validations loops even with the wrong length.
Environment
Versions

Collecting environment information...
PyTorch version: 1.10.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.33

Python version: 3.8.12 (default, Oct 12 2021, 13:49:34) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.11.0-38-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla V100-PCIE-16GB
Nvidia driver version: 470.63.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.3
[pip3] pytorch-lightning==1.4.9
[pip3] torch==1.10.0+cu113
[pip3] torchmetrics==0.6.0
[pip3] torchvision==0.11.1+cu113

@jopo666
The IterableDataset should leave the len undefined unless you actually make it so that it returns the correct value.
For this reason, Lightning prints a warning if you do define one:

/Users/adrian/repositories/pytorch-lightning/pytorch_lightning/utilities/data.py:87: UserWarning: Your IterableDataset has __len__ defined. In combination with multi-process data loading (when num_workers > 1), __len__ could be inaccurate if each worker is not configured independently to avoid having duplicate data.

Here is how we determine the number of batches in a dataloader:
https://github.com/PyTorchLightning/pytorch-lightning/blob/e4ee6df196dd0c6b17d8c695bea3f1d286114f4f/pytorch_lightning/trainer/data_loading.py#L349
If a length is defined, we take that as the truth. If not, we assume the dataloader can stop at any point. This variable trainer.num_training_batches is then later on used in the loops do determine if we have reached the last batch. If the last batch is reached, we run validation. That's where your example breaks, because these conditions are not satisfied.

Yea I think it's could be implemented differently, so that Trainer does things in "the right way" even though the user makes a mistake. For example and IterableDataset with the wrong length wouldn't miss the validation step in the case below.
for epoch in range(100):
    # Training loop.
    for batch in train_loader_with_wrong_length:
        training_step(batch)

    # Validation loop.
    for batch in val_loader:
        validation_step(batch)

To achieve the above case, Trainer should move on to the validation epoch if IterableDataset raises StopIteration, or based on the val_check_interval argument. Then the only case where the user's error in defining the __len__  would affect Trainer is if the user also passes a float to val_check_interval as then the__len__ would affect how many batches the float value corresponds to.
A few questions/comments.

Where is theIterableDataset or more generally the DataLoader.__len__ used?
As training works fine without a specific length to the DataLoader, is there a specific reason that Trainer behaves differently (except for the progress_bar etc.) for DataLoaders with length.
Is there some reason why the above warning is issued? The warning seems like something that could be raised by torch, not pytorch-lightning.

IMO, the warning should specifically say, where the IterableDataset length may cause an error if wrongly defined. Now it's just saying, "hey you might have made a mistake!" (which the user may also do with Dataset although not as easily).

@jopo666 I generally agree with your observations

Where is theIterableDataset or more generally the DataLoader.len used?

Essentially here, as the dataloader will call len() on the dataset if it has one defined:
https://github.com/PyTorchLightning/pytorch-lightning/blob/e4ee6df196dd0c6b17d8c695bea3f1d286114f4f/pytorch_lightning/trainer/data_loading.py#L349

As training works fine without a specific length to the DataLoader, is there a specific reason that Trainer behaves differently (except for the progress_bar etc.) for DataLoaders with length.

I think it is primarily for two features: 1) the progress bar will show the limits and a bar 2) the percentage based limits Trainer(limit_train_baches=0.5)

Is there some reason why the above warning is issued? The warning seems like something that could be raised by torch, not pytorch-lightning.

I added the warning to Lightning because of this comment in the PyTorch source code:
https://github.com/pytorch/pytorch/blob/2f68878a055d7f1064dded1afac05bb2cb11548f/torch/utils/data/dataloader.py#L379-L388
There is an explanation there why torch does not do anything. But we felt in Lightning we still wanted to show a useful message. Do you have any suggestions?
Related, some interesting pytorch issue: https://github.com/pytorch/pytorch/issues/40972

Can we revive this issue? Many have encountered the problem of validation loop being skipped when the actual (dynamically generated) number of batches available is smaller than the length of the batch sampler used. There should be a better way to initiate validation than just checking if the last batch is reached or using an infinite dataset.
https://github.com/Lightning-AI/lightning/blob/e7afe04ee86b64c76a5446088b3b75d9c275e5bf/src/lightning/pytorch/loops/training_epoch_loop.py#L406-L432
https://github.com/Lightning-AI/lightning/issues/2675
https://github.com/Lightning-AI/lightning/discussions/13006

+1 having this problem. In my context, I dynamically filter out a few bad samples from the dataset, while this set of filtered samples depends on the training parameters and thus varies between runs. This makes me use IterableDataset. However, the true dataset length is still very close to the original one and so I'd like to benefit from the progress bar and its time estimate.
As a workaround, I wrote a simple callback that, in the first epoch, uses an approximate number of batches provided by a data module, then counts the real number of batches. It then overwrites trainer.fit_loop.max_batches.
This doesn't change the iteration behaviour in the training loop and so is more of a decorative workaround.
from typing import Any

import lightning as pl

class BatchCounterCallback(pl.Callback):
    """
    This callback is used to set Trainer's max_batches for train, val and test loops, which fixes the progress bar when
    an IterableDataset without __len__ is used. The callback assumes that each iteration has the same number of batches
    and that only one dataloader is used for each set.

    In the first epoch it uses the datamodule's estimated number of batches for each set, which is expected to be
    an upper bound of the true number. In the following epochs it uses the number of batches counted previously.
    """

    def __init__(self):
        self.train_batch_counter = None
        self.train_true_batch_count = None
        self.val_batch_counter = None
        self.val_true_batch_count = None
        self.test_batch_counter = None
        self.test_true_batch_count = None

    def _train_set_guessed_num_batches(self, trainer: pl.Trainer):
        guessed_num_batches = self.train_true_batch_count or trainer.datamodule.get_approx_num_batches("train")
        trainer.fit_loop.max_batches = guessed_num_batches

    def _val_set_guessed_num_batches(self, trainer: pl.Trainer):
        guessed_num_batches = self.val_true_batch_count or trainer.datamodule.get_approx_num_batches("val")
        trainer.fit_loop.epoch_loop.val_loop._max_batches = [guessed_num_batches]

    def _test_set_guessed_num_batches(self, trainer: pl.Trainer):
        guessed_num_batches = self.test_true_batch_count or trainer.datamodule.get_approx_num_batches("test")
        trainer.test_loop._max_batches = [guessed_num_batches]

    def on_train_epoch_start(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
        self.train_batch_counter = 0
        self._train_set_guessed_num_batches(trainer)

    def on_validation_epoch_start(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
        if trainer.sanity_checking:
            return
        self.val_batch_counter = 0
        self._val_set_guessed_num_batches(trainer)

    def on_test_epoch_start(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
        self.test_batch_counter = 0
        self._test_set_guessed_num_batches(trainer)

    def on_train_batch_start(
            self, trainer: "pl.Trainer", pl_module: "pl.LightningModule", batch: Any, batch_idx: int
    ) -> None:
        self.train_batch_counter += 1

    def on_validation_batch_start(
            self,
            trainer: "pl.Trainer",
            pl_module: "pl.LightningModule",
            batch: Any,
            batch_idx: int,
            dataloader_idx: int = 0,
    ) -> None:
        if trainer.sanity_checking:
            return
        self.val_batch_counter += 1

    def on_test_batch_start(
            self,
            trainer: "pl.Trainer",
            pl_module: "pl.LightningModule",
            batch: Any,
            batch_idx: int,
            dataloader_idx: int = 0,
    ) -> None:
        self.test_batch_counter += 1

    def on_train_epoch_end(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
        self.train_true_batch_count = self.train_batch_counter

    def on_validation_epoch_end(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
        if trainer.sanity_checking:
            return
        self.val_true_batch_count = self.val_batch_counter

    def on_test_epoch_end(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
        self.test_true_batch_count = self.test_batch_counter


added bellmanford.java

Added BellmanFord.java. Kindly check

I'll make changes, This is my first contribution. Forgive for the mistakes.

It's absolutely fine, just make the changes as required and we'll be good to go :)

Hello @pratham-bhatnagar, we have reviewed your code, your pull request will be merged shortly.
Thank you for your patience, as well as your contribution. Have a good day :)


Update httpclient to 4.3.4.

This is to fix a bug with SSL support for certs that require SNI.

Seems fine.


Increase “Example” tab space on mobile

The Design System component "Example" preview tab needs a bit more room on mobile.
Goes hand in hand with https://github.com/coopdigital/coop-frontend/pull/157
Before

After

🚀


Release 1.5.2 on pypi but not tagged in github

I see that Oscar 1.5.2 was released on pypi, but I don't see the corresponding tag on github.
Could somebody (who knows which commit is the 1.5.2 release) tag this and push it for future reference?

Here you go: https://github.com/django-oscar/django-oscar/releases/tag/1.5.2


BackUps configuration

Set up, configure and test the backup procedures for

DB
file storages

Tasks:

[x] implement and configure all the backup mechanisms
[x] test backup monitoring mechanisms
[x] test restore mechanisms

Let's compile a list of things to backup:

[x] normal DBs,

[ ] update documentation about postgresql backups - @vsirius,

[ ] implement elasticsearch backups - @gmile,

[ ] write docs,

[ ] implement minio backups @gmile,

[ ] write docs,

[ ] implmenet prometheus backups @gmile,

[ ] write docs,

I've added description of backup generation/restoration process of elasticsearch and PG to the "E-Health Specifications" doc.
minio / prometheus backups are currently planned, hence are out of scope of this issue. I've added a respective TODOs in the doc it self.


OSX app switch to another page,user can not use xpath to lock elements any more

Our product is a OSX app.
The scene is clicking on a button on page A(This is the first page after launching our product, and I can use xpath to lock elements here), the system will jump to page B (at this time page A is not visible). My question is  I can not use xpath to lock the elements any more on page B, how to solve this problem? Thanks a lot.
I have tried to using driver.switch_to_window(driver.window_handles[x]) to solve this problem,but it can not work.

This issue is not correct, it is my mistake,sorry


DailyFilings not working

Using the DailyFilings function to get a day's filings does not work, Line 92 of daily.py has a problem with datetime.strftime.
TypeError: descriptor 'strftime' requires a 'datetime.date' object but received a 'str'
When I changed datetime.datetime.strftime to formatted_date = self._date.strftime("%y%m%d"), network_client.py gives a client-side error.
Thanks!

Thanks for reporting this bug. Can you give the full traceback?

EDGARQueryError                           Traceback (most recent call last)
~\anaconda3\lib\site-packages\secedgar\client\network_client.py in get_response(self, path, params, **kwargs)
96             try:
---> 97                 self._validate_response(response)
98             except EDGARQueryError:
~\anaconda3\lib\site-packages\secedgar\client\network_client.py in _validate_response(response)
138             else:
--> 139                 raise EDGARQueryError("The query could not be completed. "
140                                       "There was a client-side error with your "
EDGARQueryError: An error occured while making the query.
During handling of the above exception, another exception occurred:
EDGARQueryError                           Traceback (most recent call last)
 in
----> 1 today_filings.save(os.getcwd())
~\anaconda3\lib\site-packages\secedgar\filings\daily.py in save(self, directory)
191                 further by company name and form type.
192         """
--> 193         self.get_filings_dict()
194         for filings in self._filings_dict.values():
195             # take the company name from the first filing and make that the subdirectory name
~\anaconda3\lib\site-packages\secedgar\filings\daily.py in get_filings_dict(self, update_cache, **kwargs)
131         """
132         if self._filings_dict is None or update_cache:
--> 133             idx_file = self._get_master_idx_file(**kwargs)
134             # Will have CIK as keys and list of FilingEntry namedtuples as values
135             self._filings_dict = {}
~\anaconda3\lib\site-packages\secedgar\filings\daily.py in _get_master_idx_file(self, update_cache, **kwargs)
91             formatted_date = self._date.strftime("%y%m%d")
92             formatted_file_name = "master.{date}.idx".format(date=formatted_date)
---> 93             if formatted_file_name in self._get_quarterly_directory().text:
94                 master_idx_url = "{path}/master.{date}.idx".format(
95                     path=self.path, date=formatted_date)
~\anaconda3\lib\site-packages\secedgar\filings\daily.py in _get_quarterly_directory(self, update_cache, **kwargs)
70         """
71         if self._quarterly_directory is None or update_cache:
---> 72             self._quarterly_directory = self.client.get_response(self.path, self.params, **kwargs)
73         return self._quarterly_directory
74
~\anaconda3\lib\site-packages\secedgar\client\network_client.py in get_response(self, path, params, **kwargs)
100                 # Raise query error if on last retry
101                 if i == self.retry_count:
--> 102                     raise EDGARQueryError()
103         self.response = response
104         return self.response
EDGARQueryError: An error occured while making the query.

@jonzd thanks for providing the details. I won’t be able to fix this bug until the end of June, but will look into what is causing it and get it fixed as soon as possible. I appreciate your support!

@jonzd can you show the code where you create todays_filings?

@jackmoody11
today_filings = DailyFilings(date=datetime(2020,6,10))

@jonzd I had a chance to look at the code. There are a few changes I am working on. It throws an EDGARQueryError after fixing strftime because the path requires a trailing slash.
    @property
    def path(self):
        """str: Path added to client base.

        .. note::
           The trailing slash at the end of the path is important.
           Omitting will raise EDGARQueryError.
        """
        return "Archives/edgar/daily-index/{year}/QTR{num}/".format(
            year=self._date.year, num=self.quarter)

Note
The format for master.idx files changes some time in the 90's or 2000's. Will need to adapt code to take care of this. Currently causes a bug.

@jonzd this bug should be fixed now. If you can confirm that the latest version of the master branch fixes your problem that would be much appreciated! Thanks for helping improve the project. Your name is mentioned in our v0.1.7 docs.


[Mention The Feature]:  enhancing header and home page

Is your feature request related to a problem? Please describe.
the header is looking very basic according to the machine learning (ML) theme and the first image used in the background of home page is also very bright which in turn is disturbing the user's focus on the main content written over it.
Describe the solution you'd like.

i want to make some CSS changes in the header...adjust its spacing and indentation from both left and right sides.
wanted to add enhanced hover styling instead of button kind of style in the options.
little bit reduction in the opacity of this background image will bring wonders to your website....so that the white coloured text will also be more readable👍👍
also adjusting the same text with different styling will be the cherry on the cake.

Describe alternatives you've considered.
No response
Additional context.
i am Gssoc-ext contributor and hacktoberfest registered👍
i will be more than happy if you assign me this task @vivekvardhan2810
pls assign this to me with gssoc-ext and hacktoberfest-accepted and required labels.
Show us the magic with screenshots

Checklist

[X] I have checked the Existing Issues
[X] I have read the Contributing Guidelines
[X] I want to work on this issue. (optional)

Hi, I have seen many navbar and header based issues, and i have mentioned it before for contributors to not create issues regarding navbar and there should be no more navbar issues, so i have closed this issue, if you have any queries, you can join discord server for further updates and communication.
we are not looking for navbar and header updates for now, as it was done and changes many times before.
please understand.

ok


Bot on Teams is No Longer Responding in Correct Order

Describe the bug
We have an MS Teams bot that uses Bot Framework composer. It was working properly until March 1st, when it started to encounter issues. It has a flow that used to work which would route users to different types of responses and return SQL query results. It also has a flow that connects to various API-based chatbots. It is built on a resource that once used QNA maker and LUIS even though we are using neither in the current bot. It looks like QNA maker was deprecated so maybe this is part of the issue?
In short, the main issue we are seeing is that suddenly users cannot advance along the chain of dialogs despite them being able to in the past. Could this have to do with autoscaling? How can we even diagnose what is going wrong here?
Version
We are using V2 or Bot Framework Composer. We are using Dotnet 3.1.
Browser
I don't think this is relevant since we're using this on MS Teams.

[ ] Electron distribution
[x] Chrome
[ ] Safari
[ ] Firefox
[ ] Edge

OS
We are using Bot Framework Composer on a mac using Monterrey 12.6.3. The bot itself is hosted on Azure on Windows.

[x] macOS
[ ] Windows
[ ] Ubuntu

To Reproduce
Steps to reproduce the behavior:
Enter in text in Teams and send it to the bot, or click one of the multiple-choice options. The appropriate flow will not occur. For instance, a user clicks on "question", they previously were able to enter in a free response question and get an answer but currently, it will shoot the user to "internal data". Then when the user enters in a primary key to receive data, it will randomly teleport them somewhere else.
When testing in the Bot Framework Composer and Emulator, it works fine. When it goes to Teams, we encounter issues.
Expected behavior
We expect for the user to be able to advance along the flow as they once did. Unfortunately out of the blue, there are issues.
Screenshots
What we used to have that worked:

What we have now that does not work:

Additional context
We tried redeploying the bot several times and even created new resources to deploy it with. No luck :(

Hi @CharlieEvert,

Could this have to do with autoscaling?

It's certainly possible. Did you enable cloud state storage before scaling out your bot?
Once you deploy and are communicating with the hosted bot in Teams, the bot instance that you initially start the conversation flow with (responding with Question) may not be the same instance that handles the next request (responding with Wolfram|Alpha). Since the bot state is stored in the local memory of each instance, each instance cannot understand a conversation started in a different instance. The new instance then interprets it as a new conversation and restarts the flow.
Even if you're still using memory storage, when you test locally everything will work fine since there is only one instance.

Hi @anishprasad01!
Thank you for your response! I do not think we enabled cloud state storage- would this be through Cosmos DB? In the setup JSON we have it left blank as a resource.
It's a pretty weird issue we're having since everything worked fine for a couple of months and then suddenly stopped, even though we made no updates. For one user we did some Regex on their name in teams to get their first name and their name in teams was different from everyone else in the org (so the MS Bot Framework Composer regex errored out) so they cannot advance through any dialog even when we make an update. Is there a way we could reset all of these instances or have users delete chats with bots to reset them?

You could try uninstalling the Teams App and reinstalling it. Teams works a little differently compared to the other channels.
Did you have scaling out enabled for those months when everything worked fine? How was the load on the bot during that time?

We made a whole other set of bot resources and we're still having the issue, unfortunately. It does not seem to be tied to auto-scaling since we only had 2 people use the new bot and it has the same issue. We had a 4-instance limit for horizontal scaling, and we were running Standard S1 vertical scaling for that time frame. We increased both and are still having this issue. Here is the load on the bot over the past 90 days, it looks like it may have spiked with a large number of events in early March.

We figured out that we actually do have cosmos DB as a resource with our bot. We tried to add blob storage as a rescource but we got an error relating to our org's policy. Could this be it, the blob storage component?

Blob storage isn't required if you are already implementing ComsosDB for state storage.
Could you try one more thing? Does the issue persist if you limit the bot to just one instance?

It works when we limited the bot to one instance! Thanks Anish! Do you think we should keep it at one instance?


[BUGFIX] Event type incorrect when loading projection streams

This is due to the fact, that the summary field in the response
is no longer the event type.
This test uses a fixture which is returned from an event store
v3.0.3.
Resolves #13

Fixed. Magic! If this Scrutiblah thing would print out better errors I would've fixed it earlier :)


Documentation tidyup

Minor tweaks and updates from a pass through the documentation. There is one TODO in the introduction that needs some discussion (and a few more things also). cc @mihaidma

Coverage remained the same at 96.159% when pulling 6a1805c2bc3a1d393d67193f1b81a868038a6045 on docs into f5651e03a6657d8a62b1631cc8e605e92a614e3d on master.

Thanks for the feedback @mihaidma, have made modifications - let's discuss the TODO, would be good if you could simplify for sure.

Coverage remained the same at 96.159% when pulling b02c79b9a517de4e6fd2dca69ae13597782ce0ce on docs into fae9d7668380012b62a088d56a9063aec0cac8fe on master.


Fails of recipe_filler tests with install in development mode of esmvalcore

fails - I shall fix :christmas_tree:

back from Xmas shtuff, on this now :beer:


feat(perception): enable to specify region filter.

Types of PR

[x] New Features
[ ] Upgrade of existing features
[ ] Bugfix

Description
This PR enables to specify region based on x-y coordinate from ego, and you can still use distance filter.
Interface roles

You can only specify Distance or Region
Specification role is that lower_limit and upper_limit are combined by , while Distance is combined by -.
You must specify lower_limit and upper_limit of x_position or y_position if use Region.

How to review this PR
Fundamentally, follow this docs
assume that you have already done build of Autoware. To use branch is below.

driving_log_replayer_v2 -> feat/limitation_by_area
perception_eval -> feat/limitation_by_area

mkdir -p ~/driving_log_replayer_v2
gdown -O ~/driving_log_replayer_v2/sample_dataset_v2.tar.zst 'https://docs.google.com/uc?export=download&id=1iCoykBBETI_rGfKEFYYb7LFydF-RJVkC'
tar -I zstd -xvf ~/driving_log_replayer_v2/sample_dataset_v2.tar.zst -C ~/driving_log_replayer_v2/
gdown -O ~/driving_log_replayer_v2/sample-map-planning.zip 'https://docs.google.com/uc?export=download&id=1499_nsbUbIeturZaDj7jhUownh5fvXHd'
unzip -d ~/driving_log_replayer_v2/ ~/driving_log_replayer_v2/sample-map-planning.zip
mv ~/driving_log_replayer_v2/sample-map-planning ~/driving_log_replayer_v2/sample_dataset/map

SAMPLE_ROOT=${path/to/autoware}/src/simulator/driving_log_replayer_v2/sample
cp ${SAMPLE_ROOT}/perception/scenario.yaml ~/driving_log_replayer_v2/perception.yaml

source install setup.bash
 ros2 launch driving_log_replayer_v2 driving_log_replayer_v2.launch.py scenario_path:=$HOME/driving_log_replayer_v2/perception.yaml dataset_index:=0 sensing:=false

You can specify region like below
Filter:
  Distance: null # [m] null [Do not filter by distance] or lower_limit-(upper_limit) [Upper limit can be omitted. If omitted value is 1.7976931348623157e+308]
  Region:
    x_position: -10,10 # [m] null [Do not filter by x_position] or lower_limit,upper_limit [Upper limit can be omitted. If omitted value is 1.7976931348623157e+308]
    y_position: 0,20 # [m] null [Do not filter by y_position] or lower_limit,upper_limit [Upper limit can be omitted. If omitted value is 1.7976931348623157e+308]

Others

https://github.com/tier4/autoware_perception_evaluation/pull/200

@hayato-m126
I change traffic light code in order to adopt this PR (I have checked this PR on my local pc). https://github.com/tier4/driving_log_replayer_v2/pull/68/commits/718d0882137087f0860cebf50345c27e8edd0518
And, I wait for releasing autoware_perception_evaluation which related PR

The reason why colcon test is failing is because the main branch of perception_eval does not contain the update corresponding to this PR.
If you test on hand with the same version, it passes.


Automatically delete oldest record when EON is full of storage

when the storage is full, can OP delete the oldest record automatically to make it keep running?

You can ssh your eon with your phone and run this command:
rm -rf /storage/self/primary/realdata/*
I use the Terminal app on iPhone

Resolution in 3f9059fea886f1fa3b0c19a62a981d891dcc84eb


build: autom4te attempts to mkdir inside a directory that doesn't exist

autom4te attempts to mkdir inside a directory that doesn't exist
It looks like it used either pwd or $HOME instead of the intended tmp dir.
Output from running multipkg:
-bash-3.2$ git-multipkg -v -b git@github.com:ytoolshed/range.git range/libcrangeinfo: Cloning from git@github.com:ytoolshed/range.git to /tmp/gOx0zw7GjB/clone
info: Initialized empty Git repository in /tmp/gOx0zw7GjB/clone/.git/
info: archiving /tmp/gOx0zw7GjB/src to /tmp/gOx0zw7GjB/build
info: Running git archive --format tar HEAD | (cd /tmp/gOx0zw7GjB/build && tar xf -)
info: Gathering revision timestamp information for /tmp/gOx0zw7GjB/clone@HEAD
info: Invoking multipkg on /tmp/gOx0zw7GjB/src
INFO: LOADING /usr/share/multipkg/default.yaml
INFO: LOADING /tmp/gOx0zw7GjB/build/index.yaml
INFO: Using tmpdir /tmp/98Ir6tV0zF
INFO: Building from /tmp/gOx0zw7GjB/build/source
INFO: Building source
INFO: RUNNING: PERL=/usr/bin/perl INSTALLROOT=/tmp/98Ir6tV0zF/install DESTDIR=/tmp/98Ir6tV0zF/install PREFIX=/usr/local PKGVERID=libcrange-1.0.1-1320111582 PACKAGEVERSION=1.0.1 /tmp/gOx0zw7GjB/build/scripts/build

aclocal
autom4te: cannot create /home/cu/tmp/am4t2450.21640: No such file or directory
at /usr/bin/autom4te line 942
aclocal: autom4te failed with exit status: 1
exit 1
Build failed: + aclocal
autom4te: cannot create /home/cu/tmp/am4t2450.21640: No such file or directory
at /usr/bin/autom4te line 942
aclocal: autom4te failed with exit status: 1
exit 1
-bash-3.2$ ls
install_pkg.sh  multipkg-1.2-1321490626.noarch.rpm

I manually created ~cu/tmp and then ran again.  It was successful except for this error:
Unsafe to clean up /home/cu/tmp/S23GbsFtPE at /usr/lib/perl5/site_perl/5.8.8/Seco/Multipkg.pm line 294.

pull request #14 should fix that error, assuming /home/cu/tmp is coming from $TMPDIR

er, that's a bug in multipkg, rather, not range.

https://github.com/nrh/multipkg/commit/61a06865a98a7a6aee9b481093ca84cdafcec67d


rt: Keep ASCII Codes (included extended) i.e. 32-126 & 128-255

Non-ascii characters should not specify extended codes i.e. 128-255.
We need them to write on UTF-8 file (e.g. German umlauts ÄäÖöÜüß)

I incorporated my own change that keep even the non-ascii i.e. those >255 also unescaped.

I have written down here how it worked for me. so that the German letters such as Ü and ß are displayed.
@roman536
https://github.com/VBA-tools/VBA-JSON/pull/168#issuecomment-1647675192


Turn on deployment notifications

What does this pull request do?
Turns on deployment notifications and ties deployment to successful tests.
What is the intent behind these changes?
To let us know if a deployment happens, or more importantly when it fails.

Thank you!


Orchestrator functions failing if not returning anything

We have an eternal orchestrator function in our durable function app. This orchestrator function doesn’t need to return anything. However, with the void return type, the orchestrator function is failing.
Error:
System.InvalidOperationException: The function invocation resulted in a null response. This means that either the orchestrator function was implemented incorrectly, the Durable Task language SDK was implemented incorrectly, or that the destination language worker is not sending the function result back to the host.
Function Code:
@FunctionName('EternalOrchestrator')
    public void startWorkflowSchedule(@DurableOrchestrationTrigger(name = 'ctx') TaskOrchestrationContext ctx) {

        LOGGER.info('Hello World');
        ctx.createTimer(Duration.ofSeconds(10)).await();
        ctx.continueAsNew(null);
    }

Function App Name: possmartpolling-java
Instance Id: testeternal123
Please let me know if any other details are required.

Hi @akshaykumars10. I noticed that your host.json file references a "Preview" build of extension bundles v4.
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview",
    "version": "[4.*, 5.0.0)"
  },

Do you encounter this same problem if you switch to using the non-preview (GA) version of the v4 extension bundles?
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[4.*, 5.0.0)"
  },

Hi @cgillum
I am getting the same error with the non-preview (GA) version of the v4 extension bundles.
Here is the new instance id: testeternal1234

Thanks @akshaykumars10 for confirming. A couple more questions to try and narrow down the problem further:

Do you see this with other orchestrations or just with void + continueAsNew(null)?
Do you see this problem if you change the function signature to return Object and then add return null immediately after your call to ctx.continueAsNew(null)?

Hi @cgillum

If we have void then we are getting this error. Even this simple orchestrator without continueAsNewis also failing with the same error:

    @FunctionName("EternalOrchestrator")
    public void startWorkflowSchedule(@DurableOrchestrationTrigger(name = "ctx") TaskOrchestrationContext ctx) {
        LOGGER.info("Hello World");
        ctx.createTimer(Duration.ofSeconds(10)).await();
    }

I tried changing the function signature to return Object and then added return null in the end.

@FunctionName("EternalOrchestrator")
    public Object startWorkflowSchedule(@DurableOrchestrationTrigger(name = "ctx") TaskOrchestrationContext ctx) {
        LOGGER.info("Hello World");
        ctx.createTimer(Duration.ofSeconds(10)).await();
        ctx.continueAsNew(null);
        return null;
    }

This fails with a different error. Error observed:
Exception: IllegalStateException: The orchestrator has already completed
We have an observation that in the case of eternalOrchestrator continueAsNew should be the last statement in the function. Otherwise, the statements that we have after the continueAsNew start executing. In this case because of return null our eternalOrchestrator is getting marked as completed in the first iteration itself.
We have a different github issue created for this bug: https://github.com/microsoft/durabletask-java/issues/118

Seeing this also even with some basic sample code adapted from the docs.  I still get a 202 Accepted response with the usual method object properties like statusQueryGetUri.  Calling the statusQueryGetUri shows "runtimeStatus": "Failed".
package com.function;

import com.microsoft.azure.functions.annotation.*;
import com.microsoft.azure.functions.*;

import java.time.Duration;

import com.microsoft.durabletask.*;
import com.microsoft.durabletask.azurefunctions.DurableClientContext;
import com.microsoft.durabletask.azurefunctions.DurableActivityTrigger;
import com.microsoft.durabletask.azurefunctions.DurableClientInput;
import com.microsoft.durabletask.azurefunctions.DurableOrchestrationTrigger;

public class EternalOrchTest {
    @FunctionName("Trigger_Eternal_Orchestration")
    public HttpResponseMessage triggerEternalOrchestration(
            @HttpTrigger(name = "req", methods = {HttpMethod.GET, HttpMethod.POST}, authLevel = AuthorizationLevel.ANONYMOUS) HttpRequestMessage<?> req,
            @DurableClientInput(name = "durableContext") DurableClientContext durableContext) {

        String instanceID = "StaticID";
        DurableTaskClient client = durableContext.getClient();
        client.scheduleNewOrchestrationInstance("Periodic_Cleanup_Loop", null, instanceID);
        return durableContext.createCheckStatusResponse(req, instanceID);
    }

    @FunctionName("Periodic_Cleanup_Loop")
    public void periodicCleanupLoop(
            @DurableOrchestrationTrigger(name = "ctx") TaskOrchestrationContext ctx) {
        ctx.callActivity("DoCleanup").await();

        ctx.createTimer(Duration.ofHours(1)).await();
        ctx.continueAsNew(null);
    }

    @FunctionName("DoCleanup")
    public String cleanupActivity(
        @DurableActivityTrigger(name = "name") String name,
        final ExecutionContext context) {
        Integer seconds = 10;
        context.getLogger().info("Sleeping for " + seconds + " seconds...");

        try {
            java.util.concurrent.TimeUnit.SECONDS.sleep(seconds);
        } catch (InterruptedException e) {
            context.getLogger().warning("DoCleanup interrupted: " + e.getMessage());
            e.printStackTrace();
        }

        return "Cleanup executed for " + seconds + " seconds.";
    }
}

GET http://localhost:7071/api/Trigger_Eternal_Orchestration HTTP/1.1
User-Agent: vscode-restclient
accept-encoding: gzip, deflate

HTTP/1.1 202 Accepted
Connection: close
Content-Type: application/json; charset=utf-8
Date: Wed, 05 Apr 2023 21:34:52 GMT
Server: Kestrel
Location: http://localhost:7071/runtime/webhooks/durabletask/instances/StaticID?code={systemKey}
Transfer-Encoding: chunked

{
  "id": "StaticID",
  "purgeHistoryDeleteUri": "http://localhost:7071/runtime/webhooks/durabletask/instances/StaticID?code={systemKey}",
  "sendEventPostUri": "http://localhost:7071/runtime/webhooks/durabletask/instances/StaticID/raiseEvent/{eventName}?code={systemKey}",
  "statusQueryGetUri": "http://localhost:7071/runtime/webhooks/durabletask/instances/StaticID?code={systemKey}",
  "terminatePostUri": "http://localhost:7071/runtime/webhooks/durabletask/instances/StaticID/terminate?reason={text}&code={systemKey}"
}

Executing task: func host start

Azure Functions Core Tools
Core Tools Version:       4.0.5095 Commit hash: N/A  (64-bit)
Function Runtime Version: <IP_ADDRESS>96

[2023-04-05T21:34:44.184Z] Listening for transport dt_socket at address: 5005

Functions:

        Trigger_Eternal_Orchestration: [GET,POST] http://localhost:7071/api/Trigger_Eternal_Orchestration

        DoCleanup: activityTrigger

        Periodic_Cleanup_Loop: orchestrationTrigger

For detailed output, run func with --verbose flag.
[2023-04-05T21:34:45.204Z] Worker process started and initialized.
[2023-04-05T21:34:49.005Z] Host lock lease acquired by instance ID '0000000000000000000000006A2E027E'.
[2023-04-05T21:34:53.073Z] Executing 'Functions.Trigger_Eternal_Orchestration' (Reason='This function was programmatically called via the host APIs.', Id=6351548a-4fe3-42cf-b996-a0e7f4fc11a2)
[2023-04-05T21:34:53.770Z] Function "Trigger_Eternal_Orchestration" (Id: 6351548a-4fe3-42cf-b996-a0e7f4fc11a2) invoked by Java Worker
[2023-04-05T21:34:53.820Z] Executing 'Functions.Periodic_Cleanup_Loop' (Reason='(null)', Id=70a7b720-247b-43b1-b8fb-b0cfab96df57)
[2023-04-05T21:34:53.839Z] Executed 'Functions.Trigger_Eternal_Orchestration' (Succeeded, Id=6351548a-4fe3-42cf-b996-a0e7f4fc11a2, Duration=784ms)
[2023-04-05T21:34:53.867Z] Function "Periodic_Cleanup_Loop" (Id: 70a7b720-247b-43b1-b8fb-b0cfab96df57) invoked by Java Worker
[2023-04-05T21:34:53.884Z] Executed 'Functions.Periodic_Cleanup_Loop' (Failed, Id=70a7b720-247b-43b1-b8fb-b0cfab96df57, Duration=78ms)
[2023-04-05T21:34:53.884Z] System.Private.CoreLib: Exception while executing function: Functions.Periodic_Cleanup_Loop. Microsoft.Azure.WebJobs.Extensions.DurableTask: The function invocation resulted in a null response. This means that either the orchestrator function was implemented incorrectly, the Durable Task language SDK was implemented incorrectly, or that the destination language worker is not sending the function result back to the host.
[2023-04-05T21:34:53.887Z] StaticID: Function 'Periodic_Cleanup_Loop (Orchestrator)' failed with an error. Reason: Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.Periodic_Cleanup_Loop
[2023-04-05T21:34:53.887Z]  ---> System.InvalidOperationException: The function invocation resulted in a null response. This means that either the orchestrator function was implemented incorrectly, the Durable Task language SDK was implemented incorrectly, or that the destination language worker is not sending the function result back to the host.
[2023-04-05T21:34:53.887Z]    at Microsoft.Azure.WebJobs.Extensions.DurableTask.OutOfProcMiddleware.<>c__DisplayClass10_0.<<CallOrchestratorAsync>b__0>d.MoveNext() in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\OutOfProcMiddleware.cs:line 127
[2023-04-05T21:34:53.887Z] --- End of stack trace from previous location ---
[2023-04-05T21:34:53.887Z]    at Microsoft.Azure.WebJobs.Host.Executors.TriggeredFunctionExecutor`1.<>c__DisplayClass7_0.<<TryExecuteAsync>b__0>d.MoveNext() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\TriggeredFunctionExecutor.cs:line 50
[2023-04-05T21:34:53.888Z] --- End of stack trace from previous location ---
[2023-04-05T21:34:53.888Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 581
[2023-04-05T21:34:53.888Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 527
[2023-04-05T21:34:53.888Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 306
[2023-04-05T21:34:53.888Z]    --- End of inner exception stack trace ---
[2023-04-05T21:34:53.888Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 352
[2023-04-05T21:34:53.888Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 108. IsReplay: False. State: Failed. HubName: JavaTestHub. AppName: . SlotName: . ExtensionVersion: 2.9.0. SequenceNumber: 4. TaskEventId: -1

@kaibocai can you help find an owner for this item to help investigate? The problem seems to be that the orchestration middleware in the Java SDK is not returning a value back to the Durable Functions extension. That's basically what this error message means:

System.InvalidOperationException: The function invocation resulted in a null response. This means that either the orchestrator function was implemented incorrectly, the Durable Task language SDK was implemented incorrectly, or that the destination language worker is not sending the function result back to the host.

Hi @kaibocai
How do I have this fixed in my function apps? Do I need to specify the latest version of azure-functions-java-worker in pom.xml?

Hi @kaibocai How do I have this fixed in my function apps? Do I need to specify the latest version of azure-functions-java-worker in pom.xml?

Hi @akshaykumars10 , the java worker is part of the azure function plantform that you functions are running on, it's not a dependency that should be included in your function pom file. Currently we are in the process of azure functions V4.21.1 release, which contains the java worker that has the fix for this issue.
You can view the version of your functions from portal, for example

Once the runtime version is upgrade to 4.21.1, you shouldn't have this issue anymore. We are in the release process of V4.21.1, it should be completed in next week (which means sometime next week your runtime version will automatically bump up to 4.21.1). I will keep you updated once it's released to all region. At the meantime, you can also track it from the above picture. Thanks.


Test content validation update

this is a test PR for content validation

Closing this PR as testing is done

done with testing

done with testing


resolves pyspark requirement for py38

Description
Concludes the issue #618
Checklist

[x] Read the contributing guidelines
[x] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[ ] Updated the documentation to reflect the code changes
[ ] Added a description of this change and added my name to the list of supporting contributions in the RELEASE.md file
[ ] Added tests to cover my changes

Notice

[x] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.

I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorised to submit this contribution on behalf of the original creator(s) or their licensees.

I certify that the use of this contribution as authorised by the Apache 2.0 license does not violate the intellectual property rights of anyone else.

Thank you so much for handling this @kislerdm!

@yetudada my pleasure!


Add CUDA matrix-free smoother

Codecov Report

Merging #217 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #217   +/-   ##
=======================================
  Coverage   89.78%   89.78%
=======================================
  Files          57       57
  Lines        3299     3299
=======================================
  Hits         2962     2962
  Misses        337      337

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7c66df...9af44b1. Read the comment docs.


ENH: special: add ufunc for logarithm of the binomial coefficient

Is your feature request related to a problem? Please describe.
In many discrete distributions the logarithm of the binomial coefficient is used in the logpmf calculation. As the binomial coefficient is fundamentally important elsewhere as well, I think it is worth adding a dedicated function for that in special. The binomial coefficient also quickly overflows while its logarithm does not.
Describe the solution you'd like.
A function special.logbinom(x, y).
CC @steppi , as you are working on binom in #19471 , do you have a strong opinion here by any chance?

Hey, If we're implementing this I have this small implementation:
`
cimport cython
from cython.parallel import prange
cdef extern from "math.h":
double log(double x)
'''
Use Stirling's approximation if approximate = True
i.e log(n!) = n*log(n) - n
'''
cdef double _factorial(double x) nogil:
cdef double i = 1
cdef double _fact = 1
for i in prange(1, x + 1):
_fact *= i
return _fact
cdef double _stirlings_approximation(double x):
return x * log(x) - x
@cython.ufunc
cdef double _logbinomial(double x, double y, bint approximate=True):
cdef double result
if approximate:
result = _stirlings_approximation(x) - _stirlings_approximation(y) - _stirlings_approximation(x - y)
else:
result = _factorial(x) / (_factorial(y) * _factorial(x - y))
return result
`

Hey, If we're implementing this, I have a small implementation here:
cimport cython
from cython.parallel import prange

cdef extern from "math.h":
    double log(double x)

'''
Use Stirling's approximation if approximate = True
i.e log(n!) = n*log(n) - n
'''

cdef double _factorial(double x) nogil:
    cdef double i = 1
    cdef double _fact = 1
    for i in prange(1, x + 1):
        _fact *= i
    return _fact

cdef double _stirlings_approximation(double x):
    return x * log(x) - x

@cython.ufunc
cdef double _logbinomial(double x, double y, bint approximate=True):
    cdef double result
    if approximate:
        result = _stirlings_approximation(x) - _stirlings_approximation(y) - _stirlings_approximation(x - y)
    else:
        result = _factorial(x) / (_factorial(y) * _factorial(x - y))
    return result

Using stirlings isn't a great approach. I think for the general case you'd want to rely on
$${n \choose k} = \frac{1}{(n+1)\mathrm{B}(n - k + 1, k + 1)}$$
and use betaln.

@dschmitz89, I have no objections to adding this; it's just a matter of bandwidth for someone to implement it properly and someone else to review.
@aadya940, thanks for you interest, but beyond your use of Stirling's approximation, it looks like you're just computing integer factorials, which won't work for double x and y in general, leads to a lot of duplicate work in this case for the non-negative integer cases it can handle, and will cause intermediate overflow. You also seem to be using prange for a calculation that is inherently sequential. If you're interested in learning more about numerical computation of special functions I can recommend books or other learning materials.

@dschmitz89, if you want to submit a PR; the way to go would be to do log(binom) for the integer case from the binom source code if (k == kx && (std::fabs(n) > 1E-8 || n == 0)) {, and otherwise do -cephes::lbeta(1 + n - k, 1 + k) - std::log(n + 1). You could just add your implementation to _binom.h from https://github.com/scipy/scipy/pull/19471, once that PR gets in.

Hey @steppi , I would love to read more about this topic. Do recommend some material I can read. I'll also rewrite my code based on the Points you mentioned! Thanks!

Here is an implementation of Binomial Coefficients in C which use gammaln.  I though this might be useful: https://www.grad.hr/nastava/gs/prg/NumericalRecipesinC.pdf  Page 215

Just a word of caution, we cannot and actually do not use Numerical Recipes because the license is not compatible for distributing code under open source licenses. Even looking at the code is debatable in this context for inspiration hence overall we completely avoid NR.

@aadya940, I was actually planning to warn you not to read numerical recipes. Beyond the unusual and restrictive licensing, the code quality isn't great either, though the explanations are good. (I've heard the original Fortran edition did feature idiomatic Fortran which might have been considered good for its time.)
If you can get a hold of a copy, Nico M. Temme's Special Functions: An Introduction to the Classical Functions of Mathematical Physics, is excellent, but will require some familiarity with calculus and undergraduate level complex analysis. You might want to take or sit in on an intro complex analysis course at your university if you don't have experience with the subject.
I've only skimmed through them in the past, but these freely available lecture notes by John Michael Finn seem good, and are more gentle since they walk through a lot of the prerequisite material Temme assumes you already know. You won't get the same level of depth here though. Your goal should be to build yourself up to being able to grapple with  Temme.
If you haven't already, it would be good to become familiar with the content of the classic article  What Every Computer Scientist Should Know About Floating-Point Arithmetic. The Boost math documentation is also a great resource.
If you want to contribute to SciPy, make sure you check the license before studying any relevant source code.

Thanks for your suggestions @steppi . Might be a good opportunity to dig into C++ land again for me. But I am rather swamped at the moment so if anyone has time before, by all means write to the mailing list and submit a PR :).

@aadya940, I was actually planning to warn you not to read numerical recipes. Beyond the unusual and restrictive licensing, the code quality isn't great either, though the explanations are good. (I've heard the original Fortran edition did feature idiomatic Fortran which might have been considered good for its time.)
If you can get a hold of a copy, Nico M. Temme's Special Functions: An Introduction to the Classical Functions of Mathematical Physics, is excellent, but will require some familiarity with calculus and undergraduate level complex analysis. You might want to take or sit in on an intro complex analysis course at your university if you don't have experience with the subject.
I've only skimmed through them in the past, but these freely available lecture notes by John Michael Finn seem good, and are more gentle since they walk through a lot of the prerequisite material Temme assumes you already know. You won't get the same level of depth here though. Your goal should be to build yourself up to being able to grapple with Temme.
If you haven't already, it would be good to become familiar with the content of the classic article What Every Computer Scientist Should Know About Floating-Point Arithmetic. The Boost math documentation is also a great resource.
If you want to contribute to SciPy, make sure you check the license before studying any relevant source code.

Thank you so much @steppi, I skimmed through the material and It looks really interesting, I'll try to complete it as soon as possible. Also, added Complex Analysis for the next sem. Thanks :-)

@aadya940, I was actually planning to warn you not to read numerical recipes. Beyond the unusual and restrictive licensing, the code quality isn't great either, though the explanations are good. (I've heard the original Fortran edition did feature idiomatic Fortran which might have been considered good for its time.)
If you can get a hold of a copy, Nico M. Temme's Special Functions: An Introduction to the Classical Functions of Mathematical Physics, is excellent, but will require some familiarity with calculus and undergraduate level complex analysis. You might want to take or sit in on an intro complex analysis course at your university if you don't have experience with the subject.
I've only skimmed through them in the past, but these freely available lecture notes by John Michael Finn seem good, and are more gentle since they walk through a lot of the prerequisite material Temme assumes you already know. You won't get the same level of depth here though. Your goal should be to build yourself up to being able to grapple with Temme.
If you haven't already, it would be good to become familiar with the content of the classic article What Every Computer Scientist Should Know About Floating-Point Arithmetic. The Boost math documentation is also a great resource.
If you want to contribute to SciPy, make sure you check the license before studying any relevant source code.

Thank you so much @steppi, I skimmed through the material and It looks really interesting, I'll try to complete it as soon as possible. Also, added Complex Analysis for the next sem. Hoping to contribute something good to SciPy soon,  Thanks :-)

@aadya940, I was actually planning to warn you not to read numerical recipes. Beyond the unusual and restrictive licensing, the code quality isn't great either, though the explanations are good. (I've heard the original Fortran edition did feature idiomatic Fortran which might have been considered good for its time.)
If you can get a hold of a copy, Nico M. Temme's Special Functions: An Introduction to the Classical Functions of Mathematical Physics, is excellent, but will require some familiarity with calculus and undergraduate level complex analysis. You might want to take or sit in on an intro complex analysis course at your university if you don't have experience with the subject.
I've only skimmed through them in the past, but these freely available lecture notes by John Michael Finn seem good, and are more gentle since they walk through a lot of the prerequisite material Temme assumes you already know. You won't get the same level of depth here though. Your goal should be to build yourself up to being able to grapple with Temme.
If you haven't already, it would be good to become familiar with the content of the classic article What Every Computer Scientist Should Know About Floating-Point Arithmetic. The Boost math documentation is also a great resource.
If you want to contribute to SciPy, make sure you check the license before studying any relevant source code.

Thank you so much @steppi, I skimmed through the material and It looks really interesting, I'll try to complete it as soon as possible. Also, added Complex Analysis for the next sem. Hoping to contribute something good to SciPy soon, Thanks :-)

Sounds good @aadya940. Complex Analysis is a really beautiful subject. Since the complex derivative is required to be the same along all directions, complex differentiable functions have really nice structure and properties. Looking forward to seeing your first contribution.

Re: https://github.com/scipy/scipy/issues/19473#issuecomment-1793467395: this is quite good for $k > 1$.

The inaccurate band when k is close to n is mostly due to the inaccuracy of betaln itself. For instance, even when there is no loss of precision due to subtraction in the input:
from scipy import special
import mpmath as mp
mp.dps = 1000

def beta_mp(x, y):
    x, y = mp.mpf(float(x)), mp.mpf(float(y))
    return float(mp.log(mp.beta(x, y)))

n = 10000000
k = 10
x = n - k + 1
y = k + 1
assert int(x) == float(x) and int(y) == float(y)
res = special.betaln(x, y)
ref = beta_mp(x, y)
print(abs(res - ref)/abs(ref))
# 2.9874447614386427e-11

Yet if n is increased by a factor of 10, the error drops back to 1e-16. This must be at some threshold where the algorithm switches.
However, for k << 1, the binomial coefficient is very close to 1, so the logarithm is very close to zero, and the relative error of the result is poor. Part of this is because of the explicit addition of 1 to small k: if $\log_{10}k < -16$ or so, then k + 1 == 1, and the result is exactly 0, so we get a relative error of 100%. This can be avoided if you use the relationship:

and take the log of both sides. But even then the error is not good because there is subtractive cancellation error between special.betaln(n-k, k) and np.log(k) for small k. So really, to get good relative error for small $k$, I think we need another relationship, such as an asymptotic approximation for $k \approx 0$. Wolfram Alpha can give a Taylor series expansion, but there might be something better out there.


Fix boolean types

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you successfully run brew tests with your changes locally?

This fixes a few return types, which changed from Boolean to ~Fixnum in 7a38bab333c6808022fe53aac2be9ca2e329cd53. === is not my favourite bit of Ruby syntax, but Rubocop likes it more than it likes !!.

=== is not my favourite bit of Ruby syntax

I actually had in mind backporting Ruby 2.4.0's match? for a while, this might be the time for it.

I actually had in mind backporting Ruby 2.4.0's match? for a while, this might be the time for it.

That would be perfect, yeah.

Take a look at https://github.com/marcandre/backports/tree/master/lib/backports/2.4.0.

Backported the String version.

Thanks @mistydemeo, looks good now!

Should be fixed now, will merge when CI goes green.


Error Install

When installing I have this error. It seems like a version thing. How can I solve it?

This issue should be resolved, can you check again?

If the problem persists, please update with a message here. Closing for now.


[Question] How to show all the available fields for output template?

I'd like to use the output template's fields but I don't know which one is available for the link.
https://github.com/yt-dlp/yt-dlp#output-template
Is there any option to display all the fields with its value? I couldn't find the option.

I wasn't sure it was that. Thanks!


build(deps): go 1.23

Summary by Sourcery
Update CI workflows to test against Go 1.23, ensuring compatibility with the latest Go version.
CI:

Update CI workflows to include Go 1.23 in the testing matrix for addlicense, golangci-lint, and tests workflows.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>4
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 97.32%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
581

Relevant Lines:
597

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>4
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 97.32%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
581

Relevant Lines:
597

💛  - Coveralls


Improved deployment tracking

Objective
Better preservation of our digitised and born digital collections
Measures

Are we more aware of what gets deployed where and when?

Initiatives

[ ]

Dupe of #4637


Add the project to hacktoberfest

Thank you for awesome project!
Could you add this project to hacktoberfest?
It's very simple, only hacktoberfest label creation is required.

Sure! Thanks for the info 😉
I've found only one issue for beginners for now (#53). Maybe in the future there will be more simple tasks to do.
Of course the easiest way to contribute is to find typos and grammar issues in docs 😄


FT: Push metrics for object copy

@ironman-machine

Hello @rahulreddy,
Starting end to end procedure using the following payload:
{"DEFAULT_BRANCH":"master","SCALITY_S3_BRANCH":"dev/ft/pushMetricObjCopy"}
Please follow http://ci.ironmann.io/gh/scality/Integration/3611 for circle status.

@ironman-machine
SCALITY_INTEGRATION_BRANCH=dev/tests/addObjectcopy

Hello @rahulreddy,
Starting end to end procedure using the following payload:
{"SCALITY_INTEGRATION_BRANCH":"dev/tests/addObjectcopy","DEFAULT_BRANCH":"master","SCALITY_S3_BRANCH":"dev/ft/pushMetricObjCopy"}
Please follow http://ci.ironmann.io/gh/scality/Integration/3624 for circle status.

Hello @rahulreddy,
Starting end to end procedure using the following payload:
{"SCALITY_INTEGRATION_BRANCH":"dev/tests/addObjectcopy","DEFAULT_BRANCH":"master","SCALITY_S3_BRANCH":"dev/ft/pushMetricObjCopy"}
Please follow http://ci.ironmann.io/gh/scality/Integration/3625 for circle status.


Found conflicts! Looking for incompatible packages...

Hey everyone, Anaconda just released new packages for 4.2.1 with fixes for all the errors you reported. Please run
conda update spyder

if you already have 4.2.1, or
conda install spyder=4.2.1

or if you had to roll back to a previous version, and let us know if things are working for you now.
Thanks for your help!
Originally posted by @ccordoba12 in https://github.com/spyder-ide/spyder/issues/14687#issuecomment-773641452

Didnt resolve conflicts.
Result is pandas df style.formats dont function on spyder but function on Jupyter.
Spyder update fail.docx

Hey @jpcam, thanks for reporting. Please run
conda update anaconda
conda install spyder=4.2.1

to solve your problem.

Carlos,
Many thanks for taking the time to look at this.
I have since run the updates and things seem to be operating as before.
Sadly, the other issue that I was hoping this update would resolve still remains (see #14728)
I have had a response on #14728 from the Spyder team who said they have reproduced the same issue but that
It would possibly only get resolved in Spyder 5.0.
Unfortunately, I am still a python beginner so won't be able to help you make any progress with it.
So for now patiently wating for 5.0
Have a nice day
JP

From: Carlos Cordoba<EMAIL_ADDRESS>Sent: 09 February 2021 23:05
To: spyder-ide/spyder<EMAIL_ADDRESS>Cc: jpcam<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [spyder-ide/spyder] Found conflicts! Looking for incompatible packages... (#14712)
Hey @jpcamhttps://github.com/jpcam, thanks for reporting. Please run
conda update anaconda
conda install spyder=4.2.1
to solve your problem.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/spyder-ide/spyder/issues/14712#issuecomment-776303793, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK4D55GTZKEA7YS5WQDC4V3S6G5U5ANCNFSM4XIACVSQ.

Ok, I'm closing this one then because it's fixed now.

I'm installing spyder 5.0.5. It says "found conflicts! Looking for incompatible packages". It began to examine conflicts and it's been 24 hours and has not ended. Should I keep waiting till it ends?

@ArnaldoMatute, you need to run first
conda update anaconda

Hi @ccordoba12. I had done it before I tried to install Spyder 5.0.5 and I got what I told you in the other post.

@ArnaldoMatute, what ended up happening? The same thing has just happened to me..

I had to completely uninstall Anaconda the way explained here, under "Full uninstall": https://docs.anaconda.com/anaconda/install/uninstall/
Then, I installed it again.


feat: Added wrappers automatically generated via hook

This fix should make code generation of wrappers simpler (thanks to https://github.com/antonbabenko/pre-commit-terraform/pull/376).
Soon, I would like to copy it to some other modules where users often create multiple similar resources, e.g., EC2 instances, SQS, SNS, ACM, DynamoDB, ECR, IAM, Route53, Security Group (after refactoring).
@bryantbiggs How do you like my English? Should we have hcledit in your GH Action repository (clowdhaus/terraform-composite-actions/pre-commit)?

This PR is included in version 3.2.0 :tada:


clicking to enter new value in text cell shows [null]

have existing rows
add a new text column
click one to edit
see [null] (expected to see empty cell with cursor)

fixed:

Show [null] as row value
Click/Enter will show empty editor for user input


Add PR review conventions to Contributing.md

PR rules:
input from two reviewers Is needed to merge a PR.
After one reviewer approves, the next reviewer can merge.
xref #70

@Aya-S Can we expand the scope of this issue to include a general how to contribute section on the README?
Part 1: clone/fork/branch/add name to list
Part 2: Rules to PR review
PArt 3: claim existing issues  or add new issues, continue contributing.

Maybe the README.md could point to the contributing.md and we could add all these things there, so to keep the README as introductory as possible?

I concur with Martina, we shall have a link to contributing.md from
readme.md
On Tue, 6 Aug, 2019, 8:25 PM Martina G. Vilas<EMAIL_ADDRESS>wrote:

Maybe the README.md could point to the contributing.md and we could add
all these things there, so to keep the README as introductory as possible?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/python-sprints/pandas-mentoring/issues/102?email_source=notifications&email_token=AK6A2Z6OYPJD3DZ2ABARPODQDGGGFA5CNFSM4IJOX342YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3VNGLY#issuecomment-518705967,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK6A2Z2YSLBEGGOHKACNPPDQDGGGFANCNFSM4IJOX34Q
.

A link to contributing.md does make sense

Yes, it does.
Maybe something on the lines of
https://github.com/pandas-dev/pandas/blob/master/.github/CONTRIBUTING.md
But modified to suit our group's need?

That would be cool! We already have a contributing.md file in the root folder of our project (so we don't need to create one from scratch), but maybe we could add the information of the link you provided there @sara-02

@martinagvilas second that.

Is anyone working on this at the moment?

PArt 3: claim existing issues or add new issues, continue contributing.

@sara-02 did you mean the rules for taking over existing issues and how to add new issues?

@Yomdroid Noone is working on this right now.
Yes, basic rules like:
Check the issue already exists, if open add your comment on it. If closed then reopen and add comments.
For new issues: Follow issue template,
for reporting errors, give details of the version/os you are running.

Thanks, I'll work on it.

For existing issues, how long does the person assigned to it have, before someone else can attempt to do it? @sara-02

For existing issues, how long does the person assigned to it have, before someone else can attempt to do it? @sara-02

Not aware of that, for the time being, for the time being, we are just working on mutual agreement.

My opinion is that you can simply ping the person to see if it's still working or if they are happy that you take it. That could be somehow rude if you don't wait a reasonable time, but I think after few days, or a week or something that makes sense (depending on the context) that's perfectly fine.

Fixed in #113
@Aya-S feel free to open this issue in case you think more work needs to be done in the contributing.md.


Cancelling uploading file breaks the uploading

To reproduce:

Select "My computer" in "Open your own file" list, file selection dialog opens.
Cancel file selection.
Clicking "My computer" doesn't work.

I'm using chromium

Fixed.


Remove ChainFragment

ChainFragments were hardly ever used anymore, only in some tests. We use
AnchoredFragments almost everywhere. There are very few situations in which
you don't want an anchor.
This means that ChainFragments are only really there to make implementing
AnchoredFragments easier, i.e., by wrapping them. However, we can simply
inline them, dropping tons of code in the process.
In the process, simplify the chain and fragment generators. Drop a few pattern
synonyms. Stop using block number 0 for genesis and remove legacyHeadBlockNo.

I added the prop_sliceRange that I forgot about. There were no other tests for the ChainFragment that needed to be ported over.

bors merge


Statistical Methods for Recommender Systems (English Edition)

I want to translate your book into Chinese, and I hope I can get your permission.

Thanks for your interest. The book has been translated into Chinese. See: https://item.jd.com/12716548.html


Logo size in TopBar component

[ ] Fix logo styles so that it doesn't overflow the bounds of the TopBar
[ ] Fix alignment when the "logo" is just text and not an image
[ ] Fix spelling: ref-Brand-img should be rev-Brand-img

This is currently being worked on in #331.


charge-lnd: 0.2.4 -> 0.2.12

Description of changes

charge-lnd: 0.2.4 -> 0.2.12
Fixed a minor build issue with the newest version. I added myself as a maintainer.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.


Better pkgs/by-name errors, minor fixes and improvements

Description of changes
This is a bit of a random assortment of pkgs/by-name improvements:

Better errors in many cases. In particular errors when a package in pkgs/by-name is overridden with a manual definition in all-packages.nix, but it's not of the expected form.
A more spaced-out error formatting for some errors (could do all in the future)
A minor false negative fix, for when a package in pkgs/by-name is overridden with a callPackage whose first argument is not a path. This also makes the code easier to understand.
A low-hanging performance improvement by running the check on two cores, which should shave around 25 seconds from CI.
Indications about whether the PR would break master, fixes master, or doesn't affect it when merged. This should make sure that we don't run into cases like https://github.com/NixOS/nixpkgs/issues/289649 again. This essentially re-implements https://github.com/NixOS/nixpkgs/pull/257735, but now in Rust

I fully admit that the code isn't the best, but I've been getting side-tracked trying to improve it for like 2 days :upside_down_face:
This relates to https://github.com/NixOS/nixpkgs/issues/286396, more is needed to address that, probably just more docs, but that can be done separately
This work is sponsored by Antithesis :sparkles:
Things done

[x] Updated/added tests

Add a :+1: reaction to pull requests you find important.

@philiptaron Thanks for the extensive review! I either addressed or implemented all of them now. I was also able to do a nice refactoring using RelativePath[Buf]


Hello, I encountered the following bug while using the llama3-70B chat model to run the simple_pipeline.py script

Traceback (most recent call last):
  File "/Data1/home/fanziqi/func_eval/FlashRAG/examples/quick_start/simple_pipeline.py", line 35, in <module>
    output_dataset = pipeline.run(test_data,do_eval=True)
  File "/Data1/home/fanziqi/func_eval/FlashRAG/flashrag/pipeline/pipeline.py", line 80, in run
    retrieval_results = self.retriever.batch_search(input_query)
  File "/Data1/home/fanziqi/func_eval/FlashRAG/flashrag/retriever/retriever.py", line 60, in wrapper
    results, scores = func(self, query_list, num, True)
  File "/Data1/home/fanziqi/func_eval/FlashRAG/flashrag/retriever/retriever.py", line 90, in wrapper
    results, scores = func(self, query_list, num, True)
  File "/Data1/home/fanziqi/func_eval/FlashRAG/flashrag/retriever/retriever.py", line 158, in batch_search
    return self._batch_search(*args, **kwargs)
  File "/Data1/home/fanziqi/func_eval/FlashRAG/flashrag/retriever/retriever.py", line 279, in _batch_search
    batch_scores, batch_idxs = self.index.search(batch_emb, k=num)
  File "/Data1/home/fanziqi/.conda/envs/flashrag/lib/python3.9/site-packages/faiss/class_wrappers.py", line 329, in replacement_search
    assert d == self.d
AssertionError

Hello, your problem seems due to the inconsistency between the dimensions of the embedding model and the vector in the faiss index.
Since we use our pre-built index (using e5 as the embedding) in simple.pipeline.py, you may not be able to match if you use other embedding model. If you need to use your own embedding model, you need to create a new index.

I have checked your previous issue and found that the retrieval model you used is bge-small with a hidden size of 384, which does not match the hidden size of e5 (768). Therefore, the index of e5 cannot be used.

I have checked your previous issue and found that the retrieval model you used is bge-small with a hidden size of 384, which does not match the hidden size of e5 (768). Therefore, the index of e5 cannot be used.

Great, I have now changed the retrieval model to bce-embedding-base_v1 and changed the reranker model to bce-reranker-base_v1, with both models having a hidden layer of 768. Do I still need to create a new index?
And, i have meet a new bug:
Traceback (most recent call last):
  File "/Data1/home/fanziqi/func_eval/FlashRAG/examples/quick_start/simple_pipeline.py", line 35, in <module>
    output_dataset = pipeline.run(test_data,do_eval=True)
  File "/Data1/home/fanziqi/func_eval/FlashRAG/flashrag/pipeline/pipeline.py", line 80, in run
    retrieval_results = self.retriever.batch_search(input_query)
  File "/Data1/home/fanziqi/func_eval/FlashRAG/flashrag/retriever/retriever.py", line 60, in wrapper
    results, scores = func(self, query_list, num, True)
  File "/Data1/home/fanziqi/func_eval/FlashRAG/flashrag/retriever/retriever.py", line 92, in wrapper
    results, scores = self.reranker.rerank(query_list, results)
  File "/Data1/home/fanziqi/.conda/envs/flashrag/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Data1/home/fanziqi/func_eval/FlashRAG/flashrag/retriever/reranker.py", line 45, in rerank
    assert topk < min([len(docs) for docs in doc_list]), "The number of doc returned by the retriever is less than the topk."
AssertionError: The number of doc returned by the retriever is less than the topk.

but i have set 'retrieval_topk': 1,

The simple pipeline.py is only for testing the normal operation of the overall process, and the index we provide is just a toy index. If you need to use it to run experimental results, you need to create your own index, otherwise you can temporarily use the ready-made one.
For your new bug, you should set retrieval_topk larger than rerank_topk in the config.
retrieval_topk is used to control the number of documents returned by the retriever, and rerank_topk is used to control the number of documents retained after rerank.

I understand and thank you very much for your prompt reply!


fix(generator): enhance check if the generator should serve an update or a new app

Purpose of this pull request?

Enhancement

The pull request refers to the:

generator (generator-nitro)

What changes did you make?
enhance check if the generator should serve an update or a new app. Closes #194.
Is there anything you'd like reviewers to focus on?

Here we go :-) Replaces #195

danke sehr ;-)


README.md - A/B Devices Information

I've added some important information about required step for A/B devices to copy firmware from active to inactive slot.
We need Chinese translation though...

@GoldRenard I think it would be better to place it inside the project website or forum

Good idea! I will create new PR for a website.
I'm going to update current PR as well but with and A/B tags in device list with short remark.

Will make another PR.


can add matrix to rendere objects member?

Hi @imfox , sorry but this lib is deprecated.
If you have the time and will you can try to do the PR yourself and if its good I'll merge it, or I recommend to check out the successor I've recently released, Shaku: https://github.com/RonenNess/Shaku.
Thanks

thank you for your reply, I'll try to understand the new lib


Grouping permissions

it would be nice if there is an attribute "GroupName" in permission entity for two reasons
1- You could implement a method to grant or deny all permissions under certain group
2- In role permission managment UI we can display the permissions under groups
Thank you

No need to that since a permission can have children permissions. See https://github.com/aspnetboilerplate/aspnetboilerplate/issues/304


Can't run on windows with python 3.11>> because lxml fails to build

Hi there!
Just wanted to report an issue running this action on windows with python versions 3.11 or greater.
There appears to be no pre-built package for lxml 4.9.1 for these versions of python, so it falls back to a source build which fails with the following error:
    C:\Users\runneradmin\AppData\Local\Temp\pip-install-mjpb3dnp\lxml_45bbd1db508e4fd98d76804d8e1e9ffa\src\lxml\includes/etree_defs.h(14): fatal error C1083: Cannot open include file: 'libxml/xmlversion.h': No such file or directory
    Compile failed: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2

Looks like lxml versions > 4.9.1 have pre built packages for 3.11 so bumping the version number of this dependency should be enough to fix it if no other issues pop up.

Thanks for reporting, fixed in #387. Will cut a new release soon.

Awesome, thank you!

This has been released.


fix: If FluentNavMenu is shown shown programmatically the CollapsedChildNavigation doesn't work

🐛 Bug Report
If FluentNavMenu is placed inside an if and the property that if evaluates, is changed OnAfterRenderAsync, after some async work, the CollapsedChildNavigation stops working.
💻 Repro or Code Sample
I have provided an example project illustrating the problem.
🤔 Expected Behavior
The collapsed child navigation should work.
😯 Current Behavior

💁 Possible Solution
I'm willing to contribute a fix, but I don't know where the issue is.
🔦 Context
On OnAfterRenderAsync I am evaluating whether to show the collapse button of the nav menu (the Collapsible property).  This causes the collapse button to show after the rest of the menu is shown. It's much more preferable to show the complete menu later, than just the collapse button.
What happens without the check:

protected override async Task OnAfterRenderAsync(bool firstRender)
{
    if (firstRender)
    {
        var width = await JsRuntime.InvokeAsync<int>("getViewportWidth");
        await SetCollapsibleBasedOnWidth(width);
        await JsRuntime.InvokeVoidAsync("addWindowResizedEventListener", _objRef);
        _hasFinishedRendering = true;
        StateHasChanged();
    }
}

[JSInvokable("SetCollapsibleBasedOnWidth")]
public Task SetCollapsibleBasedOnWidth(int width)
{
    _isCollapsible = width > 768;
    StateHasChanged();
    return Task.CompletedTask;
}

function addWindowResizedEventListener(dotNetObjectRef) {
    window.addEventListener('resize', (event) => {
        dotNetObjectRef.invokeMethodAsync('SetCollapsibleBasedOnWidth', event.currentTarget.innerWidth)
    });
}

function getViewportWidth() {
    return window.innerWidth;
}

🌍 Your Environment

OS & Device: Windows 10 on Dell laptop
Browser: Microsoft Edge
.NET and Fluent UI Blazor library Version: 8.0.206 and 4.9.1

I have provided an example project illustrating the problem.

I'm not seeing that code/project. We can't (won't) investigate based on the few code blocks you posted. Also, we need details on how you are running the site. Server/WebAssembly/Auto and with which interactivity
From what you are describing, it seems like you want the component to do something else then what it was built for. Not sure if we can accommodate to that.

Oops forgot to attach the file 😳
nav-issue.zip

From what you are describing, it seems like you want the component to do something else then what it was built for. Not sure if we can accommodate to that.

I am using code from the examples mostly. The issue is, that I want to hide the collapse button on small screens.

Hm ok this works, but I would also need to make sure to also expand it just in case the user has not expanded the sidebar.

@vnbaaij would it be possible to expand it without js callback?

You can use @bind-Expanded="@Expanded". You can also make it use a menu in collapsed state. See https://www.fluentui-blazor.net/NavMenu#collapsible-navigation-example

Yes, but I would have to something like this and still use js:
@implements IDisposable
@rendermode InteractiveServer
@inject IJSRuntime JsRuntime

    <input type="checkbox" title="Menu expand/collapse toggle" id="navmenu-toggle" class="navmenu-icon"/>

        <FluentIcon Value="@(new Icons.Regular.Size20.Navigation())" Color="Color.Fill"/>

        <FluentNavMenu Id="main-menu" Collapsible="true" bind-Expanded="_expanded" Width="250" Title="Navigation menu" CollapsedChildNavigation="true" CustomToggle="true">
            <FluentNavLink Href="/" Match="NavLinkMatch.All" Icon="@(new Icons.Regular.Size20.Home())" IconColor="Color.Accent">Home</FluentNavLink>
            <FluentNavGroup Title="example" Icon="@(new Icons.Regular.Size20.Apps())" Expanded="true">
                <FluentNavLink Href="counter" Icon="@(new Icons.Regular.Size20.NumberSymbolSquare())" IconColor="Color.Accent">Counter</FluentNavLink>
                <FluentNavLink Href="weather" Icon="@(new Icons.Regular.Size20.WeatherPartlyCloudyDay())" IconColor="Color.Accent">Weather</FluentNavLink>
            </FluentNavGroup>
        </FluentNavMenu>

@code {
    private bool _expanded = false;
    private DotNetObjectReference<NavMenu> _objRef;

    protected override void OnInitialized() => _objRef = DotNetObjectReference.Create(this);

    protected override async Task OnAfterRenderAsync(bool firstRender)
    {
        if (firstRender)
        {
            await JsRuntime.InvokeVoidAsync("addWindowResizedEventListener", _objRef);
            StateHasChanged();
        }
    }

    [JSInvokable("SetExpandedBasedOnWidth")]
    public Task SetExpandedBasedOnWidth(int width)
    {
        if (width > 768)
        {
            _expanded = true;
            StateHasChanged();
        }

        return Task.CompletedTask;
    }

    public void Dispose() => _objRef?.Dispose();
}

function addWindowResizedEventListener(dotNetObjectRef) {
    window.addEventListener('resize', (event) => {
        dotNetObjectRef.invokeMethodAsync('SetExpandedBasedOnWidth', event.currentTarget.innerWidth)
    });
}

Admittedly this works much better though.

So your issue is solved?

Sure, if you have no other proposal. Thank you very much for your help!

No, nothing I can think of. We dont have a way to make this a pure Blazor solution as we don't have info about screen size etc.

I'm writing an update on this for anyone having a similar issue. I just found a way more elegant solution that seems to be working like a charm and has no js dependency. This will hold the state correctly no matter which button you click and persists on any screen size.
@vnbaaij maybe the demo/docs example can be updated with this. If you agree, would you like me to submit a pull request?

    <input type="checkbox" title="Menu expand/collapse toggle" id="navmenu-toggle" class="navmenu-icon" checked="@_expanded" @oninput="ExpandedChanged" />
    <FluentIcon Value="@(new Icons.Regular.Size20.Navigation())" Color="Color.Fill" />

        <FluentNavMenu Id="main-menu" Collapsible="true" Width="250" Title="Navigation menu" @bind-Expanded="_expanded" CustomToggle="true">
            <FluentNavLink Href="/" Match="NavLinkMatch.All" Icon="@(new Icons.Regular.Size20.Home())" IconColor="Color.Accent">Home</FluentNavLink>
            <FluentNavLink Href="counter" Icon="@(new Icons.Regular.Size20.NumberSymbolSquare())" IconColor="Color.Accent">Counter</FluentNavLink>
            <FluentNavLink Href="weather" Icon="@(new Icons.Regular.Size20.WeatherPartlyCloudyDay())" IconColor="Color.Accent">Weather</FluentNavLink>
        </FluentNavMenu>

@code {
    private bool _expanded = false;

    private Task ExpandedChanged(ChangeEventArgs changeEventArgs)
    {
        if (changeEventArgs.Value is bool expanded)
        {
            _expanded = expanded;
        }

        return Task.CompletedTask;
    }

    private Task ExpandedChanged(bool expanded)
    {
        _expanded = expanded;
        return Task.CompletedTask;
    }
}

I don't know why I went down that complicated rabbit hole previously, with all the callbacks.


Lack example of Authorizer and Filter

Need to have example to those in README.md example section.

closed by #34.


[BugFix] fix mv on iceberg

Fixes #issue
Include PR about str2date partitioned MV:

#32458
#32462
#32638
#32744

What type of PR is this:

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.2
[ ] 3.1
[ ] 3.0
[ ] 2.5

[FE Incremental Coverage Report]
:heart_eyes: pass : 26 / 29 (89.66%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/sql/common/SyncPartitionUtils.java
9
12
75.00%
[96, 97, 168]

:large_blue_circle:
com/starrocks/catalog/MaterializedView.java
2
2
100.00%
[]

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
15
15
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:heart_eyes: pass : 26 / 29 (89.66%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/sql/common/SyncPartitionUtils.java
9
12
75.00%
[96, 97, 168]

:large_blue_circle:
com/starrocks/catalog/MaterializedView.java
2
2
100.00%
[]

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
15
15
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:heart_eyes: pass : 26 / 29 (89.66%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/sql/common/SyncPartitionUtils.java
9
12
75.00%
[96, 97, 168]

:large_blue_circle:
com/starrocks/catalog/MaterializedView.java
2
2
100.00%
[]

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
15
15
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:heart_eyes: pass : 26 / 29 (89.66%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/sql/common/SyncPartitionUtils.java
9
12
75.00%
[96, 97, 168]

:large_blue_circle:
com/starrocks/catalog/MaterializedView.java
2
2
100.00%
[]

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
15
15
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

@Mergifyio backport branch-3.2


Show which object properties are required or not by default

Would not be amazing to show which object properties are required or not in the generated Markdown?
So this schema:
type: object
required:
  - colours
additionalProperties: false
properties:
  colours:
    [...]
  numbers:
    [...]

would generate a different Markdown than this schema:
type: object
required:
  - colours
  - numbers
additionalProperties: false
properties:
  colours:
    [...]
  numbers:
    [...]

Currently, additionalProperties is taken into account, but required is ignored AFAIK.

Which output syntax would be better?
I can think of 4 choices right now:
FIRST

colour (string, required)
flavour (string, optional)

SECOND

colour (string, r)
flavour (string, o)

THIRD

colour (string, *)
flavour (string)

FOURTH

colour, * (string)
flavour (string)

I'm in favor of the first one (eventually by specifying only the required) =>
* colour (string, required)
* flavour (string)

This is a proposal, @sbrunner: https://github.com/sbrunner/jsonschema2md2/pull/18

Released in 0.8.0 :-)

@sbrunner Any for plans for updating it in pip? 🙏

Thanks for the notification, I expected that was correctly done.
Now it's OK :-)


Fixing CodeQL action

Fixes issue #363
What

Fixes CodeQL scan action

Why

to fix a bug

Checks
https://github.com/yeagerai/genlayer-simulator/actions/runs/10298453094

@YeagerAI-Bulat please adjust the PR title and the branch name with its proper issue number and conventions


🔧 iOS Obj-C++ Frame Processor Plugin Build Failed: 'FrameHostObject.h' not found

How were you trying to build the app?
I tried to build my Frame Processor Plugin (written in Objective-C++), but it failed with 'FrameHostObject.h' file not found. The issue seems to appear in VisionCamera/FrameProcessors/FrameProcessor.h:
// FrameProcessor.h

#pragma once

#import "Frame.h"
#import <AVFoundation/AVFoundation.h>
#import <Foundation/Foundation.h>

#ifdef __cplusplus
#import "FrameHostObject.h" >>> 'FrameHostObject.h' file not found
#import "WKTJsiWorklet.h"
#import <jsi/jsi.h>
#import <memory.h>
#endif

NS_ASSUME_NONNULL_BEGIN

@interface FrameProcessor : NSObject

- (instancetype)init NS_UNAVAILABLE;

#ifdef __cplusplus
- (instancetype _Nonnull)initWithWorklet:(std::shared_ptr<RNWorklet::JsiWorklet>)worklet
                                 context:(std::shared_ptr<RNWorklet::JsiWorkletContext>)context;

- (void)callWithFrameHostObject:(std::shared_ptr<FrameHostObject>)frameHostObject;
#endif

- (void)call:(Frame*)frame;

@end

NS_ASSUME_NONNULL_END

This is weird, since the Xcode editor seems to recognize FrameHostObject.h just fine.
Reopening #1682 since it was marked as stale.
Full build logs
Showing Recent Errors Only

...

In file included from /Users/user/Projects/project/ios/LGAFrameProcessorPlugin.mm:8:
In file included from /Users/user/Projects/project/ios/Pods/Headers/Public/VisionCamera/FrameProcessorPlugin.h:12:
In file included from /Users/user/Projects/project/ios/Pods/Headers/Public/VisionCamera/FrameProcessorPluginRegistry.h:13:
In file included from /Users/user/Projects/project/ios/Pods/Headers/Public/VisionCamera/VisionCameraProxyHolder.h:10:
/Users/user/Projects/project/ios/Pods/Headers/Public/VisionCamera/FrameProcessor.h:16:9: fatal error: 'FrameHostObject.h' file not found
#import "FrameHostObject.h"
        ^~~~~~~~~~~~~~~~~~~
7 warnings and 1 error generated.

/Users/user/Projects/project/ios/Pods/Headers/Public/VisionCamera/FrameProcessor.h:16:9: 'FrameHostObject.h' file not found

Activity Log Complete    7/9/24, 12:34 AM    8.3 seconds

Project dependencies
"dependencies": {
    "@expo/vector-icons": "^14.0.2",
    "@react-navigation/native": "^6.0.2",
    "@shopify/react-native-skia": "^1.3.7",
    "expo": "~51.0.17",
    "expo-font": "~12.0.7",
    "expo-linking": "~6.3.1",
    "expo-router": "~3.5.17",
    "expo-splash-screen": "~0.27.5",
    "expo-status-bar": "~1.12.1",
    "expo-system-ui": "~3.0.6",
    "expo-web-browser": "~13.0.3",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "react-native": "0.74.2",
    "react-native-fast-tflite": "^1.2.0",
    "react-native-paper": "^5.12.3",
    "react-native-reanimated": "~3.10.1",
    "react-native-safe-area-context": "4.10.1",
    "react-native-screens": "3.31.1",
    "react-native-vision-camera": "^4.3.2",
    "react-native-web": "~0.19.10",
    "react-native-worklets-core": "^1.3.3",
    "vision-camera-resize-plugin": "^3.1.0"
  },

VisionCamera Version
4.4.1
Target platforms
iOS
Operating system
MacOS
Can you build the VisionCamera Example app?
Yes, I can successfully build the Example app here
Additional information

[ ] I am using Expo
[X] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

@mrousavy Well, I can't build my frame processor plugin because of this. Is there a way to fix it?

Ahh, now I get it. I thought you were including FrameHostObject.h yourself.
Okay I see the issue. I'll think about this a bit, but this is not easy to fix. I assume __cplusplus only for internal usage.

As a workaround, you can just build an Objective-C Frame Processor plugin, which then later calls into Objective-C++ inside it's callback body.

Ahh, now I get it. I thought you were including FrameHostObject.h yourself.

Yeah, sorry for the confusing description.
I'll just build the plugin in Swift and write necessary methods in Obj-C++ for now.

In the future everything will be Swift. It's the direction I want to move towards with VisionCamera.
With Swift 5.9/6 you can call a lot of C++ methods directly anyways, no need for Objective-C(++) here. Also, Swift method calls are faster than Objective-C method calls ("message sends").

The only reason I need to use Objective-C++ at all is because I need the OpenCV framework, which only seems to work with Objective-C++ as of now.
I'm glad you're planning to prioritize Swift in the future, it's been a nightmare trying to learn Objective-C syntax tbh

I think OpenCV can be used from Swift. Either directly, or with some simple C++ bridging code. Either way, this should be a tiny bit faster than Objective-C.


chore: move protos to canonical location

... and configure buf.

I've bring in most of this PR, the only thing I left for now is the rename of the Rust crate which is still substreams-sink-database.
I'll check to publish a new name when we will bridge entity-change and database-change together.