V3 client bulk operaations cause more 429 than client V2 bulk executor

We recently do client migrate from v2 to v3.
For the same batch size and same server RU settings, v3 client causes extremely higher throughout than v2.
Is anything options can be set to control the rate ? or We can only control the batch size ourselves?

@RetYn the SDK should be scaling back based on 429. The following PR added support.
Can you provide some more information?

How many documents is each client processing? Is the v3 client processing more documents than the v2?
How many more 429s are being thrown?
Which SDK version are you using?

@RetYn V3 Bulk is faster in terms of sending requests than V2 Bulk, this means that the bottleneck on the client side (networking) was further reduced and improved and the potential throughput is increased. The reason the volume of 429s was less with V2 Bulk is because the processing was a bit slower on the client side, so it might have more "breath" time between requests.
The fact that you get 429s it means the volume of data you are trying to send exceeds your provisioned throughput. So either increase the throughput (or use something like Autoscale) to accommodate your data needs, or reduce the volume of data you want to send (you could create smaller batches of documents each time).
You have full control on the batch size, you are the one executing the concurrent operations. Following the bulk documented pattern you can choose the volume of operations to perform at a given time.

Closing due to lack of activity, please reopen with answers to any of the questions


Restoring the previous value does not work properly for lights off

Home Assistant 2021.11.1
Component Version 0.1.4
The logs:
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX state restored: 1 <- 13
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX entity brightness restored: 1 <- 13
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX state restored: 2 <- 13
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX entity brightness restored: 2 <- 13
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX state restored: 3 <- 13
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX entity brightness restored: 3 <- 13
....
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX state restored: 1 <- 133
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX entity brightness restored: 1 <- 0
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX state restored: 2 <- 164
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX entity brightness restored: 2 <- 0
....
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX state restored: 10 <- 255
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX entity brightness restored: 10 <- 0
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX state restored: 11 <- 255
2021-11-10 18:41:36 DEBUG (MainThread) [custom_components.dmx.light] DMX entity brightness restored: 11 <- 0
So if the light had OFF state before restarting HA it does not take it into consideration during the restore previous status of the entity and set restored dmx_values for this light.
It seems there is a bug in
async def async_added_to_hass(self) -> None:
    if old_state.attributes.get("dmx_values"):
        old_dmx_values = old_state.attributes.get("dmx_values")
        _LOGGER.debug(
            f"DMX state restored: {self._channel} <- {str(old_dmx_values)}"
        )
        _LOGGER.debug(
            f"DMX entity brightness restored: {self._channel} <- {str(self._brightness)}"
        )

        asyncio.ensure_future(
            self._dmx_gateway.set_channels_async(
                self._channels, old_dmx_values, self._send_when_added
            )
        )

after code update my lights behave strange - turning on two lights in the same time gives only one light flashing, but after turning another light (third one) missing one start to flash.
I am not sure, but probably issue opened by sapdemon have connection with my problem.


deploy failed error

I am trying to deploy a nextjs app and i keep getting this error in the build logs, my repo is not associated with any organization also. its just a private repo.
Build started for scrapeweb
An explicit image name was provided (ghcr.io/smyja/scrapweb:df37ec852f198237d0b214848675edf0df0f99b6-gitsha). Therefore, no build process is needed.
Pulling this image: ghcr.io/smyja/scrapweb:df37ec852f198237d0b214848675edf0df0f99b6-gitsha This process might take a few minutes.
Build has failed!
----------------------
Deploy failed!
Error: (HTTP code 500) server error - Head "https://ghcr.io/v2/smyja/scrapweb/manifests/df37ec852f198237d0b214848675edf0df0f99b6-gitsha": denied

here is my build_and_push.sh
#!/bin/bash

set -e

cd $CONTEXT_DIR
rm /tmp/build_args || echo OK
env >/tmp/build_args
echo "--build-arg \""$(cat /tmp/build_args | sed -z 's/\n/" --build-arg "/g')"IGNORE_VAR=IGNORE_VAR\"" >/tmp/build_args
BUILD_ARGS=$(cat /tmp/build_args)
docker logout ghcr.io
COMMAND="docker build -t $FULL_IMAGE_NAME -t $IMAGE_NAME_WITH_REGISTRY:latest -f $DOCKERFILE $BUILD_ARGS --no-cache ."
/bin/bash -c "$COMMAND"
docker push $IMAGE_NAME_WITH_REGISTRY:latest
docker push $FULL_IMAGE_NAME
rm /tmp/build_args

Here is my yaml file
name: Deploy to caprover

env:
    CONTEXT_DIR: './'
    IMAGE_NAME: ${{ github.repository }}
    DOCKERFILE: ./Dockerfile
    # CAPROVER_APP: myapp-staging
    DOCKER_REGISTRY: ghcr.io

on:
    push:
        branches:
            - main
        # you can specify path if you have a monorepo and you want to deploy if particular directory is changed, make sure to update `CONTEXT_DIR` too
        # paths:
        #   - "backend-app/**"

jobs:
    build-and-publish:
        runs-on: ubuntu-latest
        steps:
            - uses: actions/checkout@v1
            - run: |
                  echo "IMAGE_NAME_WITH_REGISTRY=$DOCKER_REGISTRY/$IMAGE_NAME" >> $GITHUB_ENV
                  export IMAGE_NAME_WITH_REGISTRY=$DOCKER_REGISTRY/$IMAGE_NAME
                  echo "FULL_IMAGE_NAME=$IMAGE_NAME_WITH_REGISTRY:$GITHUB_SHA-gitsha" >> $GITHUB_ENV

            - name: Log in to the Container registry
              uses: docker/login-action@f054a8b539a109f9f41c372932f1ae047eff08c9
              with:
                  registry: ${{ env.DOCKER_REGISTRY }}
                  username: ${{ github.actor }}
                  password: ${{ secrets.GITHUB_TOKEN }}

            - name: Build and Push Release to DockerHub
              shell: bash
              run: chmod +x ./build_and_push.sh

            - name: Use Node.js ${{ matrix.node-version }}
              uses: actions/setup-node@v3
              with:
                node-version: ${{ matrix.node-version }}
                cache: "npm"
            - run: npm ci
            - run: npm run build --if-present
            - run: npm run test --if-present

            - name: Deploy to CapRover
              uses: caprover/deploy-from-github@d76580d79952f6841c453bb3ed37ef452b19752c
              with:
                  server: ${{ secrets.CAPROVER_SERVER }}
                  app: ${{ secrets.APP_NAME }}
                  token: '${{ secrets.APP_TOKEN }}'
                  image: '${{ env.FULL_IMAGE_NAME }}'

Dupe of https://github.com/caprover/caprover/issues/1756


Update intrinsic_funs.json

Descriptions for MAX and MIN functions were swapped

Nice catch, thanks for the contribution!


Use own testing tool instead of JMeter [Feature Request]

Is your feature request related to a problem? Please describe.
JUnit is limited in what kinds of tests it can perform.
Describe the solution you'd like
I'd like to be able to plug in my own testing tool instead of JUnit, and otherwise take advantage of this solution.

Hi @vid Did you mean JMeter instead of JUnit? Could you share the scenarios you are looking to test and the tool you use? I can check if there is a workaround using JMeter. I will add the feature request to our backlog, we don't have a strong opinion on this topic yet.

@kaumishra you're right,I meant JMeter, edited.
We are using a headless browser approach (with playwright). Of course it's more resource intensive, but lets us reuse our e2e tests and offers more opportunity for analysis. If we knew what statistics to emit and how to plug it into azure-load-testing, we can do the work to plug it in.
Thanks!

@vid Thanks for sharing. That scenario is very interesting. I am adding @johnsta from our team to comment on this.

Hi @johnsta, anything we can do to help move this forward?

Hi @vid, we're still very early in our thinking here. I've followed up with you via email to see if we can set up a call to discuss more.

I would like to add a 👍 to adding Playwright to AZL. Currently using this for E2E testing of our web apps and would very much would like to increase the scale of the tests with AZL to reproducere realistic simulated user load.


Suricata: only list physical interfaces

On the settings tab in Intrusion Detection, there is a interfaces field.
As soon as we click on it, we're presented with all our interfaces as opposed to just the list of physical interfaces.
The logic should be modified to only present physical interfaces as virtual ones are not supported.

Something slightly different went in with #935

Well, you've added something to the help file, but people who don't read it may still end up with a non-working config. Isn't there a way to detect physical interfaces?

We certainly can't fix people who don't read help files. :)

True, but in terms of usability, you can hide options which a user should never see because they don't work. It's not always a good idea when there isn't a clear distinction since users could feel there is something wrong, but in this case, showing something like WAN (em1), LAN1 (em0), LAN2 (em2) could be useful.
I'm just wondering is there is already an easy way to query for the real interfaces used in OPNsense.


Screens update issue after some actions are performed

Description
Somethings the platform doesn't refresh by its own and the changes are shown just after the user performs some other actions. For example: like + remove the like, follow + unfollow.
Related to: #223
Steps to reproduce

Like someone's note
Remove your like
Most of the time, the like stills appear unless you do some other actions such as going to a profile or entering a note (a screen change action).

Evidence
https://github.com/user-attachments/assets/6fa00573-e5c5-426b-8455-36d1f50c2f85
https://github.com/user-attachments/assets/e39064f1-b690-4ea2-ac87-8fc572cf4cce

Hi! @ugur-eren
Can I hop on this? I'd like to fix it if it's open for contribution.

We've found the root cause of this issue, we're working on it.

Got it. Thanks!


Sidekiq workers

Description of the bug
After upgrading just the opentelemetry-sdk gem from v1.2.0 to v1.2.1, I started seeing a lot of "Calling finish on an ended Span." warnings from Sidekiq. I also found that I only received a single span for all of my Sidekiq workers. After reverting back down to v1.2.0 these problems went away.

Share details about your runtime
Operating system details: Linux, Ubuntu 20.04 LTS
RUBY_ENGINE: "ruby"
RUBY_VERSION: "3.1.1"
RUBY_DESCRIPTION: "ruby 3.1.1p18 (2022-02-18 revision 53f5fc4236) [arm64-darwin21]"

Closing in favor of https://github.com/open-telemetry/opentelemetry-ruby/issues/1495


Move 4-Technical to DB and change folder structure

4-technical is a database which barely depends on the scenario. It should then be moved to cea/db together with the Archetypes and the weather data DBs.
This brings about the opportunity to make the folder structure much more easy to read. here is an example @architecture-building-systems/cea please comment until 27.06.17

possible.

This is the existing one btw:

due to the use of the input locator, making these changes should be
easy-peasy :)
On Thu, May 26, 2016 at 6:08 AM, JIMENOFONSECA<EMAIL_ADDRESS>wrote:

This is the existing one btw:
[image: capture22222]
https://cloud.githubusercontent.com/assets/8973186/15563358/8ba1254a-233a-11e6-9327-91732fd7cb06.PNG
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly or view it on GitHub
https://github.com/architecture-building-systems/CEAforArcGIS/issues/218#issuecomment-221772679

Agreed that it makes no sense to have “technical” as an input here. Question: I was including the SIA Standard-based mobility calculations in the technical folder (in order to generate “embodied”, “operation” and “mobility” results). Should these go in the “db” folder as well then? Won’t this make the tool very Swiss-based?
The general folder structure looks a lot clearer to me now than before, agreed on that too :)
On 26 May 2016, at 06:07, JIMENOFONSECA <notifications<EMAIL_ADDRESS>wrote:
4-technical is a database which barely depends on the scenario. It should then be moved to cea/db together with the Archetypes and the weather data DBs.
This brings about the opportunity to make the folder structure much more easy to read. here is an example @architecture-building-systems/ceahttps://github.com/orgs/architecture-building-systems/teams/cea please comment until 27.06.17
[capturenew]https://cloud.githubusercontent.com/assets/8973186/15563345/704994bc-233a-11e6-961b-e71de6e2902a.PNG
possible.
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly or view it on GitHubhttps://github.com/architecture-building-systems/CEAforArcGIS/issues/218

About the 2000-watt society we can may be put it in a folder called ..db/benchmarks?.
we could also create all these databases for different regions. Perhaps we put later on everything in a folder called db/switzerland/benchmarks
db/switzerland/archetypes
db/switzerland/weather
etc....

@shanshanhsieh @gabriel-happle @amraladdin @shizhongming please comment in the folder structure. today i expect to start implementing the changes

@JIMENOFONSECA The new folder structure does make more sense now!
one question, under the folder "building geometry", are both district.shp and zone.shp providing building geometry information? or is the district.shp containing building geometry information, and zone.shp is indicating the group of building in the district that will be simulated?

@shanshanhsieh i see them both storing building geometry. one of the entire district and one of the zone to be analyzed. but probably we would like to include a way to do zoning in the other areas of the district

This structure makes sense to me.

db/switzerland/etc. sounds good to me
On 26 May 2016, at 11:05, JIMENOFONSECA <notifications<EMAIL_ADDRESS>wrote:
About the 2000-watt society we can may be put it in a folder called ..db/benchmarks?.
we could also create all these databases for different regions. Perhaps we put later on everything in a folder called db/switzerland/benchmarks
db/switzerland/archetypes
db/switzerland/weather
etc....
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly or view it on GitHubhttps://github.com/architecture-building-systems/CEAforArcGIS/issues/218#issuecomment-221817295

can we use the country codes instead?

db/switzerland vs. db/ch/

On Fri, May 27, 2016 at 10:22 AM, martin-mosteiro<EMAIL_ADDRESS>wrote:

db/switzerland/etc. sounds good to me
On 26 May 2016, at 11:05, JIMENOFONSECA<EMAIL_ADDRESS><EMAIL_ADDRESS>wrote:
About the 2000-watt society we can may be put it in a folder called
..db/benchmarks?.
we could also create all these databases for different regions. Perhaps we
put later on everything in a folder called db/switzerland/benchmarks
db/switzerland/archetypes
db/switzerland/weather
etc....
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly or view it on GitHub<
https://github.com/architecture-building-systems/CEAforArcGIS/issues/218#issuecomment-221817295>
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
https://github.com/architecture-building-systems/CEAforArcGIS/issues/218#issuecomment-222088927,
or mute the thread
https://github.com/notifications/unsubscribe/AC1P3KMmXg5qRtv0-a8zqgBL_iXM35mhks5qFqm7gaJpZM4InLk0
.

sure thing

I think the new structure is much nicer, and I agree with Daren to use country codes


Images not get fully processed even if background is filled with base color?

Hi,
im not sure if it s a problem with the lib but im experience some issues where images not get fully „rendered“?
Looks like this: Image
If anyone has a tip why this happens it would be great.

Maybe you didn't flush your buffered Writer, if you're using one?


RuntimeError

I'm using Pytorch CPU-only, on a ubuntu 18 OS.  I'm trying to run ESM-fold. I'm getting the following error:
RuntimeError: Keys 'trunk.structure_module.ipa.linear_kv_points.linear.bias, tru                                                                                nk.structure_module.ipa.linear_q_points.linear.weight, trunk.structure_module.ip                                                                                a.linear_q_points.linear.bias, trunk.structure_module.ipa.linear_kv_points.linea                                                                                r.weight' are missing.
Is this an issue with Pytorch? Or some other dependency?

Hi everyone, first thanks for your work!
I have the same problem as @tonyageall. This error indicates a problem with the structure of the neural network. There is no correspondence between the keys expected, and the keys found by the downloaded model. I try with different models but, I have the same error. How can we fix that? Can anyone help us?
Thanks for your answers.

same question

I also have the same problems ,by use cuda 11.8 torch 2.2.0 openfold 2.0
import esm
import torch
model = esm.pretrained.esmfold_v1()
###################################
RuntimeError: Keys 'trunk.structure_module.ipa.linear_q_points.linear.weight, trunk.structure_module.ipa.linear_kv_points.linear.bias, trunk.structure_module.ipa.linear_q_points.linear.bias, trunk.structure_module.ipa.linear_kv_points.linear.weight' are missing.
Thanks for your answers.

Same issue:
Traceback (most recent call last):
  File "./esm-fold.py", line 5, in <module>
    model = esm.pretrained.esmfold_v1()
  File "/home/ark/miniconda3/envs/esmfold3/lib/python3.7/site-packages/esm/pretrained.py", line 420, in esmfold_v1
    return esm.esmfold.v1.pretrained.esmfold_v1()
  File "/home/ark/miniconda3/envs/esmfold3/lib/python3.7/site-packages/esm/esmfold/v1/pretrained.py", line 54, in esmfold_v1
    return _load_model("esmfold_3B_v1")
  File "/home/ark/miniconda3/envs/esmfold3/lib/python3.7/site-packages/esm/esmfold/v1/pretrained.py", line 29, in _load_model
    raise RuntimeError(f"Keys '{', '.join(missing_essential_keys)}' are missing.")
RuntimeError: Keys 'trunk.structure_module.ipa.linear_q_points.linear.bias, trunk.structure_module.ipa.linear_kv_points.linear.bias, trunk.structure_module.ipa.linear_kv_points.linear.weight, trunk.structure_module.ipa.linear_q_points.linear.weight' are missing.

You may refer to issue: 435 for any specific solutions.


Fetching exchange rates doesn't work

The "exchange rate table publication date" shouldn't be editable (it should be filled automatically). Even if I fill the date, it doesn't convert currencies properly, displaying NaNs.

Even though the exchange rate is displayed correctly, I suspect it isn't saved in the Firebase at all:

On the preview page it's still NaN:

Just confirming the suspicion:


footer improved

@Abhi24iitg please add some images and videos for review.
Also specify the issue you are solving


They need to be a standard height, I think. Can you do that?

I made it (and all slideshow images) have a height of 550px.


COMPAT: remove NaT comparison warnings with numpy >= 1.11

cc @sinhrks

[Current coverage][cc-pull] is 83.82%

Merging [#12969][cc-pull] into [master][cc-base-branch] will decrease coverage by -0.00%

@@           master   #12969   diff @@
======================================
  Files         136      136
  Lines       49642    49663    +21
  Methods         0        0
  Branches        0        0
======================================
+ Hits        41617    41632    +15
- Misses       8025     8031     +6
  Partials        0        0

[][cc-pull]

Powered by Codecov. Last updated by 5b766d2
[cc-base-branch]: https://codecov.io/gh/pydata/pandas/branch/master?src=pr
[cc-pull]: https://codecov.io/gh/pydata/pandas/pull/12969?src=pr


Add which

#7

Could you merge master again?  The build failure in CI is my fault and is an issue in master that I just fixed

Other than those comments this looks good to me

Done

The PR needs another rebase on master to fix the build failure on GHC 7.6.3 (which was due to need being unavailable).

Have you tested that the function works correctly in all cases? If yes, it is good to merge.

To be honest I didn't do much testing, just poked at it with a stick in ghci. It's pretty simple, though: tokenize $PATH, see if there is an executable thing there. I'm sure it fails in weird corner cases, like if there's a directory in $PATH that is -x or something.
Is there any specific test you'd like to see?

Note that this is basically the same behavior as the Unix which so it should be well-behaved.  See: https://opensource.apple.com/source/shell_cmds/shell_cmds-118/which/which.c

Alright, this looks great to me!  I'll go ahead and squash and merge.  Thanks for the contribution! :)


Adding Swag Submission Page

Feature or Problem

Related Issues

Release Information

Consumer Impact

Testing

Unit Test(s)

Acceptance or Integration

Manual Verification

@brooksmtownsend @LiamRandall I've adjusted some of the language on the form, added a t-shirt size and moved the page to /swag. I also added a link in the footer instead of in the docs sidebar.
https://deploy-preview-550--dreamy-golick-5f201e.netlify.app/swag


How to use Mixins?

First Check

[X] I added a very descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the SQLModel documentation, with the integrated search.
[X] I already searched in Google "How to X in SQLModel" and didn't find any information.
[X] I already read and followed all the tutorial in the docs and didn't find an answer.
[X] I already checked if it is not related to SQLModel but to Pydantic.
[X] I already checked if it is not related to SQLModel but to SQLAlchemy.

Commit to Help

[X] I commit to help with one of those options 👆

Example Code
class TimeStampMixin:
    """Provides last created/modified timestamps"""

    created_at: Optional[datetime] = Field(
        sa_column=Column(
            DateTime,
            default=datetime.utcnow,
            nullable=False,
        )
    )

    updated_at: Optional[datetime] = Field(
        sa_column=Column(
            DateTime,
            default=datetime.utcnow,
            onupdate=datetime.utcnow,
        )
    )

class MyModel(SQLModel, TimestampMixin, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    name: str

Description

Utilize mixins that don't necessarily need to subclass SQLModel

Perhaps something that resembles @declarative_mixin from SQLAlchemy?

The above example errors with:
AttributeError: type object 'TimeStampMixin' has no attribute '__config__'

Operating System
Linux
Operating System Details
No response
SQLModel Version
0.0.4
Python Version
3.9.0
Additional Context
No response

Try this out
from datetime import datetime
from typing import Optional

from pydantic import BaseModel
from sqlmodel import SQLModel, Column, DateTime, Field, Session, create_engine, select

class TimeStampMixin(BaseModel):
    created_at: Optional[datetime] = Field(
        sa_column=Column(
            DateTime,
            default=datetime.utcnow,
            nullable=False,
        )
    )

    updated_at: Optional[datetime] = Field(
        sa_column=Column(
            DateTime,
            default=datetime.utcnow,
            onupdate=datetime.utcnow,
        )
    )

class MyModel(SQLModel, TimeStampMixin, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    name: str

engine = create_engine("sqlite:///database.db")
SQLModel.metadata.create_all(engine)

with Session(bind=engine) as session:
    session.add(MyModel(name="A Name"))
    session.commit()

Try this out
from datetime import datetime
from typing import Optional

from pydantic import BaseModel
from sqlmodel import SQLModel, Column, DateTime, Field, Session, create_engine, select

class TimeStampMixin(BaseModel):
    created_at: Optional[datetime] = Field(
        sa_column=Column(
            DateTime,
            default=datetime.utcnow,
            nullable=False,
        )
    )

    updated_at: Optional[datetime] = Field(
        sa_column=Column(
            DateTime,
            default=datetime.utcnow,
            onupdate=datetime.utcnow,
        )
    )

class MyModel(SQLModel, TimeStampMixin, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    name: str

engine = create_engine("sqlite:///database.db")
SQLModel.metadata.create_all(engine)

with Session(bind=engine) as session:
    session.add(MyModel(name="A Name"))
    session.commit()

That works! This also enables a Generic[_T] base class

Why do we need to extend the Mixins from pydantic BaseModel and not SQLModel? Or is both possible? Also, with SQLModel, the definition would be simpler:
class TimeStampMixin(SQLModel):     created_at: Optional[datetime] = Field(default=datetime.utcnow(), nullable=False)     updated_at: Optional[datetime] = Field(default_factory=datetime.utcnow, nullable=False)


End of Candidates

As discussed in #118, closes #118.

As far as I can see, this requires no updates in the saltyrtc-task-webrtc-js library. The Java implementation just needs to update some annotations.
Note by the way that the candidate gathered by WebRTC is null, while we are sending {candidate: null, sdpMid: null, sdpMLineIndex: null}. That's intended, right? It means that the event.candidate field of the browser event can't just be passed on to SaltyRTC directly, instead the event.candidate field needs to be inspected to generate the empty candidate data for SaltyRTC in case the field is null.

Oh, the whole thing is null, you're right. I'll update the PR. I did not intend to make this incompatible with the API.
Could you rephrase your last sentence? I don't understand what would be error prone.

Edit: I think I understand now. You mean, that there could be further fields, right?

Exactly :) Yes, although it's convenient, it's probably better to always use explicit field assignment.


Update Microsoft.Extensions.Logging.Console to 5.0.0

I have a NuGet dependency that uses Microsoft.Extensions.Logging.Abstractions 5.0.0, scripts using it won't compile due to Dotnet.Script's version differing. 5.0.0 is the latest stable version of the logging packages.

That makes sense. This second commit conditionally uses 5.0.0 for net5.0 and uses 3.1.12 (update from 3.1.6) for the rest.

I am not completely sure if this is actually necessary. The Microsoft.Extensions.Logging.Abstractionspackage is not a SDK dependency and the 5.0 version is compatible with netstandard2.0 so there should be no problem using the 5.0 version with a netcoreapp3.1 app. Also we still support netcoreapp2.1 which also can consume this package. It should also be noted that the Microsoft.Extensions.Logging.Abstractions is only used in the the dotnet-script executable and is not shared with OmniSharp. Just want to make sure that this is actually needed before we start to add conditional package references. It is sort of a potential rabbithole 😀

I think this update will resolve the Microsoft.Extensions conflict problem when using the net5.0 package.
It can be reproduced with this short script, but EFCore 5.0 is currently unavailable.
(I know 3.1 is available, but 5.0 is preferable.)
#r "nuget: Microsoft.EntityFrameworkCore, 5.0.5"
class MyDbContext : Microsoft.EntityFrameworkCore.DbContext{}
new MyDbContext();

I don't know how it behaves when combined with older runtimes, but I think it would be useful to update it if it doesn't cause any problems.

Fixed by #614


fix: ZeBadgeViewModelTest not compiling

Summary
Fix compilation error: add missing weatherService argument to ZeBadgeViewModel constructor in ZeBadgeViewModelTest.
How It Was Tested
The unit tests now build successfully.

Closing this PR as it is now a duplicate of #372


Unable to Sync Data from Edgedb to Snowflake via Fivetran

EdgeDB Version: 3.2
EdgeDB CLI Version: EdgeDB CLI 3.4.0+160d07d

Current Behavior
I'm experiencing an issue while trying to synchronize data from Edgedb (version 3.2) to Snowflake using Fivetran.
Here are the steps I've taken and the error I'm encountering:

I've connected to Edgedb instance deployed on GKE. Backend is a Cloud SQL PostgreSQL instance with version 15.
I migrated the data from an Edgedb version 2.7 instance (with PostgreSQL 14 backend) to the Edgedb version 3.2 instance (with PostgreSQL 15 backend).
I used the psql CLI to connect to the Edgedb instance and execute read-only SQL queries successfully:

psql -h host -p 5656 -U postgres -d dbname

I want to synchronize this data to Snowflake using Fivetran.

Error Message
Fivetran is displaying an error message that reads: Unable to determine the availability of XMIN extensions
Steps to Reproduce

Deploy Edgedb version 3.2 on GKE with Cloud SQL PostgreSQL instance version 15 as the backend.
Migrate data from an Edgedb version 2.7 instance to the Edgedb version 3.2 instance.
Set up Fivetran to synchronize data from Edgedb to Snowflake.
Observe the Unable to determine the availability of XMIN extensions error..

Additional Context
I also attempted changing the PostgreSQL version to 14 and creating a default example database to troubleshoot the issue. However, the same error persisted.
Please let me know if there are any further details or logs I can provide to help resolve this issue.

Attachments:

Edgedb Deployment Configuration (YAML)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: edgedb3
  labels:
    app: edgedb3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: edgedb3
  template:
    metadata:
      labels:
        app: edgedb3
    spec:
      containers:

        - name: edgedb3-container
          image: edgedb/edgedb:3.2
          ports:
            - containerPort: 5656

          volumeMounts:
          - name: edgedb-secret-volume
            readOnly: true
            mountPath: "/etc/secret-volume"

          env:
            - name: EDGEDB_SERVER_INSTANCE_NAME
              value: edgedb3-server
            - name: EDGEDB_SERVER_TLS_CERT_MODE
              value: require_file
            - name: EDGEDB_SERVER_PASSWORD
              value: password
            - name: EDGEDB_SERVER_BACKEND_DSN
              value: postgresql://user:password@<IP_ADDRESS>:5432
            - name: EDGEDB_SERVER_TLS_CERT_FILE
              value: /etc/secret-volume/tls-cert
            - name: EDGEDB_SERVER_TLS_KEY_FILE
              value: /etc/secret-volume/tls-key
            - name: EDGEDB_SERVER_USER
              value: postgres
            - name: EDGEDB_SERVER_ADMIN_UI
              value: enabled
            - name: EDGEDB_SERVER_BINARY_ENDPOINT_SECURITY
              value: optional

        - name: cloud-sql-proxy
          image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.1.0
          args:
            - "--port=5432"
            - "gcpProject:zone:instance"
            - "--credentials-file=/secrets/cloudsql/credentials.json"
          securityContext:
            runAsNonRoot: true
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true

      volumes:

        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials

        - name: edgedb-secret-volume
          secret:
            secretName: tls-secret

It looks like Fivetran is verifying that it is able to use XMIN system columns, and that verification fails. This will probably be because it is looking at some internal Postgres APIs (or even an API only available in Google Cloud's Postgres).
XMIN system columns should be available by default in a plain Postgres instance, but I don't know if that is true for the Google Cloud's Postgres.
Would it be possible to use plain Postgres Fivetran connection?
If this does not work, I'd have to replicate the error myself, enable appropriate debug logging and see what SQL queries are failing. If you can find this in Fivetran's console yourself, that would greatly speed-up my process.

Thanks @aljazerzen. In our initial testing phase, we evaluated a standard Postgres connection through Fivetran. However, it prompted us to consider utilising Google Cloud PostgreSQL instead. I was not able to found the exact reason for this.

Hi @aljazerzen, I tried using a Postgres Docker Container postgres:15 instead of Cloud SQL.
Fivetran is able to connect (with a warning: XMIN extensions not enabled) and can show the schema, but it's not able to fetch data and sync it to Snowflake.
Below are the attached screenshots for Fivetran logs, sample data, and the list of installed extensions in Edgedb.
Warning during Fivetran Connect :

Data Sync logs

I cannot find any documentation in "XMIN extensions" that the Fivetran error is talking about.
Wait, have you tried using Postgres 14? I don't think Postgres 15 is officially supported by EdgeDB 3.x.
(regardless, this is probably a bug with our emulation of Postgres APIs)

Hi @aljazerzen  tried using Postgres 14, getting the same warning: XMIN extensions not enabled and not able to fetch data.

Hi @aljazerzen any update on this?

This is not my top priority right now. As there (probably) won't be a minor or major release in next two weeks, I'll postpone this issue until then.

I deployed a GKE cluster in an effort to reproduce this issue.
I don't think the "Unable to determine the availability of XMIN extension" is problematic. XMIN columns are available, regardless of the Fivetran's check.

But I do get another error before I was able to inspect the schema:
Type: com.fivetran.setup_test_runner.exceptions.CustomServiceException
Details: com.fivetran.setup_test_runner.exceptions.CustomServiceException:
  Request has been processed with error:
    Unable to get standard config for integration studies_stop:
      Failed to retrieve schema information for source.

fileName:StandardConfigExecutorWithMetrics.java
lineNumber:43
methodName:getAndSave
className:com.fivetran.setup_test_runner.standard_config.StandardConfigExecutorWithMetrics

I see that you did manage to sync the schema, so this is a bit confusing. Where did it stop working for you? Were you able to select which tables to sync? Was the initial sync successful?


possible memory leak with celery processes

Hey, we are currently running 176bac890 using our private fork of https://github.com/shoonoise/cabot-docker and we have an issue where cabot eats more and more memory after running for some time.
I found a few issues for "celery memory leak", but all the issues are very old and did not contain enough information for me to take any action.
I don't want to set up a cron job to restart cabot regularly.
Please let me know if you have any suggestions or if I can provide any useful information.

After reading again, this might be related: https://github.com/celery/celery/issues/2927

Could you bump the celery dependency to at least 3.1.21? Newest is currently 3.1.23. This contains the above mentioned fix, which might help us.

Happy to merge a PR as long as tests pass ;)

There you go :-)

I have just face with issue. Please turn off debug on settings.py file of cabot. DEBUG=False to run produciton

To clarify how to switch off debug - just make sure that env variable DEBUG is not set... The standard production.env.example doesn't set this key.

I fix in cabot/settings.py. DEBUG = False. Another way
in settings.py have source likes as
TEMPLATE_DEBUG = DEBUG = os.environ.get("DEBUG", False)
It will set debug option using DEBUG environment.

@lamhaison, setting the env variable DEBUG=False will not set the Django variable DEBUG to a falsy value.

I think you have mistake. because when you run python manager.py shell on cabot source, It will import variable by execute code in settings.py of django cabot. Howerver, in this you example, it only run python console, do not load anything define on then settings.py file of the cabot app.


ZF1 EOL and future

Since ZF1 is EOL and there are many projects still using it i would appreciate if there would still be a way to merge bugfixes and security updates?
perhaps you could release ZF1 as OS or change license to not be your proprietary license?
Would you let someone external handle it? or would you permit branching off a free version of ZF1 to someone external who could handle merges and perhaps link to that?
Why not migrate to ZF2+? They are incomplete so that many missing CORE functions from ZF1 and the overcomplicated MVC / CI requirements make it financially not feasible to migrate certain open source and business applications.

@tibor78
The "Zend Framework" is open source.
See the license here on GitHub, on the ZF website or in any of the project files.


Help with bitcoin cash on pybitcoin

Hi fyookball,
I'm sure this is not the best way to contact you. Not sure of a better way though. Not directly related to electron-cash.
I wanted to add Bitcoin Cash support to walkingliberty. and decided the best approach would be to add in electrum support, then I could support both electrum and electron-cash.
I did that, and it's working. You can see the latest commits for walkingliberty. I also forked pybitcoin and added support for electrum there.
I was able to get transactions to go to electron-cash, but they were rejected. Sure enough, didn't have the SIGHASH_FORK 0x40 bit set. So I have a branch where I'm trying to do that. It is working in the sense that the error message changes, now it seems like the signature is just bad although the deserialze call shows a transaction that at first glance, looks good to me. I'm in way over my head, most of this is all new to me. I've been comparing electrum and electron-cash as a basis for updating the signing functions in pybitcoin.
Here's a transaction as generated with this setup:
0100000001edc990f08c1ddd5102dfc7f4af32e7927a14fd8d0bc9996725f38ede549b91f8010000008a47304402203c8cfb468757abae2d3563fc37048c6213096d894f43e4b9f0ddb7c6851471f202204c6ac9a1e16aa80234a3aec208beef80977c38832efd593d849d4a8312eb5004414104e1c33577152151109fe5e02c600f386fda865594c1b989ce0fcfbc0fb0b144a2c8fd98053fd86a4ccc7e0dcbcdd6a15c38eb32a27b3653c54f78ba00ffd71d19ffffffff0210270000000000001976a9141a586975175967c83e2b90422bcd8af9a623156288ac25cc0d00000000001976a91470fb690a177e8721098d90284ab3c3304ee19cd588ac00000000

I compared it with an accepted transaction and I did notice that my pub_key and x_pub_key is a lot longer. Maybe compressed vs uncompressed public key? Not sure if it matters, though.
You can see my pybitcoin changes here, where I am setting SIGHASH and such: https://github.com/teran-mckinney/pybitcoin/tree/bitcoincash
Would greatly appreciate any help you can offer. Happy to move this over to my pybitcoin fork's issues section if that works better for you. Not really sure who else to ask at this point.
Thank you,
Teran

Ah, I did not know that. I think pybitcoin hasn't implemented bip143 at all.
https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki
Will try to understand that more and see if there's another Python library that might be easier to port over. If it supports bip143, can I just alter the SIGHASH and be set?

essentially yes.  check the early commits on electron cash in late july.  you should see the relevant code


Run unstable tests in separate environments

*Issue #, if available:*764
Description of changes:
This commit splits up the troublesome tests of models into two parts, one stable and one unstable part.
The unstable part of the tests are moved to a new file and are run during a seperate test execution.
The unstable parts of the tests are moved to the folder:
/tests/model/unstable_tests/test_name.py
for example the troublesome parts of the file:
tests/model/deepstate/test_model.py
is moved to:
/tests/model/unstable_tests/test_deepstate.py
In order to run these tests after a pull request ive modified the file:
.github/workflows/tests.yml
More specifically the change made should allow for the tests to run in parallel. However since this file is run on GitHub itself I have not been able to try this locally. I did however try partial changes on a private github repo.
The change
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Hey @arangatang the style checks are failing: can you make sure to run
pip install -r requirements/requirements-setup.txt
./dev_setup.sh

This will install black and mypy, and set some pre-commit hooks which will make sure you don't commit type-unsafe or not-properly-styled code. To fix the style in this PR, you can just run
black src test

and commit & push the changes.

Will do, however I am going to redo the commit with another approach before that though as I need to add regression tests for the accuracy instead of quarantining the flaky values.

I believe we can close this pull request as the proposed solution is to be replaced by #379


[BUG] Extra objects in colab cell outputs

I created a notebook in colab (the notebook was running fiftyone==0.7.2) and then downloaded it to my machine and opened it with Jupyter.
In cells that launch App instances (which I may have run multiple times in Colab), I see some extra <IPython.core.display.Javascript object> outputs in Jupyter that I don't see in Colab.
Ideally these would not appear. And if there's underlying data stashed in the .ipynb but not being shown, that would ideally not be there either. There was one cell that had two App images in its output, for example.

Oops this is a duplicate of https://github.com/voxel51/fiftyone/issues/745. Closing this one.


Updating get started ToC

Added by Keerthi on behalf of Nitin

Docs Build status updates of commit dbd18ee:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

Articles/TOC.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Is there a way to open slides in the browser?

It seems that podium uses ramark.js under the hood (looking at the example deck) but if I open the slides.html in the browser it can't find anything from the /resources/ directory (which I guess is packaged with the Podium.app).
Is there a way to use podium as both the Desktop presentation tool but also be able to export as a webpage for hosting slides online?
Thanks!

So - yes, it is possible; however, there isn't currently an easy way to do it.
As you've noted, a Podium slidedeck uses remark.js under the hood; so if you follow the boilerplate instructions on the Remark homepage you can get the content displayed in your browser. For the moment, at least, you'll need to roll that webpage yourself.
The slides.html file in the repo is a first attempt to get a simple HTML file that can load the markdown content and inject it into the webpage; however, it obviously depends on having a /resources path available. The eventual goal is to provide simple instructions for converting a repository full of Podium slide decks into a website of presentations.
The contents of the resources folder should be the same as this folder

I see, thanks for the explanation!
Btw, would be cool to integrate https://github.com/astefanutti/decktape into podium to allow PDF export.

Agreed - PDF export (whether using decktape, or something else) is another thing on my wish list .

Wrt. viewing a presentation as Webpage.
The following works:

Add a resources folder to your .podium folder
Add style.css and remark.js to that folder
Use the slides.html from the sample
Run python3 -m http.server in the directory that contains slides.html
Open http://localhost:8000/slides.html in your browser

PDF export does not work then but it got me up and running quickly.

Thanks for the walthrough, @edthamm! It worked for me. The only catch for me was that I needed to add a little additional styling to get the slides to render at the full page width:
@media print {
    .remark-slide-scaler {
        position: static;
        margin: 0 auto;
    }
}

And after printing the PDF from Chrome, I used Acrobat to crop off the whitespace from the bottom of the page.


Add Fixed Classifier

Added a classification function train_classifier to train a model to classify documents into a set of predefined labels (as opposed to the more open-ended topic modeling in build_clusters!). You can either provide a small set of labeled documents to train the model (that are in the index), or just specify the num_samples and the LLM model will generate its own training data based on the labels and labels_description provided.
Added a new template TextWriterFromLabel to generate synthetic documents for any given label (=topic).
Updated to use PromptingTools 0.15.

Codecov Report
Attention: Patch coverage is 1.49254% with 66 lines in your changes are missing coverage. Please review.

Project coverage is 79.26%. Comparing base (3efddbb) to head (4e4668f).

Files
Patch %
Lines

src/classification.jl
0.00%
49 Missing :warning:

src/utils.jl
0.00%
10 Missing :warning:

src/types.jl
0.00%
7 Missing :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##             main      #14       +/-   ##
===========================================
- Coverage   91.74%   79.26%   -12.49%
===========================================
  Files           8        9        +1
  Lines         424      487       +63
===========================================
- Hits          389      386        -3
- Misses         35      101       +66

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Duplicate definitions if multiple codes mapped to same CUI

Statement of problem
In the UBKG, Definition nodes are linked to Concept nodes. If an ingested node has an associated node_definition in the nodes_metadata file, the generation framework script will create a Definition node with DEF=node_label, associated with the CUI for the Concept associated with the code.
It is possible for multiple codes to map to the same CUI, even from the same SAB. This is common for CL nodes, for example: both CL:0000669 and CL:0002503 map to CUI C0598800.
Because definitions are mapped to concepts and not codes, it is not possible to resolve a definition to a code for the case in which multiple codes from the same SAB map to the concept. If, for example, we try to obtain the definition of CL:0000669 from a query, we will get definitions for both CL:0000669 and CL:0002503.
Workaround
The current workaround is to collect definitions and then take the first one in the list--e.g.,

OPTIONAL MATCH (cGene:Code)<-[:CODE]-(pGene:Concept)-[:inverse_HRA_has_marker_component]->(pCL:Concept)-[:CODE]->(cCL:Code),(pCL:Concept)-[:DEF]->(dCL:Definition) WHERE cGene.CodeID='HGNC:60' AND cGene.SAB='HGNC' AND cCL.SAB='CL' AND dCL.SAB='CL' RETURN toInteger(cGene.CODE) AS hgnc_id,'cell_types_definition' as ret_key, cCL.CodeID + '|'+ COLLECT(DISTINCT dCL.DEF)[0]  as ret_value

Proposed solution
The definitive solution would be to allow for Definition nodes to be associated with either Code nodes or Concept nodes, similarly to how Term nodes are treated.

Unintended consequence of earlier bug fix
This has been identified as a consequence of an earlier code change to address self-references.
There is a continuing difficulty with the logic that assigns a CUI to a code.
The script orders a set of potential CUIs for each code by preference. The set members are:

The UMLS CUIs for cross-references that existed previously in the ontology CSVs. There may be more than one cross-reference.
The UMLS CUI for the node that existed previously in the ontology CSVs
The non-UMLS CUIs for a cross-reference that existed previously in the ontology CSVs
A new CUI for a node with value in format SAB:CODE CUI. (This was originally base64-encoded.)

The script then selects the first CUI from the set, assuming that this has the highest preference.
For some cases, this results in multiple assignments of the same CUI to codes.
In an earlier version of the script, the logic looped through the entire node_metadata and assigned a unique CUI from the list. The original logic is below. It had a number of problems, as described in issue 9. The main problem was that it required significant resources when dealing with large datasets because it relied on a Python loop and iterrows.
The replacement of this logic with the simpler selection of the first in the list addressed all known issues except for multiple code-CUI assignments.
node_idCUIs = []
nmCUI = []
for index, rows in node_metadata.iterrows():
    addedone = False
    for x in rows.cuis:
        if ((x in node_idCUIs) | (addedone == True)):
            dummy = 0
        else:
            nmCUI.append(x)
            node_idCUIs.append(x)
            addedone = True
    # JAS 17 oct 2022
    # For the edge case in which multiple nodes have the same cross-reference AND the
    # cross-reference maps to a single CUI (the case for around 20 nodes in MP--e.g., all
    # those that share CL 0000959 as a cross-reference), only one of the nodes will be
    # assigned the CUI of the cross-reference. The rest of the nodes will lose the
    # cross-reference.
    # The following block keeps the nmCUI list the same size as node_metadata['CUI'].
    if addedone == False:
        nmCUI.append('')
        node_idCUIs.append('')

node_metadata['CUI'] = nmCUI

Revision to code
The logic has been revised to reflect a compromise between the original and newer assignment logic.
The original logic looped through the cui list for every node to select the first CUI that had not already been assigned.
The newer logic selected the first CUI in the cuilist for every node.
The latest logic:

Selects the first CUI in the list by default. This is sufficient for most codes.
Identifies CUIs that are assigned to multiple codes. This will be a subset of the codes in the node metadata.
Applies the original selection logic for each set of codes that are assigned to the same CUI.

Regression testing
Test 1 - CL:0000669 and CL:0002503
The CL codes each map to a FMA code that maps to UMLS CUI C0598800.
Prior to the revision of assignment logic, the resulting graph had both CL codes mapping to CUI C0598800, which resulted in the duplicate definition problem.
After the revision, CL 0002503 is mapped to CUI C0598800 and CL:0000669 is mapped to the newly minted CUI "CL:0000669 CUI". This corrects the duplicate definition problem:
match (t:Term)-[r:PT]-(c:Code)<-[:CODE]-(p:Concept)-[:DEF]->(d:Definition) where c.CodeID in ['CL:0000669','CL:0002503'] AND c.SAB='CL' AND d.SAB='CL'  AND r.CUI=p.CUI return c.CodeID,p.CUI,d.DEF,t.name

c.CodeID
p.CUI
d.DEF
t.name

1
"CL:0000669"
"CL:0000669 CUI"
"An elongated, contractile cell found wrapped about precapillary arterioles outside the basement membrane. Pericytes are present in capillaries where proper adventitia and muscle layer are missing (thus distingushing this cell type from adventitial cells). They are relatively undifferentiated and may become fibroblasts, macrophages, or smooth muscle cells."

2
"CL:0002503"
"C0598800"
"A cell of the adventitial layer of ductal structures such as the uterer, defent duct, biliary duct, etc"

Test 2 - AZ:0000004, AZ:0000025, and AZ:0000080
This was a test case from the earlier bug. The three AZ codes share a cross-reference with CL:0000236 (B cell), which itself cross-references to 2 CUIs (C1267837 and C0004561). The issue was that AZ:0000080 lost its cross-reference in the CUI assignment logic, which I saw as a problem.
The revised logic distributes CUIs among the 3 AZ codes as before. Because later logic assigns all CUIs in the cuilist to the code, AZ:000080 actually does map to the 2 CUIs; however, only the "preferred CUI" is assigned a term, so it is possible to filter out the other CUIs.
optional match (t:Term)<-[r:PT]-(c:Code)<-[:CODE]-(p:Concept) where c.CodeID IN ['AZ:0000004','AZ:0000025','AZ:0000080'] and c.SAB='AZ' and r.CUI=p.CUI  return c.CodeID, p.CUI, t.name

c.CodeID
p.CUI
t.name

1
"AZ:0000004"
"C1267837"

2
"AZ:0000025"
"C0004561"

3
"AZ:0000080"
"AZ:0000080 CUI"

Test 3 - CHLO and GTEXEQTL
This test was to determine whether the revised logic worked for large datasets. The ingestion of the 2 SABs completed within the expected time.

https://github.com/x-atlas-consortia/ubkg-etl/pull/93


TextPlain in: body does not permit non-string values

The following did not compile, and probably still does not compile. Opening a ticket to track it.
      parameters:
        - name: epoch_seconds
          in: body
          required: true
          schema:
            type: integer
            format: int64

This doesn't seem to be an issue anymore - see #714


Storage meta-package

Fix https://github.com/Azure/azure-sdk-for-python/issues/8044
Should wait for https://github.com/Azure/azure-sdk-for-python/pull/8024 before merging, since this PR contains a few engsys adaptation for meta-packages

(message created by the CI based on PR content)

Will resurrect the PR if necessary, but it seems we're not doing it.


coqPackages.coq-extensible-records: support for coq 8.9 and 8.10

Motivation for this change
coq-extensible-records can be made to work for recent versions of Coq. Sadly, upstream is not responding to pull requests. This provides support for those Coq versions in the meantime.
Things done

Built on macOS, tested with Coq versions 8.8, 8.9, and 8.10

@GrahamcOfBorg build coqPackages.coq-extensible-records

Thanks.
Maybe you would like to contribute to the maintenance of this library as part of the Coq community effort.

Ah, good to know, thanks!


Update turbo

Adds support for a few tweaks I made to the Turbo API.
This doesn't compile as-is because CodeEditorWindow.cpp contains a reference to compiler::TokenKind::kUntil, which was removed in 9e3c34d283144480bbcb48d288e428624be3e1e3.

Whoops, good catch! I must have a mistake in my Makefile dependencies; it didn't rebuild CodeEditorWindow.cpp after I removed kUntil from the enum and thus I didn't notice the problem. I'll merge and then fix the build issue on master. Thank you!


grub: 2.0.4 -> 2.0.6-rc1

Motivation for this change
Release
https://lists.gnu.org/archive/html/grub-devel/2021-03/msg00219.html
Security Issues
https://lists.gnu.org/archive/html/grub-devel/2021-03/msg00007.html:
*******************************************************************************

CVE-2020-14372 grub2: The acpi command allows privileged user to load crafted
               ACPI tables when Secure Boot is enabled
CWE-184
7.5/CVSS:3.1/AV:L/AC:H/PR:H/UI:N/S:C/C:H/I:H/A:H

GRUB2 enables the use of the command acpi even when Secure Boot is signaled by
the firmware. An attacker with local root privileges to can drop a small SSDT
in /boot/efi and modify grub.cfg to instruct grub to load said SSDT. The SSDT
then gets run by the kernel and it overwrites the kernel lock down configuration
enabling the attacker to load unsigned kernel modules and kexec unsigned code.

Reported-by: Máté Kukri

*******************************************************************************

CVE-2020-25632 grub2: Use-after-free in rmmod command
CWE-416
7.5/CVSS:3.1/AV:L/AC:H/PR:H/UI:N/S:C/C:H/I:H/A:H

The rmmod implementation for GRUB2 is flawed, allowing an attacker to unload
a module used as dependency without checking if any other dependent module is
still loaded. This leads to an use-after-free scenario possibly allowing an
attacker to execute arbitrary code and by-pass Secure Boot protections.

Reported-by: Chris Coulson (Canonical)

*******************************************************************************

CVE-2020-25647 grub2: Out-of-bound write in grub_usb_device_initialize()
CWE-787
6.9/CVSS:3.1/AV:P/AC:H/PR:H/UI:N/S:C/C:H/I:H/A:H

grub_usb_device_initialize() is called to handle USB device initialization. It
reads out the descriptors it needs from the USB device and uses that data to
fill in some USB data structures. grub_usb_device_initialize() performs very
little bounds checking and simply assumes the USB device provides sane values.
This behavior can trigger memory corruption. If properly exploited, this would
lead to arbitrary code execution allowing the attacker to by-pass Secure Boot
mechanism.

Reported-by: Joseph Tartaro (IOActive) and Ilja van Sprundel (IOActive)

*******************************************************************************

CVE-2020-27749 grub2: Stack buffer overflow in grub_parser_split_cmdline
CWE-121
7.5/CVSS:3.1/AV:L/AC:H/PR:H/UI:N/S:C/C:H/I:H/A:H

grub_parser_split_cmdline() expands variable names present in the supplied
command line in to their corresponding variable contents and uses a 1kB stack
buffer for temporary storage without sufficient bounds checking. If the
function is called with a command line that references a variable with a
sufficiently large payload, it is possible to overflow the stack buffer,
corrupt the stack frame and control execution. An attacker may use this to
circumvent Secure Boot protections.

Reported-by: Chris Coulson (Canonical)

*******************************************************************************

CVE-2020-27779 grub2: The cutmem command allows privileged user to remove
               memory regions when Secure Boot is enabled
CWE-285
7.5/CVSS:3.1/AV:L/AC:H/PR:H/UI:N/S:C/C:H/I:H/A:H

The GRUB2's cutmem command does not honor Secure Boot locking. This allows an
privileged attacker to remove address ranges from memory creating an
opportunity to circumvent Secure Boot protections after proper triage about
grub's memory layout.

Reported-by: Teddy Reed

*******************************************************************************

CVE-2021-3418 - grub2: GRUB 2.05 reintroduced CVE-2020-15705
CWE-281
6.4/CVSS:3.1/AV:L/AC:H/PR:H/UI:N/S:U/C:H/I:H/A:H

The GRUB2 upstream reintroduced the CVE-2020-15705. This refers to a distro
specific flaw which made upstream in the mentioned version.

If certificates that signed GRUB2 are installed into db, GRUB2 can be booted
directly. It will then boot any kernel without signature validation. The booted
kernel will think it was booted in Secure Boot mode and will implement lock
down, yet it could have been tampered.

This flaw only affects upstream and distributions using the shim_lock verifier.

Reported-by: Dimitri John Ledkov (Canonical)

*******************************************************************************

CVE-2021-20225 grub2: Heap out-of-bounds write in short form option parser
CWE-787
7.5/CVSS:3.1/AV:L/AC:H/PR:H/UI:N/S:C/C:H/I:H/A:H

The option parser in GRUB2 allows an attacker to write past the end of
a heap-allocated buffer by calling certain commands with a large number
of specific short forms of options.

Reported-by: Daniel Axtens (IBM)

*******************************************************************************

CVE-2021-20233 grub2: Heap out-of-bound write due to mis-calculation of
               space required for quoting
CWE-787
7.5/CVSS:3.1/AV:L/AC:H/PR:H/UI:N/S:C/C:H/I:H/A:H

There's a flaw on GRUB2 menu rendering code setparam_prefix() in the menu
rendering code performs a length calculation on the assumption that expressing
a quoted single quote will require 3 characters, while it actually requires
4 characters. This allow an attacker to corrupt memory by one byte for each
quote in the input.

Reported-by: Daniel Axtens (IBM)

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 115310 at ad51fef9 run on aarch64-linux 1

2 packages failed to build:

grub2_pvgrub_image
grub2_xen

3 packages built:

grub2 (grub2_full ,grub2_light)
grub2_efi
woeusb

2 suggestions:

warning: maintainers-missing
Package does not have a maintainer. Consider adding yourself?
Near pkgs/tools/misc/grub/2.0x.nix:137:3:
    |
137 |   meta = with lib; {
    |   ^

warning: missing-patch-comment
Please add a comment on the line above, explaining the purpose of this patch.
Near pkgs/tools/misc/grub/2.0x.nix:57:5:
   |
57 |     ./fix-bash-completion.patch
   |     ^

Review per https://nixos.org/manual/nixpkgs/stable/#reviewing-contributions-package-updates
Reviewed points

[x] package name fits guidelines
[ ] package version fits guidelines
[x] package build on x86-64
[x] executables tested on x86-64
[ ] all depending packages build

More:

[x] booted x86-64 mbr/bios system
[x] booted x86-64 gpt/efi system
[x] checked package license: still GPL3+ according to source repo

Possible improvements

Naming guidelines require package name and version to be grub-unstable and 2021-01-11 (date of upstream commit).  I'm not sure if that is a good idea.  However, this grub greets the user as "GNU GRUB  version 2.05", so it should probably not be "2.04" in the effective package name.  Maybe use "2.05pre-2021-01-11" or something similar?

Comments

Tested inside VirtualBox.

Relevant part of NixOS configuration for tests:
{
  boot.loader.grub.efiSupport = true;
  boot.loader.grub.device = "nodev";
  boot.loader.grub.enable = true;
  boot.loader.timeout = 4;
  boot.loader.efi.canTouchEfiVariables = false;
  boot.loader.grub.version = 2;
  boot.loader.grub.splashImage = null;
  boot.loader.grub.font = null;
  boot.loader.grub.efiInstallAsRemovable = true;
  boot.loader.grub.extraConfig = ''
    set color_normal=white/black
    set color_highlight=yellow/dark-gray
  '';
}

Likewise on mbr system, with necessary changes.

Tested after cherry-picking the pull requests's only commit ad51fef929583a43a5a3d791eba98d79c3a74039 on top of current nixos-unstable, that is 102eb68ceecbbd32ab1906a53ef5a7269dc9794a.

nixos-rebuild boot yields this warning message when installing the bootloader
  updating GRUB 2 menu...
  installing the GRUB 2 EFI boot loader into /boot...
  Installing for x86_64-efi platform.
  /nix/store/jp4shpma6ia2npq8wcicdkvvzif64ryd-grub-2.04-340-g8fcfd1e0f/sbin/grub-install: warning: cannot open directory `/nix/store/jp4shpma6ia2npq8wcicdkvvzif64ryd-grub-2.04-340-g8fcfd1e0f/share/locale': No such file or directory.
  Installation finished. No error reported.

or this
  updating GRUB 2 menu...
  installing the GRUB 2 boot loader on /dev/disk/by-id/ata-VBOX_HARDDISK_VBd720cde3-a0dcf48f...
  Installing for i386-pc platform.
  /nix/store/zibr85lkdsnxfq1g40zflrclamflq76q-grub-2.04-340-g8fcfd1e0f/sbin/grub-install: warning: cannot open directory `/nix/store/zibr85lkdsnxfq1g40zflrclamflq76q-grub-2.04-340-g8fcfd1e0f/share/locale': No such file or directory.
  Installation finished. No error reported.

I haven't seen this message on NixOS 20.09.  However, it boots fine afterwards.

I've tested this on my desktop and it still boots. The logs looks, as usual, unnecessarily concerning. I'm not sure why the /share/locale directory is missing.
Installing for x86_64-efi platform.
/nix/store/bfkhdxc7s79c40vfkz3y5zgzwr9kqqw1-grub-2.06-rc1/sbin/grub-install: warning: cannot open directory `/nix/store/bfkhdxc7s79c40vfkz3y5zgzwr9kqqw1-grub-2.06-rc1/share/locale': No such file or directory.
You have a memory leak (not released memory pool):
[0x18c68d0] dtree
Internal error: Unreleased memory pool(s) found.
You have a memory leak (not released memory pool):
[0x18c68d0] dtree
Internal error: Unreleased memory pool(s) found.
You have a memory leak (not released memory pool):
[0x18c68d0] dtree
Internal error: Unreleased memory pool(s) found.
You have a memory leak (not released memory pool):
[0x18c68d0] dtree
Internal error: Unreleased memory pool(s) found.
You have a memory leak (not released memory pool):
[0x18c68d0] dtree
Internal error: Unreleased memory pool(s) found.
You have a memory leak (not released memory pool):
[0x18c68d0] dtree
Internal error: Unreleased memory pool(s) found.
Installation finished. No error reported.

Another Review per https://nixos.org/manual/nixpkgs/stable/#reviewing-contributions-package-updates
Reviewed points

[x] package name fits guidelines
[x] package version fits guidelines
[x] package build on x86-64
[x] executables tested on x86-64
[ ] all depending packages build

More:

[x] booted x86-64 gpt/bios system
[x] booted x86-64 gpt/efi system
[x] checked package license: still GPL3+ according to source repo

Comments

Tested after cherry-picking the pull requests's only commit 55eb638d8922f723788cd019345caeaa6dc69ec7 on top of current nixos-unstable, that is 916ee862e87ac5ee2439f2fb7856386b4dc906ae.

Tested with same configuration.nix as in my previous review comment.

I'm seeing the same error messages as reported in my previous review comment (with different package hashes, naturally).  Both systems (this time gpt/bios instead of mbr/bios) boot fine.

I'm not seeing the "Internal error: Unreleased memory pool(s) found." error messages reported by @mweinelt.

git-bisect in grub's repository showed me this commit causing the error message about the missing locale directory:
https://git.savannah.gnu.org/cgit/grub.git/commit/?id=8451454889f3b46312cc16eb07970d6c20da0812
However, the commit message point to this commit as the root cause:
https://git.savannah.gnu.org/cgit/grub.git/commit/?id=3d8439da8c9a4acf9bc1c41b364ec6e1680ef052
I successfully tested adding --disable-nls to the configureFlags in the build recipe: The error message no longer appears on x86-64/linux and grub still boots gpt/efi, gpt/bios and mbr/bios systems.
I don't know if this breaks nls-support on other plattforms.  Also, it might be better to actually enable nls support, possibly by adding a necessary library to grub's dependencies.  I followed some traces in the source code up to a configure flag --localedir that might control the location of that "missing" directory, but I couldn't figure out how that flag would help anything.

@Yarny0 This improves things AFAICT.

Partial Review per https://nixos.org/manual/nixpkgs/stable/#reviewing-contributions-package-updates
Reviewed points

[x] package build on x86-64
[ ] all depending packages build

More:

[x] booted x86-64 gpt/bios system
[x] booted x86-64 gpt/efi system

Comments

Tested after rebasing the pull request's branch 3105075b87cd65f74029eeed5dd0b25f81352f5e on top of current nixos-unstable, that is still 916ee862e87ac5ee2439f2fb7856386b4dc906ae.
The error message about missing "share/locale" is still visible when the bootloader gets installed.  I have the impression that --enable-nls was already implied before it got set explicitely, so the new commit probably has no effect at all.

Dropped the commit that enabled nls. It didn't change anything.

Deployed to AWS.
I got this error:
Installing for i386-pc platform.
/nix/store/mr443rqlbkim37zhy0pwgzq4hnhvgf19-grub-2.06-rc1/sbin/grub-install: warning: cannot open directory `/nix/store/mr443rqlbkim37zhy0pwgzq4hnhvgf19-grub-2.06-rc1/share/locale': No such file or directory.
** awsserver
updating GRUB 2 menu...
1 block

Asking host to reboot ... Remote host disconnected.
OK
Waiting for host to come online .ssh: connect to host <IP_ADDRESS> port 22: Connection refused
.... OK
Running healthchecks on awsserver (<IP_ADDRESS>):
Health checks OK
Done: awsserver

:+1:

vs. master:
Executing 'boot' on matched hosts:

** awsserver
updating GRUB 2 menu...
1 block
installing the GRUB 2 boot loader on /dev/xvda...
Installing for i386-pc platform.
Installation finished. No error reported.

Asking host to reboot ... Remote host disconnected.
OK
Waiting for host to come online ..ssh: connect to host <IP_ADDRESS> port 22: Connection refused
.. OK
Running healthchecks on awsserver (<IP_ADDRESS>):
Health checks OK
Done: awsserver

maybe we can fix the locale issue? I'm merging, though, hopefully someone fixes the locale error up.

I think this broke the bootloader for me. I got error: symbol 'grub_calloc' not found. Entering rescue mode.... I won't try again but I think the issue is between master and nixos-unstable - probably between f5e8bdd07d1afaabf6b37afc5497b1e498b8046f...4787a38c707c36fd74e92abd6ad8ce0979f4a4f2

@mweinelt I get this reproducibly - Do you have any idea why? I now had to revert the bootloader from a live system twice and I'm sure it's this commit.

I do not, I can only speculate. You are also not mentioning your system architecture and boot method.

I strongly suspect this has something to do that I also have ubuntu installed. I assume it's mixing up both bootloaders or so. I will try some methods and report back later. But I don't think this affects many people also because nobody else reported anything yet.
It's x86_64 and I'm booting using grub with efiSupport on.
  boot.loader.efi.canTouchEfiVariables = false; # https://github.com/NixOS/nixos-hardware/pull/134#discussion_r361146814
  boot.loader.efi.efiSysMountPoint = "/boot";
  boot.loader.grub.efiSupport = true;
  boot.loader.grub.device = "nodev";
  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  boot.loader.grub.configurationLimit = 5;

I think I need to reenable canTouchEfiVariables.

I could imagine that the files GRUB loads from your EFI partition are incompatible with the version installed.

Same error.
My configuration:
  boot = {
    loader.grub = {
      enable = true;
      version = 2;
      device = "/dev/sdc";
      extraConfig = "
        serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1
        terminal_input console
        terminal_output console
      ";
      splashImage = null;
    };
...

@Izorkin x86_64? UEFI or BIOS (looks like bios)? And do you have other operating systems installed?

For my case - my current "solution" was to but Ubuntu's (my second operating system) boot files onto a separate efi partition. But I just did this because I didn't want to research even more and I also need this device for an exam in a few days.

  boot.loader.grub = {
    enable = true;
    configurationLimit = 2;
    version = 2;
    device = "nodev";
    efiSupport = true;
    enableCryptodisk = true;
  };

wfm.

@Izorkin x86_64? UEFI or BIOS (looks like bios)? And do you have other operating systems installed?

System - x86_64. Booting from gpt disks. Not using UEFI.

So no other operating systems? @Izorkin

So no other operating systems? @Izorkin

Installed only NixOS.

If that's the case it could also be that grub doesn't reinstall the bootloader properly. What do you think @mweinelt

If that's the case it could also be that grub doesn't reinstall the bootloader properly (e.g. forgetting to update some old files). What do you think @mweinelt

Really not a grub expert. I updated the package due to the boothole security vulnerability. If it doesn't work out for multiple people we might need to revert and report this issue upstream, or consult someone more knowledgable about the whole GRUB shebang in nixpkgs.

Maybe someone can give this workaround a try?
https://github.com/etu/nixconfig/commit/cbccfb8a11f276ce862411d4842695cb8a134780

I encountered the issue in https://github.com/NixOS/nixpkgs/pull/115310 because I had a leftover grub 2.04 on the other drive that was not being updated. I fixed with
     loader.grub = {
       enable = true;
       version = 2;
-      device = "/dev/sda";
+      # Make sure it is updated on both drives to avoid an issue where a leftover grub is
+      # not updated. https://github.com/NixOS/nixpkgs/pull/115310#issuecomment-805840600
+      devices = [ "/dev/sda" "/dev/sdb" ];
     };

Also Hetzner users note that with the KVM, you can send Ctrl-Alt-Delete to go into the BIOS and use the boot menu (not the default boot setting) to temporarily boot it from the other drive with the correct grub.

I also  encountered the issue in https://github.com/NixOS/nixpkgs/pull/115310#issuecomment-805840600 . But in my case, my grub.devices is up to date.

Are you sure you don't have another drive with an older grub in the MBR?

Yeah, I am positive about that. I have only one hard drive which has gpt partition table.

I experienced the same problem, using GRUB with a second bootable partition. Fixed by switching to systemd-boot.

Bleh. Just encountered this same error on one of my NixOS systems after an update and reboot:

No other OS were ever installed on these drives
Two UEFI grub devices, and both are failing with this error

I'm going to bed; will boot from a rescue system and attempt recovery tomorrow.

Got my server back up, eventually. I did some experimenting and determined the cause of the issue in my case: at some point, I had efi.canTouchEfiVariables = true; set, but later commented it out. My system has been defaulting to booting from this old EFI menu entries, and the older grubx64.efi they use is not compatible with the grub module files from post-boothole-patch grub.
A thought occurs: if NixOS configured grub installations to use their own, per-installation module files directory, this issue would not have happened. Although I would have also been left unknowingly booting an older grub version, but uh... that honestly doesn't really affect me at all, although it would some others.

A thought occurs: if NixOS configured grub installations to use their own, per-installation module files directory, this issue would not have happened. Although I would have also been left unknowingly booting an older grub version, but uh... that honestly doesn't really affect me at all, although it would some others.

Things get more complicated. They do get put into a subdirectory if the bootloader is given an id. But not all of the files. Last time I looked at this it was because of how much of the grub-install we were doing by hand, and not precisely how it's done by grub proper. This is further shown with issues that indicate we in fact, can't, support systemd-recommend ESP-mount configurations because of how our grub installer works. This in turn makes things like extending our grub with a signed-shim more complicated than it needs to be too.


Have multiple repositories open gives rendering issues wrt branch name and push state

Issue Type: Bug
Open 2 repositories , go to source control tab, switch branch on one of the repos, observe that "branch name" changes on all open repos to the selected branch, even though the branch does not even exist on the other repo. Similar issues occur with synchronizing changes
New on latest version
VS Code version: Code 1.61.0 (ee8c7def80afc00dd6e593ef12f37756d8f504ea, 2021-10-07T18:13:09.652Z)
OS version: Windows_NT x64 10.0.19043
Restricted Mode: No

/duplicate
See https://github.com/microsoft/vscode/issues/134638#issuecomment-938471497 for a workaround.
Scheduled to be fixed in upcoming recovery release.


What is the colorscheme?

@halfo
Hi,
What is the coloscheme used in the screenshot?
I love the theme and I would love to duplicate the effect.
Regards,
Budziq

Hello! I'm using a darker version of "Paraiso Dark theme" for my terminal.

Awesome Thanks!


[DRAFT] chore(docs): Updating Guide and Waydroid in docs home

capitalized sections for consistency
added update, rollback, and rebasing to the index
and added waydroid so now there's 3 blocks on top and 3 on the bottom

i hope this doesn't break anything... :)

reminder to myself to see @Zeglius's corrections later.


construction: Add overlay for number of items held by servant

Closes #4214

Below pasting all available servant responses for reference:
Rick
Here you go, mate. Oop, I see your hands are full. I'll just hold these X items till you're ready.
Here you go, mate. Oop, I see your hands are full. I'll just hold this last thing till you're ready.
Here you go, mate. Got a bit lost on the way back, sorry 'bout that.
Maid
I, er, I see your hands are full. I'll just hold these X items for you till you're ready. Yes, I'll hold them.
I see your hands are full. I'll hold the last one for you till you're ready.
Phew! I got back from the bank okay! Here you are, sir.
Cook
I go all the way to the bank for you and then you can't even take these X items because your hands are full!

Here you are.
Butler
I will hold these X items until sir has space to take them.
I will hold this one until sir has space to take it.
Your goods, sir.
Demon butler
Master, I have returned with what thou asked me to retrieve. As I see thy inventory is full, I shall wait with these X items until thou art ready.
Master, I have returned with what thou asked me to retrieve. As I see thy inventory is full, I shall wait with the last item until thou art ready.
Master, I have returned with what you asked me to retrieve.

can you set the text colour to green when you have enough space in your inventory to take all items?

I am still interested in this if you do want to change it to an infobox.


Refactor and fixes

Brings back auto-generation of SSL certificates.
Brings back dnsmasq.
Brings back kong cluster .. CLI
Implements #1281 (already implemented in next).
Bugfixes and other minor changes (like improving Serf info messages at startup).
Organizing pids, logs and Serf files respectively in pids, logs and serf folders.

Agree that passing configuration for stopping a service is not ideal (and it can also be error prone: what happens if somebody updates the configuration files at runtime and then trie to stop/start Kong?). You could store the configuration using nginx's PID as a key (e.g., create a folder for the PID somewhere and store the configuration files for the duration of the application's lifecycle)


Tutorial to run train/evaluation/inference for a 2D/3D Deepgrow Model

Depends on: https://github.com/Project-MONAI/MONAI/pull/1395
Deepgrow Examples
This folder contains examples to run train and validate a deepgrow 2D/3D model.
It also has notebooks to run inference over trained model.
More details

@wyli @Nic-Ma
The tutorial examples are verified by me and YuanTingHsieh.. and its ready for final review


Calculate requests to prune in RateLimitedQueue.prune

Description
Currently, I prune overflow requests in multiple loop iterations. This PR calculates how many requests we should prune directly to avoid this while loop.
This PR also:
*Put the hardcoded worker name to variables for code reuse.
*Fixed a bug in RateLimitedRequestWorker’s constructor that doesn’t assign variables in the setting update consumer.
Testing done:

added unit tests for related changes.

Issues Resolved
https://github.com/opensearch-project/anomaly-detection/issues/66
Check List

[ X ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Can you put the original PR comment link here? So others can know the context.

Can you put the original PR comment link here? So others can know the context.

It is linked after the "Issues Resolved" in the commit message.


Run the simulator with the robotology binaries installed through Conda

Since https://github.com/robotology/robotology-superbuild/pull/652, we should be able to install the matlab-whole-body-simulator dependencies from the Conda robotology channel, using the conda-forge or Mamba installer.
This would significantly simplify the installation of the matlab-whole-body-simulator simulator on  local host and unlock its easy setup and run from MATLAB online.
In view of validating such procedure, we shall:

[ ] Test the installation on a local machine using the dependency binaries from conda robotology channel.
[ ] Test the installation on MATLAB online as explained in https://github.com/robotology/robotology-superbuild/pull/652#issuecomment-794027294.
[ ] Fix any encountered issues.
[ ] Document the installation README accordingly.

WIP:
Test the installation on a local machine using the dependency binaries from conda robotology channel
We followed the steps listed below:

Install Mamba, create a new environment and install the robotology dependency binaries:$ conda install mamba
$ conda create -n robotologyenv
$ conda activate robotologyenv
$ mamba install -c robotology iDynTree qpOASES icub-models wb-toolbox whole-body-controllers

Checks... the MATLABPATH env should now have:<user-home-dir>/miniforge3/envs/robotologyenv/mex: <user-home-dir>/miniforge3/envs/robotologyenv/share/WBToolbox: <user-home-dir>/miniforge3/envs/robotologyenv/share/WBToolbox/images

Mex libraries:$ ls <user-home-dir>/miniforge3/envs/robotologyenv/mex/

+iDynTree
+iDynTreeWrappers
+wbc
BlockFactory.mexmaci64
BlockFactory.tlc
SwigGet.m
SwigMem.m
SwigRef.m
iDynTreeMEX.mexmaci64
mesh2tri.m

Clone the repository  matlab-whole-body-simulators$ git clone https://github.com/dic-iit/matlab-whole-body-simulator.git

Run matlab in the same conda environment.
Change working directory to the root path of repository matlab-whole-body-simulators
Open and run the model test_matlab_system_2020b.mdl.

Notes
After the environment is activated (conda activate robotologyenv), the terminal prompt starts with (robotologyenv) .
Issues
README
The two dependencies  wb-toolbox and whole-body-controllers were missing from the dependencies list in the repo main README.

Matlab/Simulink 2019b
iDynTree
OSQP
osqp-matlab (the OSQP MATLAB bindings).
qpOASES
icub-models to access iCub models.

OSQP and osqp-matlab are not needed anymore, at least until OSQP replaces qpOASES.
The main README shall be updated accordingly.
MATLAB

...

2. MATLAB issue

=> In the getMeshes(...) function, we were skipping the loop where the output map was set.

=> This was due to the number of links of the model being 0, which was due to a wrong model path:robot_config.modelPath = '/share/iCub/robots/iCubGenova04/'

=> due to icubModelsInstallPrefix being empty.

=> due to environment variable ROBOTOLOGY_SUPERBUILD_INSTALL_PREFIX not properly set.

I was expecting this variable to be automatically set by the activation of the Conda environment.
Setting that variable would make the use of the superbuild binaries backward compatible with software depending on a locally built superbuild. What do you think @traversaro ?

Setting that variable would make the use of the superbuild binaries backward compatible with software depending on a locally built superbuild. What do you think @traversaro ?

I understand that it should be more robust to use more relevant variables already set in the scripts conda/multisheller/<component>_activate.msh, right?

Setting that variable would make the use of the superbuild binaries backward compatible with software depending on a locally built superbuild. What do you think @traversaro ?

I understand that it should be better to use as much as possible more relevant variables already set in the scripts conda/multisheller/<component>_activate.msh, right?

Yes, to be honest the ROBOTOLOGY_SUPERBUILD_INSTALL_PREFIX env variables was just an helper variable used inside the setup scripts, and not something that was supposed to be used. In general, robotology-superbuild is one of the many ways in which software is distributed, and hardcoding the use of a given generic software to a given distribution solution typically reduce re-usability.
For an example, consider the existing README, it writes that:

It is recommended to install iDynTree, icub-models, OSQP and qpOASES using the robotology-superbuild:

it just suggests the use of robotology-superbuild, while this use of ROBOTOLOGY_SUPERBUILD_INSTALL_PREFIX basically makes this use compulsory.
In general, the code in https://github.com/dic-iit/matlab-whole-body-simulator/blob/30642d6c195ae1c663d6ebbf43e81f95eff1946a/init.m is quite hardcoded to the use of iCub and robotology-superbuild. Could it make sense to change that and find the model via one of the many ways already supported by icub-models (see https://github.com/robotology/icub-models), for example by specifying the model via a package:// URI?
We can think of having a code such as:
robotName = 'iCubGenova04';
[modelPath, fileName] = getFileFromROSPackageURI('package://iCub/robots/' robotName)
Config.modelPath = modelPath;
Config.fileName = fileName;

The getFileFromROSPackageURI function (that can be directly implemented in MATLAB) then can just find the file via the ROS_PACKAGE_PATH and AMENT_PACKAGE_PATH env variables (that are set if you install icub-models also via conda). See https://github.com/robotology/idyntree/blob/918084ff3ab4a9fb9e28c88f09dc6dbf9d6bddcc/src/model/src/SolidShapes.cpp#L200 for a related code.

Yes, indeed, I saw the environment variables set in conda/multisheller/icub-models_activate.msh, among which ROS_PACKAGE_PATH and AMENT_PACKAGE_PATH as you said.
Initially I was planning to use them as you suggest, but avoiding to do a search of the file in those paths, I instead used directly the Yarp resource finder (via the MATLAB system command), which was also installed along with icub-models via Conda. This way, the model is searched the same way it is searched by the WBToolbox models.
I pushed the code in the branch fix/for-using-superbuild-ninaries. What do you think?

Fix any encountered issues. ONGOING: $ conda init failing on MATLAB online.

If this is tricky to solve we can also solve it later.

@traversaro , if it's ok for you we can close this

Ok, feel free to close yourself!


Unregister the UDTs after use in ArrayViewTest and ArrayWriterTest

ArrayViewTest and ArrayWriterTest register UDTs, which should be unregistered
after use to avoid causing instability in the test results of
CustomTypeTest.getCustomTypeNames.

cc @mbasmanova, thanks.

cc @mbasmanova, thanks.

Fixed a typo.

Fixed https://github.com/facebookincubator/velox/issues/9169.


[BUG]: Apple Pay Button Type Continue throws error

What happened?
According to the docs (https://docs.stripe.com/js/elements_object/create_express_checkout_element#express_checkout_element_create-options-buttonType-applePay) we should be able to change the text for the apple pay button to "continue" however when I do so I get the following error:

Environment
No response
Reproduction
No response

Hi, this looks like an integration question, and we aren’t able to answer those here on GitHub. Please reach out to Stripe Support for help.

@fruchtose-stripe the documentation says we're allowed to pass "continue" but the types don't allow for it.

@stevez86 Thanks for clarifying that. I will pass this along to the appropriate engineering team.

And to further clarify, it's not just types, the integration also throws an error.

Thanks @stevez86. I'll add that to the internal report.

Thanks for pointing this out! For the time being, I have removed the type as an option from our docs. Unfortunately, the continue Button Type cannot be displayed using CSS, which is how we render it.
I have reached out to Apple to have them investigate if support can be added and will update when we hear back.

Update: Apple confirmed this is not supported and they will update their docs as well to indicate that continue is not a supported option using CSS.


Trucks assignment for VRP

Hi,
Here a scenario, I have 3 plants to deliver material to multiple customer sites. Take for an example,
Plants: P1, P2, P3
Customer sites: S1, S2, S3, S4, S5, S6
Constraints

Only allow trucks to deliver material from a plant to a site and return to the nearest plant after delivery material to customer site, such as P1 -> S1 -> P1 (or P2 if P2 is the nearest plant for return trip).

The trucks are not to delivery material from any plant to another plant

The trucks are not to delivery from customer site to another customer sites, such as P1 -> S2 -. S3. (this is not possible)

The trucks have to deliver the orders base on each orders start time.

I have tried TSP, VRP, CVRP, pickup and delivery and seem doesn't fit in the constraints where i have mentioned above. Seek some advice for the solutions.
Thank you.
Alex

This is not a tsp. You do not create routes.
This is an assignment.
Le jeu. 26 sept. 2019 à 03:29, alexpoh<EMAIL_ADDRESS>a écrit :

Hi,
Here a scenario, I have 3 plants to deliver material to multiple customer
sites. Take for an example,
Plants: P1, P2, P3
Customer sites: S1, S2, S3, S4, S5, S6
Constraints

Only allow trucks to deliver material from a plant to a site and
return to the nearest plant after delivery material to customer site, such
as P1 -> S1 -> P1 (or P2 if P2 is the nearest plant for return trip).
2.
The trucks are not to delivery material from any plant to another plant
3.
The trucks are not to delivery from customer site to another customer
sites, such as P1 -> S2 -. S3. (this is not possible)
4.
The trucks have to deliver the orders base on each orders start time.
I have tried TSP, VRP, CVRP, pickup and delivery and seem doesn't fit in
the constraints where i have mentioned above. Seek some advice for the
solutions.
Thank you.
Alex
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/google/or-tools/issues/1598?email_source=notifications&email_token=ACUPL3M6FVYT623J5C4YQ43QLQGABA5CNFSM4I2TYUDKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HNYBPBQ,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACUPL3IEKX3CQUIHZFRBA33QLQGABANCNFSM4I2TYUDA
.

Thanks for your advice, will looking into it.
By the way, i will like to understand how is the flow being compute from the maximum cost flow, such as flow for Arc 0 -> 1 is 12?
Arc    Flow / Capacity  Cost
0 -> 1    12  /  15        48
0 -> 2     8  /   8        32
1 -> 2     8  /  20        16
1 -> 3     4  /   4         8
1 -> 4     0  /  10         0
2 -> 3    12  /  15        12
2 -> 4     4  /   4        12
3 -> 4    11  /  20        22
4 -> 2     0  /   5         0


Add safety check to that TransposeBigMLFloat16 test passes

Added a check that queries hipGetDeviceProperties and returns false if gridDim.y is larger than the device maximum supported gridDim.y.
This check existed inside CUDA EP but was absent from ROCM EP and as a result TransposeBigMLFloat16 test was failing.

Upstreaming this.


Update Composer dependencies (2020-11-06-00-08)

Visual regression test passed!
View the visual regression test report


get_twin_x509.py error "SSL/TLS has already been configured"

I've successfully registered a device using group enrollment with x509 certificates, and have been able to run the scripts from this SDK for a few days.
But suddenly the script get_twin_x509.py stopped working. The error it cites is:
future: <Task finished coro=<main.<locals>.twin_patch_listener() done, defined at get_twin_x509.py:42> exception=SystemExit(1)> Traceback (most recent call last):   File "/usr/local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_stages_base.py", line 94, in run_op     self._run_op(op)   File "/usr/local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 104, in _run_op     raise e   File "/usr/local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 100, in _run_op     password=self.sas_token, client_certificate=self.trusted_certificate_chain   File "/usr/local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 157, in connect     self._mqtt_client.tls_set_context(context=ssl_context)   File "/usr/local/lib/python3.7/site-packages/paho/mqtt/client.py", line 688, in tls_set_context     raise ValueError('SSL/TLS has already been configured.') ValueError: SSL/TLS has already been configured.
The system in question is Raspbian, Python 3.7.4

Oh, apparently the script setting env variables with certificate file paths was outdated.


New Blog Post

Blog post on Istanbul ETHPrivacy Hackathon.

Thanks. Will implement changes you mentioned. @vivianjeng


Edit: TGRT Haber

Channel ID (required)
TGRTHaber.tr
Channel Name
No response
Alternative Names
No response
Network
No response
Owners
No response
Country
No response
Subdivision
No response
City
No response
Broadcast Area
No response
Languages
No response
Categories
No response
NSFW
None
Launched
No response
Closed
No response
Replaced By
No response
Website
No response
Logo
No response
Notes
https://tv.ensonhaber.com/tgrthaber/tgrthaber_1080p.m3u8

https://github.com/iptv-org/iptv/issues/new?assignees=&labels=streams%3Aadd&projects=&template=-----streams_add.yml&title=Add%3A+


2024-12-19_01-36-10_master-next_aws-crt-python

Automatically created.

Successfully created backport PR for kirkstone-next:

#10870

Successfully created backport PR for scarthgap-next:

#10871

Successfully created backport PR for styhead-next:

#10872


Fixes #178 - Unsupported ARIA attribute fails with global attributes

supported was sometimes an object, sometimes an array.
Probably best as an object for fast lookup.

@alice PTAL

@alice hopefully it's a bit more consistent with other constants now... Happy to tweak it further if required.

@googlebot alice is the author of those commits and has signed the CLA / works for Google.

alice (the project owner) is the author of the other commits and has signed
the CLA / works for Google
On 27 July 2015 at 09:36, googlebot<EMAIL_ADDRESS>wrote:

We found a Contributor License Agreement for you (the sender of this pull
request) and all commit authors, but as best as we can tell these commits
were authored by someone else. If that's the case, please add them to this
pull request and have them confirm that they're okay with these commits
being contributed to Google. If we're mistaken and you did author these
commits, just reply here to confirm.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleChrome/accessibility-developer-tools/pull/195#issuecomment-125048205
.

:+1: