Impersonation credentials issue -  Airflow

Hi Team,
I am trying to use this tool to validate some tables in BigQuery. I have am using the code example mentioned here https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/samples/airflow/dvt_airflow_dag.py
I am getting the below mentioned error:
google.api_core.exceptions.Forbidden: 403 GET .... Access Denied: Table project_id:dataset_id.table_name : Permission bigquery.tables.get denied on table project_id:dataset_id.table_name (or it may not exist.)

I am using the correct impersonated credentials. e.g.
project_id = projectId
target_scopes = [
            "https://www.googleapis.com/auth/cloud-platform"
        ]
    default_creds, _ = google.auth.default()
    impersonated_creds = impersonated_credentials.Credentials(
        source_credentials=default_creds,
        target_principal=SERVICE_ACCOUNT,
        target_scopes=target_scopes,
    )
handler = bqhandler.BigQueryResultHandler.get_handler_for_project(project_id = project_id, credentials=impersonated_creds)

Please help me figure out what am I doing wrong?

This looks good to me.. I wonder if Composer has a specific service account that the environment uses that needs BigQuery permissions.
@helensilva14 May have more insight

Hi Team,
On the above, the issue we are facing is with connecting to the source by itself and not just result handler. And we are passing the service account credentials and the composer uses the same and it has access to the BQ tables. So we are not able to figure out the problem.
Also In source connection string we pass below per the Dag sample.
	BQ_CONN = {"source-type": "BigQuery", "project-id": project}

And we see option to pass key path for connectivity string as below
# (Optional) BigQuery JSON Config File for On-Prem usecases
"google-service-account-key-path": "/path/to/key.json"

But our project doesn't have a key path. Is there any other way to pass the credentials in the connection string other than json key path or secret manager.

Hi @chandantroughia! Are BQ and Composer in the same GCP projects? Could you please confirm if the SERVICE_ACCOUNT you are providing have at least one of the following BQ roles? Asking that because these roles contain the missing permission bigquery.tables.get described in the error.

https://cloud.google.com/iam/docs/understanding-roles#bigquery.admin
https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor
https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataOwner
https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataViewer

Hi @bharath-cv! What errors are you facing besides the one listed in the first comment? It can help us to have more understanding of the issues you're facing.
Also I'm going to re-create the sample DAG in our environment to see if I can reproduce the error.

Hi @helensilva14
I am sure that the service account has these roles as when we run the DAG to run batch process and query the tables it is able to fetch the results. It is only with DVT we are facing this error.
I am also getting same error as Chandan.
The only difference is our project doesn't have a keypath file defined.

Hi @helensilva14
I am sure that the service account has these roles as when we run the DAG to run batch process and query the tables it is able to fetch the results. It is only with DVT we are facing this error.
I am also getting same error as Chandan.
The only difference is our project doesn't have a keypath file defined.

Hi @bharath-cv, on top of @dmedora comment, can you provide more specifications such as the versions of Airflow, Composer, DVT, Python that you're using? Also could you clarify if the BQ table and Composer are on the same GCP project?

Hi @chandantroughia and @bharath-cv! Have you been able to take a look at the Cloud Logging
records or do you have any updates on this case so we can continue to investigate it? If not, could you please try to do it within a week? Otherwise we'll consider to close the issue as "Can't reproduce/Stale". You can always reopen the issue, if needed. Thank you!

Hi @helensilva14
Please find the logs attached. The first log is the log from the DAG. The second one name log2 is the info from the link below
google.api_core.exceptions.Forbidden: 403 GET https://bigquery.googleapis.com/bigquery/v2/projects/--/datasets/poa_de_hcb_dev/tables/cob_param?prettyPrint=false: Access Denied: Table --:poa_de_hcb_dev.cob_param: Permission bigquery.tables.get denied on table --**:poa_de_hcb_dev.cob_param (or it may not exist).
DVT_airflow_log1.txt
DVT_airflow_log2.txt
Please take a look and let us know how we can use the DVT tool. Our use case here is  to compare data between on-prem (hive) to gcp.

Hi @bharath-cv,
Today, when you pass the impersonated credentials to the BigQueryResultHandler, it would only be used for writing the validation results to BigQuery. If your source or target system is BigQuery, it is currently not possible to pass impersonated credentials to the clients used to execute those queries. We have opened feature request #794 for that. So today, even if you pass impersonated creds to the result handler, the source and target queries are using default creds.
For now, the options are to either provide the appropriate permissions to service account bound that's bound to Composer or provide a service account key file to be used by the job.
In your earlier comment https://github.com/GoogleCloudPlatform/professional-services-data-validator/issues/768#issuecomment-1466824114, you mentioned that the service acc you're using for Composer already has BQ permissions, so we can try to debug that issue further.
Firstly, could you try running this DAG and let us know if it runs successfully? It runs a query using the BQ operator/client directly (instead of via DVT) so we can narrow down the cause of the issue. Note the TODOs indicating values you need to update.
from datetime import datetime
from airflow import models
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator

default_args = {"retries": 0, "owner": "dvt_debug"}

with models.DAG(
    "example_bigquery_insert",
    schedule_interval="@once",
    default_args=default_args,
    start_date=datetime(2021, 1, 1),
    tags=["dvt_debug"],
) as dag:
    insert_job = BigQueryInsertJobOperator(
        task_id="insert_query_job",
        project_id="your-project", # TODO: edit with your project ID
        configuration={
            "query": {
                "query": "SELECT * FROM `my-project-id.my-dataset.my-table`",  # TODO: edit with you query
                "useLegacySql": False,
            }
        },
    )

    insert_job

Once you've run it, could you also please search in Cloud Logging as requested in my last comment and share the full entry details. You can use this filter:
resource.type="bigquery_resource"
protoPayload.methodName="jobservice.insert"
<EMAIL_ADDRESS>

@bharath-cv Is this still an issue?

Hi @nehanene15 @helensilva14,
I am facing the exact same issue.
I was able to run the insert query,
but I has to pass the parameter: impersonation_chain =<EMAIL_ADDRESS>into BigQueryInsertJobOperator function for a successful run.
I appreciate that a feature request was added.
But, is there any work around for this issue for now?

Hi @mylar-pr, thanks for reaching out! Unfortunately, we do not have a workaround for this issue at the moment. Is it possible for you to execute DVT using other services or you only have Composer as option?

Hi Helen,
Thank you for the response.
The purpose of the DVT in my case to perform data validation between BQ table and Dataproc Hive tables. Could you please suggest any alternative services/solutions to accomplish this task ?
Thank you,
Regards,
Prajwal Mylar
Get Outlook for iOShttps://aka.ms/o0ukef

From: Helen Cristina @.>
Sent: Tuesday, September 5, 2023 12:37:37 PM
To: GoogleCloudPlatform/professional-services-data-validator @.>
Cc: Prajwal Mylar @.>; Mention @.>
Subject: Re: [GoogleCloudPlatform/professional-services-data-validator] Impersonation credentials issue - Airflow (Issue #768)
Hi @mylar-prhttps://github.com/mylar-pr, thanks for reaching out! Unfortunately, we do not have a workaround for this issue at the moment. Is it possible for you to execute DVT using other services or you only have Composer as option?
—
Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/professional-services-data-validator/issues/768#issuecomment-1707206635, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOKLUGS6NXARV7BUFCRNWYDXY55QDANCNFSM6AAAAAAVX47VY4.
You are receiving this because you were mentioned.Message ID: @.***>

To use Composer, the options are to either provide the appropriate permissions to service account that's bound to Composer or provide a service account key file to be used by the job.
Alternatively, you can use Compute Engine, Cloud Run, or Functions depending on how large of validations you need to perform.

Hi, are there plans to support impersonation for running dvt inside airflow dag? We are also in the same situation and want to use this library and impersonation would be really helpful. Thanks

@moizarafat Can you try the following in Airflow:
impersonated_creds = impersonated_credentials.Credentials(
    source_credentials=default_creds,
    target_principal=SERVICE_ACCOUNT,
    target_scopes=target_scopes,
)
BQ_CONN = {"source_type": "BigQuery", "project_id": project, "credentials": impersonated_creds}

This should connect with the appropriate credentials.
When you create a BQ connection via the CLI, we take the SA key path, but then use google.oauth2.service_account.Credentials.from_service_account_file() to extract the Credentials object.
You should be able to pass that in directly and it will populate the BQ Client here.

Thanks @nehanene15 that solved it. We ahad to pass the same credentials to result handler too for bq load of the results.


LEGO: Pull request from lego/hb_04604851-bac4-4681-9f74-73de611d6e48_20210715033730844 to main

LEGO: Pull request from lego/hb_04604851-bac4-4681-9f74-73de611d6e48_20210715033730844 to main with localized lcls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 42.73%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
26222

Relevant Lines:
55694

💛  - Coveralls


Allow specifying own Hugging Face tokenizer instance

This gives the option of specifying your own PreTrainedTokenizer instance rather than specifying a model name and having outlines construct the tokenizer for you. This might be convenient for situations where you are already using the LLM in other parts of your application and only need outlines for a specific use case.
Example of use:
from transformers import AutoTokenizer, AutoModelForCausalLM
from outlines.models.transformers import TransformerTokenizer, Transformer

model_name = "hf-internal-testing/tiny-random-GPTJForCausalLM"
hf_tokenizer = AutoTokenizer.from_pretrained(model_name)
hf_model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = TransformerTokenizer(hf_tokenizer)
model = Transformer(hf_model, tokenizer)

As compared to before this change:
from transformers import AutoTokenizer, AutoModelForCausalLM
from outlines.models.transformers import TransformerTokenizer, Transformer

model_name = "hf-internal-testing/tiny-random-GPTJForCausalLM"
hf_tokenizer = AutoTokenizer.from_pretrained(model_name)
hf_model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = TransformerTokenizer(model_name) # Can't make use of hf_tokenizer here, even if
                                             # you already instantiated it for other reasons!
model = Transformer(hf_model, tokenizer)

This also corrects a type annotation on the outlines.models.transformers.Transformer class which incorrectly stated that the tokenizer argument was of type transformers.PreTrainedTokenizer when actually it should be of type outlines.models.transformers.TransformerTokenizer. In order to fix the annotation, the classes also had to be rearranged so that the name of the other class would be defined at the right time.

This part of the interface has been bugging me for a while, thank you for helping!


Not able control the threads using maven failsafe plugin

I am using cucumber v4 . I configured my maven threads failsafe plugin to execute scenarios based on the number of threads provided. However, it executes all of them in parallel and ignores the thread count set.
Expected Behavior
The scenarios should execute based on the number of threads provided.
Current Behavior
All scenarios are getting executed in parallel.
below is my testRunner class:
package hook;

import cucumber.api.CucumberOptions;
import cucumber.api.testng.AbstractTestNGCucumberTests;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

@CucumberOptions(plugin = {"json:target/cucumber-report.json","html:target"}
,features = "D:/git-projects/cucumberVer4/src/test/features"
,extraGlue = "abhishek8908.stepdefs"
,monochrome = true)
public class TestRunner extends AbstractTestNGCucumberTests{
    @Test(dataProvider = "scenarios")

    @Override
    @DataProvider(parallel = true)
    public Object[][] scenarios() {
        return super.scenarios();
    }

}

POM config:
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.22.0</version>
                <executions>
                    <execution>
                        <id>acceptance-test</id>
                        <phase>integration-test</phase>
                        <goals>
                            <!--<goal>integration-test</goal>-->
                            <goal>integration-test</goal>
                        </goals>

                        <configuration>

                            <threadCount>1</threadCount>
                            <includes>
                                <include>**/*Runner.java</include>
                            </includes>
                            <!--<parallel>classes</parallel>-->
                            <testSourceDirectory>src/test/java/hook</testSourceDirectory>

                        </configuration>

                    </execution>
                </executions>
            </plugin>

Steps to Reproduce (for bugs)

Create a runner in example
Configure threads in the failsafe plugin
Run the test using failsafe plugin.

Your Environment

Version used: 4.2.0
Link to your project: https://github.com/abhishek8908/cucumber-v4-example

This issue was moved to cucumber/cucumber-jvm#1496


[mgmt] refresh rlc mgmt packages

https://github.com/Azure/azure-sdk-for-js/issues/30680

API change check
APIView has identified API level changes in this PR and created following API reviews.
@azure-rest/arm-appservice


Target netcoreapp2.0 in Shell so it can be compiled for MacOS

Added .gitattributes to avoid CL/LF commit issues cross-plat
Normalized all line endings (LiteDB.csproj and LiteDB.Tests.csproj)
Multi-target based on OS so LiteDB.Shell can be compiled for MacOS
Added launch.json and task.json to more easily debug LiteDB.Shell in VSCode
Fixed a warning in TempDiskService.cs

Can be published for MacOS with the following:
dotnet public -c Release -t netcoreapp2.0 -r osx-x64
Resolves #955
Note: The LiteDB.Demo project does give an "Error to load project file"
on Mac in VSCode since it's targeting net46. It looks like it could
easily switch to target netcoreapp2.0 very easily, but I didn't want
to presume anything in this PR.

Thanks @rhullah, that so great when community helps with LiteDB! I have almost no knowledgment about non-windows .NET framework.
I will fix LiteDB.Demo to use netcore2. Thanks again!


[REQUEST] Add Exposure Time

Okay hear me out. This has amazing potential for milky way photography. The A7/R/S and the A6000/A5100 all suffer from the star eater issue in bulb mode. If you could add the option to set exposure time (through T mode, not bulb [B] mode) and added options for 1,2,4,8,15 minutes, then these cameras would once again be relevant for astrophotography. Please consider it and thank you for your time.

That would be really great, but Sony does this filtering directly in the firmware so there is no way for an app to disable or evade it. The only thing we can do is increase the pressure on Sony to add a menu option to disable it.
https://www.change.org/p/sony-remove-star-eater-en

You are correct, but the A7/S/R only does this in bulb mode, so my hypothesis was that if an app could initiate a time through T mode that it would circumvent the filtering. I'm pretty sure it would work. If you are of the opinion that you are unable to initiate a different exposure time than the ones preset in T mode then I agree that an app has no chance.
Thank you for your response.


Enhancement 1 to create_poster.ps1

Export the line used to create the poster so that the user can recreate it in the future and share the line with others if they want to use the same settings

done


Fix ambiguous location issue

Describe your changes
Addressing invalid location returns for incident locations containing 'between', 'to', mulltiple 'and's, etc. This change should be the final one that allows me to return valid addresses for 99.4% of incidents.
Checklist before requesting a review

[x] The code runs successfully.

(ucpd-incident-scraper-py3.11) michaelp@MacBook-Air-18 ucpd-incident-scraper % make correct_location
python -m incident_scraper correct-location
[nltk_data] Downloading package wordnet to
[nltk_data]     /Users/michaelp/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
API queries_quota: 60
Unable to process and geocode address: 5100 S. Drexel Ave. to 5700 S. Hyde Park Blvd.
Unable to geocode address: 5100 S. Drexel Ave. to 5700 S. Hyde Park Blvd.
S. Dorchester Ave. between E. 51st St. and E. 52nd St. was geocoded to: 5120 South Dorchester Avenue, Chicago, IL 60615
S. Blackstone Ave. between E. 55th St. and E. 56th St. was geocoded to: 5520 South Blackstone Avenue, Chicago, IL 60637
...
E. 54th St. between S. Maryland Ave. and S. Drexel Ave. was geocoded to: 5401 S Maryland Ave, Chicago, IL 60615
S. Kimbark Ave. between E. 50th St. and E. 51st St. was geocoded to: 5020 South Kimbark Avenue, Chicago, IL 60615-2922
E. 55th St. and S. Lake Park Ave. was geocoded to: 5500 South Lake Park Avenue, Chicago, IL 60637-1917
2458 of 16883 had their address updated.
2458 addresses were updated.
Waiting up to 5 seconds.
Sent all pending logs.
(ucpd-incident-scraper-py3.11) michaelp@MacBook-Air-18 ucpd-incident-scraper %

Updated XGBoost model as well:
python -m incident_scraper build-model
[nltk_data] Downloading package wordnet to
[nltk_data]     /Users/michaelp/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
Accuracy Score: 0.707519351271655
Precision Score: 0.8793890449438202
Recall Score: 0.7726361252506556

Extra test:
(ucpd-incident-scraper-py3.11) michaelp@MacBook-Air-18 ucpd-incident-scraper % make three_days
python -m incident_scraper days-back 3
[nltk_data] Downloading package wordnet to
[nltk_data]     /Users/michaelp/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
Beginning the UCPD Incident scraping process.
Finished with the UCPD Incident scraping process.
11 total incidents were scraped from the UCPD Incidents' site.
API queries_quota: 60
This incident has an insufficient number of keys: {}
1 of 11 contained malformed or voided information.
0 of 11 could not be processed by the GoogleMaps' Geocoder.
10 of 11 incidents were successfully processed.
Adding 10 of 11 incidents to the GCP Datastore.
Completed adding 10 of 11 incidents to the GCP Datastore.
0 of 1 'Information' incidents predicted into other categories.
1 of 11 incidents could NOT be added to the GCP Datastore.
Program shutting down, attempting to send 2 queued log entries to Cloud Logging...
Waiting up to 5 seconds.
Sent all pending logs.
(ucpd-incident-scraper-py3.11) michaelp@MacBook-Air-18 ucpd-incident-scraper %


Why is OID not Eq?

OID impls PartialEq but not Eq even though (AFAICT) OID equality satisfies Eq - is this just an oversight? Would you accept a patch adding Eq?

I believe it's an oversight. If you can add it, and everything compiles and passes the test suite, I'm cool with a patch.


Core GPP string serving for US geographies

Closes https://ethyca.atlassian.net/browse/PROD-1362
Description Of Changes
Allows serving GPP strings for US regions!
https://www.loom.com/share/38960cec78a34cf6aa198cbc6632ffaa
Code Changes

[x] Added notice key map which captures how privacy notices map to GPP fields
[x] Update calls to CMP API to update GPP fields when privacy notices are served, and when privacy notices are opted in/out of
[x] Integrates with the new GPP settings backend update https://github.com/ethyca/fidesplus/pull/1248
[x] Removes the env var for GPP_ENABLED now that we can get it from the backend
[x] Unit tests
[x] Cypress tests

Steps to Confirm

[ ] In the admin-ui, enable "data sales and sharing" privacy notice
[ ] For convenience, you can also set the privacy experience for an overlay to "always enable" the banner
[ ] Run the privacy center via FIDES_PRIVACY_CENTER__IS_GPP_ENABLED=true turbo dev
[ ] Visit http://localhost:3000/fides-js-demo.html?geolocation=us-ca
[ ] In your console, enter __gpp('ping', (data) => {console.log({data})})
[ ] You should get a response back that is something like

{
    "gppVersion": "1.1",
    "cmpStatus": "loaded",
    "cmpDisplayStatus": "visible",
    "signalStatus": "not ready",
    "supportedAPIs": [],
    "cmpId": 407,
    "sectionList": [
        8
    ],
    "applicableSections": [
        2
    ],
    "gppString": "DBABBg~BWAAAAAA.QA",
    "parsedSections": {
        "uscav1": {
            "Version": 1,
            "SaleOptOutNotice": 1,
            "SharingOptOutNotice": 1,
            "SensitiveDataLimitUseNotice": 2,
            "SaleOptOut": 0,
            "SharingOptOut": 0,
            "SensitiveDataProcessing": [
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0
            ],
            "KnownChildSensitiveDataConsents": [
                0,
                0
            ],
            "PersonalDataConsents": 0,
            "MspaCoveredTransaction": 0,
            "MspaOptOutOptionMode": 0,
            "MspaServiceProviderMode": 0,
            "GpcSegmentType": 1,
            "Gpc": false
        }
    }
}

The important things to note here is that the fields related to data_sales_and_sharing "Notice" are marked as 1:

"SaleOptOutNotice": 1, meaning notice was provided

"SharingOptOutNotice": 1, meaning notice was provided

And the other *Notice fields are marked as 2, meaning notice was not provided (since they don't fall under data_sales_and_sharing which is the only notice we have enabled

[ ] Now opt in or out of all notices. If you opt out, if you issue the ping command again, you should get a result like

{
    "gppVersion": "1.1",
    "cmpStatus": "loaded",
    "cmpDisplayStatus": "hidden",
    "signalStatus": "ready",
    "supportedAPIs": [],
    "cmpId": 407,
    "sectionList": [
        8
    ],
    "applicableSections": [
        8
    ],
    "gppString": "DBABBg~BWUAAAAA.QA",
    "parsedSections": {
        "uscav1": {
            "Version": 1,
            "SaleOptOutNotice": 1,
            "SharingOptOutNotice": 1,
            "SensitiveDataLimitUseNotice": 2,
            "SaleOptOut": 1,
            "SharingOptOut": 1,
            "SensitiveDataProcessing": [
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0
            ],
            "KnownChildSensitiveDataConsents": [
                0,
                0
            ],
            "PersonalDataConsents": 0,
            "MspaCoveredTransaction": 0,
            "MspaOptOutOptionMode": 0,
            "MspaServiceProviderMode": 0,
            "GpcSegmentType": 1,
            "Gpc": false
        }
    }
}

Note that SaleOptOut and SharingOptOut are now both 1, and the gppString has changed slightly (can be decoded here)
Pre-Merge Checklist

[ ] All CI Pipelines Succeeded
Documentation:

[ ] documentation complete, PR opened in fidesdocs
[ ] documentation issue created in fidesdocs

[ ] Issue Requirements are Met
[ ] Relevant Follow-Up Issues Created
[ ] Update CHANGELOG.md
[ ] For API changes, the Postman collection has been updated

This is looking great. Just want to confirm that you have everything answered that's needed from @mfbrown ?

@mfbrown and I met today and we should be good here! some takeaways...

known child consents + personal data consents have been separated out into their own notices
known child consents + sensitive data are often bitfields which we decided to continue handling the way we have it now (either setting the whole array to 1 or 2) though this may change in the future


Can't pass workers_per_resource to the bentoml container

Describe the bug
I have a machine with two GPUs, I run the model with openllm start command and everything went well.
CUDA_VISIBLE_DEVICES=0,1  TRANSFORMERS_OFFLINE=1  openllm start mistral --model-id  mymodel  --dtype float16  --gpu-memory-utilization 0.95 --workers-per-resource 0.5

there are two process appear on the two GPUs in this case one for the service and another for ray instance.

when I run start command without  --gpu-memory-utilization 0.95 --workers-per-resource 0.5, only one GPU is running the service and CUDA out of memory is occured.

When I build the image and follow the steps to create container, however when i run the docker image, it issue error of cuda out of memory, such as the second case without passing these args: --gpu-memory-utilization 0.95 --workers-per-resource 0.5
steps:

openllm build mymodel --backend vllm --serialization safetensors
 bentoml containerize mymodel-service:12345 --opt progress=plain
docker run --rm --gpus all -p 3000:3000 -it mymodel-service:12345

To reproduce
No response
Logs
No response
Environment
$ bentoml -v
bentoml, version 1.1.11
$openllm -v
openllm, 0.4.45.dev2 (compiled: False)
Python (CPython) 3.11.7
System information (Optional)
No response

@aarnphm What is the difference between the previous two cases, so the first case can launch two processes one for ray worker and other for bentoml service (that when using --gpu-memory-utilization 0.95 --workers-per-resource 0.5

Same issue: https://github.com/bentoml/OpenLLM/issues/872

close for openllm 0.6


vmss: support eviction policy for low priority scaleset

[x] The PR has modified HISTORY.rst describing any customer-facing, functional changes. Note that this does not include changes only to help content. (see Modifying change log).

[x] I adhere to the Command Guidelines.

View a preview at https://prompt.ws/r/Azure/azure-cli/6145
This is an experimental preview for @microsoft users.


Get rid of OperatingSystem enum

Currently, the OperatingSystem enum is bloated with values that are practically not used elsewhere in the launcher. This includes constants for specific Windows versions as well as other Unix-like systems. We may store the OS information with the newer Config API if required and remove this enum completely.

We have a slimmed down version of that as https://github.com/MovingBlocks/TerasologyLauncher/blob/948f229cca152efe61929f3e58d2c1290f1cb956/src/main/java/org/terasology/launcher/util/Platform.java now, which is needed to check for MacOS specific settings and such.
Also, the config package has been removed.


Fix Metaclass usage for Python 3

Fix the way SqlAlchemyItemMeta is used in SqlItem to be compatible with
Python 2.x and 3.x.

For anybody else having this issue, a workaround until this PR can be merged is to import SqlAlchemyItemMeta from scrapy_sqlitem.sqlitem and use, e.g.
class MyItem(SqlItem, metaclass=SqlAlchemyItemMeta)`:
     ...


Contratiction in TraceRedisAutoConfiguration with regards to the Sampler class

There is a contradiction in TraceRedisAutoConfiguration when it comes to the brave.sampler.Sampler.
In line 61, the brave.sampler.Sampler is used in a method signature. That means that it has to be on the classpath for the class loader to be able to load the LettuceConfiguration class.
However, line 67 attempts to instruct Spring to create a different bean for when brave.sampler.Sampler is missing from the classpath.
I'm not 100% sure on this is supposed to work but my guess is, that the @ConditionalOnClass and @ConditionalOnMissingClass have to be moved to class level on two separate LettuceConfiguration classes, something like this:
@Configuration(proxyBeanMethods = false)
@ConditionalOnProperty(value = "spring.sleuth.redis.enabled", matchIfMissing = true)
@ConditionalOnBean(Tracer.class)
@ConditionalOnClass(Tracing.class)
@AutoConfigureBefore({ RedisAutoConfiguration.class })
@AutoConfigureAfter(BraveAutoConfiguration.class)
@EnableConfigurationProperties(TraceRedisProperties.class)
public class TraceRedisAutoConfiguration
{
    @Configuration(proxyBeanMethods = false)
    @ConditionalOnProperty(value = "spring.sleuth.redis.legacy.enabled", havingValue = "false", matchIfMissing = true)
    @ConditionalOnBean(Tracing.class)
    @ConditionalOnClass(name = "brave.sampler.Sampler")
    static class BraveLettuceConfiguration
    {
        @Bean
        TraceLettuceClientResourcesBuilderCustomizer braveTraceLettuceClientResourcesBuilderCustomizer(
            Tracing tracing, Sampler sampler)
        {
            eagerlyInitializePotentiallyRefreshScopeSampler(sampler);
            return new TraceLettuceClientResourcesBuilderCustomizer(tracing);
        }

        private void eagerlyInitializePotentiallyRefreshScopeSampler(Sampler sampler)
        {
            sampler.isSampled(0L);
        }
    }

    @Configuration(proxyBeanMethods = false)
    @ConditionalOnProperty(value = "spring.sleuth.redis.legacy.enabled", havingValue = "false", matchIfMissing = true)
    @ConditionalOnBean(Tracing.class)
    @ConditionalOnMissingClass("brave.sampler.Sampler")
    static class FearfulLettuceConfiguration
    {
        @Bean
        TraceLettuceClientResourcesBuilderCustomizer otherTracersTraceLettuceClientResourcesBuilderCustomizer(
            Tracing tracing)
        {
            return new TraceLettuceClientResourcesBuilderCustomizer(tracing);
        }
    }
}

Have you seen any exceptions because of this? It would be great to see a sample or a test case that replciates this issue?

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.


added getAsker and getOfferer method in booking

Q
A

Branch?
master

Bug fix?
no

New feature?
yes

Tests pass?
no

Fixed issues

License
MIT

This PR adds somes helpers to get offerer and asker of booking directly

Hi,
This modification will be added when calls to these methods will replace current methods calls.
Thanks!


Javascript API: FS events

I would like to monitor folders / files for changes, an reflect those changes in my applications.
Node.js have nice functions to watch over files and directory changes, which is impossible in Tauri's current Javascript API.
https://nodejs.org/api/fs.html#fs_class_fs_fswatcher
Node.js' way of doing is returning an object emitting event on changes, something like that for Tauri would be a nice addition.
There is also the same stuff for file's stats, which would be appreciated in the same time :
https://nodejs.org/api/fs.html#fs_class_fs_statwatcher
But likely would depends on #996

Following the explanation on #996, I believe this should be a community plugin, specially since it's a niche use case and brings additional dependencies.


Add set -euo pipefail

Add euo pipefail to all non-trial shell scripts, note if x is already set we maintain it.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 83.225%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
19066

Relevant Lines:
22909

💛  - Coveralls

Oh, interesting. This PR broke CI but CI didn't run on it?

Frustrating. Ok, cool, thanks for fixing it.


1.20.1 New player unable to get Blind Diversion advancement

Modloader
Fabric
Minecraft version
1.20.1
Hex Casting version
0.11.1-7-pre-634
Modloader version
Fabric: Loader 0.15.9 + API 0.92.1+1.20.1
Modpack info
https://github.com/walksanatora/HexxyTestPack/blob/main/modpack.zip
The latest.log file
No response
Issue description
I looked but didn't see this bug listed yet.  Apologies if I missed it.  A fresh player using the 1.20.1 builds for hex casting (at least on build 634) cannot get the Blind Diversion advancement or further.  This blocks them from the ability to use health for hexcasting.
When a player attempts to cast a great spell, the great spell just works, instead of fizzliing out like in 1.19
Steps to reproduce

Start a new single player or server world
Cast a great spell

Expected: Get Blind Diversion and need to take 19 damage in a single spell to get the next advancement
Actual: Great Spell succeeds, no advancement achieved.
Other information
No response

Thanks for reporting thIs @starji; I'd been meaning to report it but once you have a workaround like the datapack we've been using on Chloe's server it's easy to let these things slide.
Yeah we've been working around this one for a while. About time someone thought to report it and followed through on the thought slightly embarrassed grin

this no longer occurs in development. no clue when/where it was fixed (tested in forge/fabric build 648)

can't reproduce on latest build. tried on server as well just to make sure.
also no idea when/how/where this got fixed? best guess is some addon was causing issues?


Update cpputil to no mesh

Adds the C++ part of no mesh

This is not a PR to master but to a branch which will only be merged after everything is ported. The idea is to do partial reviews to make it easier to follow.


Key Pair didn't work - UNKNOWN_ALGORITHM

It seems PKCS8 is not supported?  It was the default suggested by Snowflake when following their instructions here:
https://docs.snowflake.com/en/user-guide/key-pair-auth.html
Error output from VS Code is:
[1661205581755] INFO  (ls): Connection instance created for SFKeyPair.
ns: "conn-manager"
[1661205581756] ERROR (ls): Connecting error: {"code":-32001,"data":{"driver":"Snowflake","driverOptions":{}},"name":"Error"}
ns: "conn-manager"
[1661205581757] ERROR (ext): ERROR: Error opening connection error:13000077:PKCS8 routines:OPENSSL_internal:UNKNOWN_ALGORITHM, {"code":-32001,"data":{"driver":"Snowflake","driverOptions":{}}}
ns: "error-handler"

I was able to work around this by converting my encrypted private key to an unencrypted version, and configuring the connection to use that:
openssl pkcs8 -in rsa_key.p8 -topk8 -out rsa_key_unencrypted.p8 -nocrypt

When I try @alexnorton solution I get this other error:
Could not read key from my_rsa_key.p8
8045180501000000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:crypto/store/store_result.c:151:
8045180501000000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:341:Global default library context, Algorithm (PBKDF1 : 0), Properties ()


shifted language around end-users.

@Identitywoman : Thanks for this improvement. We should merge it quickly. However, it fails our DCO check because it consisted of an unsigned commit. I'm sorry for that formalism getting in the way of simple stuff, but it is the nature of the beast when lawyers get careful. :-) Two options to fix:

You can follow the instructions here

You can authorize me or Kyle or someone, here in this comment stream, to update the commit on your behalf, adding your DCO attestation. Since this creates a written record, it provides the necessary documentation without forcing you to jump through quite so many hoops.

Just say the word, and we'll merge this under option 2. Or feel free to update under option 1.

PR #301 is a cherry-pick of @Identitywoman 's commit from this PR (d7679f1), minus all the cruft that was bogging this down. I modified the commit message to include Kaliya's DCO signoff as listed in the comment stream, except that I had to use the same email address that is used for the author's identity (the one in the github account).
Since PR 301 has been merged, this PR is now redundant and can be closed.


[Bug]: DNN docs link point to wrong demain name

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
The official url for DNN documentation should be https://docs.dnncommunity.org/ in the persona bar instead of dnndocs.com
Steps to reproduce?

Hover the DNN icon in the persona bar
Inspect the link to the docs

Current Behavior
Links to dnndocs.com
Expected Behavior
Should link to https://docs.dnncommunity.org/
Relevant log output
No response
Anything else?
No response
Affected Versions
9.13.7 (alpha)
What browsers are you seeing the problem on?
Chrome
Code of Conduct

[X] I agree to follow this project's Code of Conduct

The dnndocs.com domain does a 301 redirect to docs.dnncommunity.org, but yeah, it should probably be updated.  😉👍🏼


Add all fields to span name

I'm not sure if this is desirable in its current form, but this is super useful for me. I'm open to input for how to improve it.

Thanks for the contribution!
I was thinking I wanted to add fields at some point and I like that using FormattedFields makes this code simpler.
Is it important that the field values be in the name of each span for your application? I was thinking we could put them in the "args" object for each span enter and exit event. This is where I put the file and line number for each span.
Also, an option to enable/disable including fields would be good, as they can make the resulting file much larger.

I was motivated to make this PR because of a problem with naming Spans in the bevy game engine. It schedules a bunch of functions to run at different times based on structs that are derived for functions. These structs contain the name of the function, obtained by std::any::type_name<T>() -> &'static str where T is the type of the function, and it's stored in a Cow<'static, str>. As tracing Span names must be constants (it seems), I have had to add the name of the function being executed as a field on the span. That then leads to this PR because a lot of the interesting Spans added to bevy's code have a name field and visualising the traces is only really useful by adding the field into the name, otherwise all the spans are called 'system' or 'stage'. So, I'm trying to see if there's a compile-time way to automatically instrument all systems.

@superdump I've just published 0.3.0 with a new name_fn option which should enable your use case. Let me know if it works for you.

Works like a charm! Thanks!


Add PHP CS Fixer to the CI and fix the existing code

The config is based on the API Platform one.

Looks weird that the tests are failing now; is it rebased against master?

Yes, the problem is related to strict types. The code base isn't currently compatible with this mode.


Vendor publish conf is not working

Hey guys,
I've just installed the package and wanted to get the config using the command as is mentioned in readme:
php artisan vendor:publish --tag='youtube-config'

but I'm getting the error: Unable to locate publishable resources..
Any idea about it?

I have same problem.

@milosb793 @SwierczKacper  Sorry for the late response.
this is fixed in v1.0.3 of this package.


Update pytuya to tinytuya

I had problems getting pytuya to run with the latest version of pytuya. I noticed it was deprecated and it appears tinytuya is recently updated and well maintained. It is mostly a drop-in replacement as well.
I did also change the update time to 10 seconds instead of 30. Ideally, I'd break that out into a configuration file, but I'm a bit time constrained at the moment.

Thanks, I'll merge it.


update some page

update Recently Written Card background
update Library background color and padding top
add title & fetch own logbook
run npm run format

19 commits behind HEAD and 3 commits ahead, better do a rebase now?


Fix: Responsive Iframe for Smaller Devices

This pull request addresses the issue of the iframe being unresponsive on smaller devices due to its hardcoded width. The iframe's width is now set to 100%, with a maximum width of 850px and an aspect ratio of 16:9. This makes the iframe responsive and improves the user experience on mobile devices.
Fixes

Fixes issue #3000

Changes Made

Set the iframe width to 100% to allow it to dynamically adjust based on the container's width.
Set the maximum width to 850px to maintain readability and structure on larger screens.
Added an aspect ratio of 16:9 to ensure the iframe scales properly on all screen sizes.

Screenshots

BEFORE
AFTER

Checklist

[x] Changed the iframe width to 100% to ensure responsiveness.
[x] Set the max-width to 850px for large screens.
[x] Added an aspect ratio of 16:9 for proper scaling.
[x] Tests have been updated to cover the changes.
[x] Code follows the established coding style guidelines.

Please review and merge. Thank you!

Solves Issue #3000


Should move repo to azure/azure-functions-kafka-extension

Not urgent but just tracking, we have the code for all other extensions under the "azure" org instead of the "microsoft" org.  Would be good to move this over the "azure" sometime.

My thinking was that once this became an "official" Azure supported extension available in Consumption plan then it would go live there. Thoughts? Happy either way.

I'm ok with that - the only potential issue is that people who are looking to file an issue / contribute may be used to the azure/azure-functions-x-extension pattern and won't find this repo right away, but it's not a huge deal to me either way

if azure/azure-functions-x-extension is the pattern, then the CosmosDB extensions are in the wrong place b/c there is no azure/azure-functions-cosmosdb-extension … that's under Azure/azure-webjobs-sdk-extensions/tree/dev/src/WebJobs.Extensions.CosmosDB

Correct - we have a work item to break out Event Hubs, Service Bus, CosmosDB, etc. to seperate repos to follow pattern. More historic with them as they were built as WebJobs extensions and not Azure Functions extensions. https://github.com/Azure/azure-webjobs-sdk/issues/2011

Have started the process to move this. Will close once repo has been moved.

And we're done.


Container app fail to provision from ACR

[Provisioning step fails and there is no logs to check what failed]
When I try to create a container app running an image that I have on ACR it shows in the revision management tab that the status of the provisioning is "Failed". I can't see any logs or anything so I did not find a way to see what actually failed. I followed the steps described in the doc either on the portal or using azure cli and the result is always the same. Is there a way to check what failed and why?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 31e7a872-ea31-d8c0-d611-136c4a2eb139
Version Independent ID: 878b52bb-65aa-bc9e-8051-5a9230054735
Content: Quickstart: Deploy an existing container image with the Azure CLI
Content Source: articles/container-apps/get-started-existing-container-image.md
Service: container-apps
GitHub Login: @craigshoemaker
Microsoft Alias: cshoe

charlesleaub, No worries. Thanks for taking time to post this question on Azure docs feedback channel.
Much appreciate your follow-up and closure.
-As a side note, this channel is reserved for ‘Azure docs’ feedback- for sharing doc feedback and suggesting content improvements.


For angular patternfly downgrading lodash

For whenever we can go back to 4.x, sum will have to be changed back to sumBy

@miq-bot add_label euwe/no, bug

Why are we downgrading?

Angular Patternfly still requires lodash 3.10. Soon Angular Patternfly 4.x will release with upgraded dependencies including lodash 4.x.


Revert use-cythonize branch

Current install script has some bugs.
This PR reverts #1755.

LGTM

The reverted PR is not #1755, but #1775.


ES6 feature: Modules: Tracking Issue

In order to land modules, we must:

[ ] Land the Classes feature
[ ] Review and Test Token Generation
[ ] Review Error Messages
[ ] Fix #311

The espree fork may land it shortly as well, so we have the benefit of those dependencies as well https://github.com/eslint/espree/pull/43

@ariya can you confirm that this is the known set of tasks for modules? cc @caridy

LGTM

We need to provide a structure/template for feature inclusion. A possible example is https://code.google.com/p/esprima/issues/detail?id=621.

@ariya fair enough. I'll submit a PR for a template. In this case, feel like modifying the description to also include the BNF description of Module Syntax?

@ariya: https://github.com/jquery/esprima/pull/315

@ariya added maintainers guide with section on proposing features for inclusion: https://github.com/jquery/esprima/wiki/maintainers-guide Let me know if that looks right, and if so, I'll fix up this description and the one for classes.

Closed in favor of #1000


fix: display parallel test suite steps nicely

Changes

Show properly test suite steps in the Execution Details drawer
Show a warning message on the form for editing test suite steps, redirecting to YAML edition

Fixes

https://github.com/kubeshop/testkube/issues/4139

How to test it

Check it here

Screenshots

Checklist

[x] tested locally
[ ] added new dependencies
[ ] updated the docs
[ ] added a test

Looks ok for me, although didn't test with execution result with parallel tests. Approved, but found one small bug, probably not related to PR, but should be fixed. On test suite edit, when I make some updates to step, then click cancel, list is actually cleared out, while should be returned in a default state

[...] found one small bug, probably not related to PR, but should be fixed. On test suite edit, when I make some updates to step, then click cancel, list is actually cleared out, while should be returned in a default state

Thank you @pavloburchak, I fixed this bug now in this PR too 👍


What type of update are you requesting?
A change to the Package Identifier
Current Package Identifier
SomePythonThings.WingetUIStore
Package Version
All
Please describe the changes you would like to see
Suggested new package identifier:
MartiCliment.UniGetUI
Motivation:

The official publisher and author of the app is Martí Climent, not "SomePythonThings".
The official name of the app is UniGetUI, not WingetUI.

Compatibility:
Anyone searching for the old name (winget search wingetui) will find the new name, since the package name and tags already have the old name:
https://github.com/microsoft/winget-pkgs/blob/f490e2c726a9aec4497822708c181673f2c0f6e8/manifests/s/SomePythonThings/WingetUIStore/3.1.1/SomePythonThings.WingetUIStore.locale.en-US.yaml#L11
https://github.com/microsoft/winget-pkgs/blob/f490e2c726a9aec4497822708c181673f2c0f6e8/manifests/s/SomePythonThings/WingetUIStore/3.1.1/SomePythonThings.WingetUIStore.locale.en-US.yaml#L30-L31
Additional information:
Official repository (the app is written in C#, not Python):
https://github.com/marticliment/UniGetUI/
Official website (mentioned in the repository):
https://www.marticliment.com/unigetui/
This is referenced in the manifest file itself:
https://github.com/microsoft/winget-pkgs/blob/f490e2c726a9aec4497822708c181673f2c0f6e8/manifests/s/SomePythonThings/WingetUIStore/3.1.1/SomePythonThings.WingetUIStore.locale.en-US.yaml#L7-L16

@marticliment would you agree with this change, or would you suggest leaving it as is?

Yes, it definitely makes sense to make the change

@marticliment do you think we should remove previous WingetUI in total and just start the packages from the beginning not including any previous version of WingetUI (i.e. only including UniGetUI), or should we keep WingetUI here, and just create a new one for UniGetUI?

Ideally it would be better to follow option 1, to prevent confusion, but I don't know if it is feasible and acceptable to delete an entire package and then publish it again with different names. Having that said, either if it ends up being option 1 or 2 it will be ok for me.

@stephengillie, would you suggest option 1 or 2 based on Marti's feedback?

I agree with @ marticliment that the first option is better. While it would be possible for us to continue offering WinGetUI as a historical record, I'm not sure how much demand there is for it.
A third possible option could be leaving the manifests before the name change as-is, and only moving the post-rename manifests. This could reduce workload while also preserving the older manifest versions.

[!note]
Moving packages from one PackageIdentifier to another can be a lot of work. Foreach manifest version, 2 PRs are needed - one to add new and one to remove old. This process always winds up having some level of coordination between the submitter and a moderator to approve, and with the appropriate preparation can go very smoothly.

I agree with @ marticliment that the first option is better. While it would be possible for us to continue offering WinGetUI as a historical record, I'm not sure how much demand there is for it.
A third possible option could be leaving the manifests before the name change as-is, and only moving the post-rename manifests. This could reduce workload while also preserving the older manifest versions.
Note
Moving packages from one PackageIdentifier to another can be a lot of work. Foreach manifest version, 2 PRs are needed - one to add new and one to remove old. This process always winds up having some level of coordination between the submitter and a moderator to approve, and with the appropriate preparation can go very smoothly.

So what I will do is follow what you suggested which is to create the new manifest then remove the old one. 😊

I will continue the rest when these commit just to make sure everything is correct

Assuming we chose to delete the old manifests and just start from scratch (Option 1), would it be possible to delete all the manifests in one single PR? Or we'd still need to do one manifest at a time?

I don't think WingetUI is still in demand since most people always want the latest version, which in this case is UniGetUI

I agree,  and all the installers for older versions are still (and will be) available on GitHub, so installing older versions shouldn't be an issue.

Assuming we chose to delete the old manifests and just start from scratch (Option 1), would it be possible to delete all the manifests in one single PR? Or we'd still need to do one manifest at a time?

They will have to be removed in separate PRs one-by-one but that is easy with Komac, the hardest part is transferring all of them because they have to be done manually

When these are committed I will start removing all the packages from SomePythonThings.WingetUI

Multiple PRs have been scheduled so that this can be completed

Not entirely finished, still a bit more to go

This has officially been completed, I thank you all for your patience. :)


CB-11077 backbutton does not minimize app

Jira issue

It is mandatory to implement this kind of backbutton handling by default due to  Windows Store Policies. My last app submission was declined because this bahaviour was not implemented. The problem occurs only on<EMAIL_ADDRESS>where back button handling was introduced.

@beat-at-hartwig, you can handle both first page and sub page cases in backbutton event and do throw or history.back(); correspondingly.
This can be used as a temporary measure but I agree that the behavior isn't consistent with Android now, which has the correct behavior out of the box.

@daserge Thank you for your suggestions. I already implemeted similar code fitting my needs. Doing a throw was a nice hint. Thanks!

@nikhilkh, I've updated the PR, please take a look once more.


[Bug] Audio channels bug

[x] I didn't find a similar issue already open.
[x] I read the documentation (README AND Wiki)
[x] I have installed FFMpeg
[x] My problem is related to Spleeter only, not a derivative product (such as Webapplication, or GUI provided by others)

Description
When use the command:
python3 -m spleeter separate -p spleeter/resources/2stems.json -o output ../Audio/BeiJingWelcome.mp3

it will raise a ValueError:
ValueError: Cannot feed value of shape (10978, 2, 7) for Tensor 'mix_stft:0', which has shape '(None, 2049, 2)'

The reason is that there is an error in the source code
spleeter/spleeter/separator.py

line 264, change:
stft = stft[:, :2]

into:
stft = stft[:,:,:2]

then it works well.

Hi @AaronPeng920,
I cannot reproduce your bug with the latest version of spleeter.
If you're using a spleeter version installed through pip, the only reason I can see that may result in such a bug is that you'd be using a custom configuration file spleeter/resources/2stems.json that would contain incorrect parameters.


Convert legacy JS files into Typescript files

Requirement

Convert caucus/src/component/TextChat/ChatMessage.jsx into a .tsx file

Convert CodeMirrorAdapter.js into a ts file.

This issue is not difficult, but a good knowledge of typescript patterns is required.

Hi @Rishabh-malhotraa, I would like to work on this issue if it is still available.


Avoid only exact duplicates when creating the accepted CA names list

This avoids situations where third party client is unable to recognize
that the client certificate was issued by the same CA with name differring
only by case or insignificant characters.
Fixes #4512

I've amended the commit to fix the comment formatting.


[Coverity CID :187821] Memory - illegal accesses in /subsys/bluetooth/host/hci_core.c

Static code scan issues seen in File: /subsys/bluetooth/host/hci_core.c
Category: Memory - illegal accesses
Function: bt_id_delete
Component: Bluetooth
CID: 187821
Please fix or provide comments to square it off in coverity in the link: https://scan9.coverity.com/reports.htm#v32951/p12996

Please check Coverity status before filing issues. This was already flagged as a false-positive.


[Bug] RediSearchIndexer.createIndexFor method use hardcoded entity prefix as class name making RedisMappingContext and FallbackKeySpaceResolver unusable.

spring-data-redis allows to define custom RedisMappingContext bean with configurable fallbackKeySpaceResolver; this is useful when the requirement is to prefix/suffix every redis key with some custom value e.g. environment/service name in shared redis instance. This can be configured as below bean definition;
  @Bean
  @Primary
  public RedisMappingContext keyValueMappingContext(
      @Value(value = "${redis.custom.keyspace.prefix}") String keyspacePrefix) {
    RedisMappingContext mappingContext = new RedisMappingContext();
    mappingContext.setFallbackKeySpaceResolver(type -> keyspacePrefix + ":" + type.getSimpleName());
    return mappingContext;
  }

What this bean is supposed to do is every spring-data-redis operation will use fallback keyspace response and prefix the keyspace with some configurable value. As an example if we are trying to store/lookup com.example.Company entity instead of the key being com.example.Company:ID1 it would be <prefix>:Company:ID1.
This works for standard spring-data-redis however not for redis-om-spring. I did some investigation and to me this looks like a bug with RediSearchIndexer.createIndexFor(class) method. It has hardcoded logic to always use entity classname as prefix in json schema creation. RediSearchIndexer already has an instance of RedisMappingContext and straightforward fix for this would be to add the following lines for entityPrefix logic;
      String entityPrefix = cl.getName() + ":";
      if (mappingContext.hasPersistentEntityFor(cl)) {
        RedisPersistentEntity<?> persistentEntity = mappingContext.getRequiredPersistentEntity(cl);
        entityPrefix = persistentEntity.getKeySpace() != null ? persistentEntity.getKeySpace() + ":" : entityPrefix;
      }

I have created reproducible example in this repository. I am happy to get inputs on this and also contribute the fix if agreed.

A PR will be most welcomed!

Ok I will apply the fix and open PR


Wi-Fi Network Diagnostics Cluster attributes with default value "null" showing response as "expected to be null but isn't"

Summary:
As per spec Wi-Fi Network Diagnostics Cluster some  attributes with default value "null" showing response as "expected to be null but isn't"  when we executed
Spec screenshot:

Execution logs and screenshot:

wifidiag.txt

@manjunath-grl @cjandhyala Is this still an issue?

@bzbarsky-apple issue is resolved can you please close it.


[Question] How to avoid duplicate queries when one is running

In my case, I want to avoid duplicate queries when one is running.
Such as, when user open a dashboard:

A long query(3mins) for a chart is running
Another one trigger a same query when 1) is still running
result: there will be two same queries sent to database.

How to avoid duplicate queries in this case? Thanks.

This is a known limitation in the current architecture. However, this should be resolved once Global Async Queries become generally available: #9190 . If you feel like testing the feature, feel free to enable the GLOBAL_ASYNC_QUERIES feature flag. See the CONTRIBUTING.md documentation for details: https://github.com/apache/superset/blob/master/CONTRIBUTING.md#async-chart-queries

This is a known limitation in the current architecture. However, this should be resolved once Global Async Queries become generally available: #9190 . If you feel like testing the feature, feel free to enable the GLOBAL_ASYNC_QUERIES feature flag. See the CONTRIBUTING.md documentation for details: https://github.com/apache/superset/blob/master/CONTRIBUTING.md#async-chart-queries


Develop E2E Tests

This is a proposal to develop simple end to end (E2E) unit tests by simply using the current (and future) examples/.
The design is as follows:

Spin up a global instance of headless chromium
Write unit tests in selenium python for each example. These will basically just make sure that each example runs as expected.

The unit test "framework" will:

Compile each example with cargo web deploy
Run the example specific test defined in examples/<EXAMPLE>/tests which will test against target/deploy/index.html using the global headless browser.

I had a similar framework with artifact so I can go ahead and start on this if @DenisKolodin is okay with the design.

with the examples all combined into a "single example" this will be much easier. Good job!

I think I will be doing this for artifact using https://github.com/jonhoo/fantoccini (selenium for rust basically) within the next month or so. Once I've got the hang of it I could do the same treatment to yew.
I think this would help development and stability a lot. Let me know what you think

I think this would be great! Were you able to make any progress on this @vitiral? Would you like to continue to work on this? If not, I'd be happy to do this after I finish the DevTools (if I'm ever actually able to get them to work).

Glad you like the idea! No I won't be able to work on it as I haven't been doing web-dev lately and have a million other projects, including a third child to be born soon :D

No worries :) I'll try and work on this if I have some time.


Download Kubernetes

@samlove8611  Hi, Could you please share the details on the issue you would like to report or the assistance you require? Kindly provide details in the https://github.com/kubernetes/website/issues/44428#issue-2048341979 of the issue.
/triage needs-information

Closing this issue, as no information is provided. If you have any concerns, please reopen it with the details in the description.
/close


Sep 22nd 2020 - Currency Ref Data Meeting Minutes

Currency Reference Data Meeting
Date
09 22 2020 - 10am EST / 3pm BST
Untracked attendees

Fullname, Affiliation, (optional) GitHub username
...

Agenda

[ ] Convene & roll call (5mins)
[ ] Approve Sep 8th meeting minutes
[ ] Update on Cloud Storage solution

Proposal to consider Git Large File Store: https://git-lfs.github.com/

[ ] Review progress on Project Plan
[ ] AOB, Q&A & Adjourn (5mins)

Decisions Made

[ ] Decision 1
[ ] Decision 2
[ ] ...

Action Items

[ ] Action 1
[ ] Action 2
[ ] ...

WebEx info

WebEx Meeting URL
Meeting Number:<PHONE_NUMBER>
Join by phone: +1.415.655.0003 USA Toll | +44-20319-88141 UK Toll | Access code:<PHONE_NUMBER>
Password: VXcd2eQgh82
Global call-in numbers

Present

Hi

Hi

Present

Anseo

hi

Hello!


Use latest shared types

Changes in this PR
The latest API changes change some previously optional API type fields to required, causing the UI to fail to compile. This PR addresses this
Release checklist
Release process documentation.
Pre-merge checklist

[ ] Are any changes required to dependent services for this change to work?
(eg. CAS API)
[ ] Have they been released to production already?

Post-merge checklist

[ ] Manually approve
release to test
[ ] Manually approve
release to preprod
[ ] Manually approve
release to prod

 Use latest shared types


sizes not working properly on mobile native

I am trying to use sizes to specify the height of a Pressable. It works fine on the browser, but I can't seem to make it work for mobile native other than inlining the value itself.
// theme.ts
export const theme = makeTheme({
  /* ... */
  sizes: {
    $0: 0,
    $1: 4,
    $2: 8,
    $3: 12,
    $4: 16,
    $5: 20,
    $6: 24,
    $7: 32,
    $8: 40,
  }
  /* ... */
})

// Button.tsx
const Button = ({ children, variant = 'primary' }) => (
    <Pressable
      sx={{
          variant: `buttons.${variant}`,
          justifyContent: 'center',
          backgroundColor: 'red',
          height: '$8'
       }}>
        {children}
</Pressable>
)

Browser output:

iOS Simulator error:

Am I doing something wrong?

yeah something is probably wrong here. can you try making sx a function of your theme and log it?

any luck?


Using all observations after the pipeline for downstream analysis

Hello,
I am trying to run the pyInfinityFlow pipeline and want to have all the observations.
It seems with the settings there is no clear option to have all the observations after running the pipeline.
Would you recommend to increase the ratio_for_vaildation to 1.0?
Current settings:
pyInfinityFlow --data_dir /...
--out_dir /output
--backbone_annotation backbone_selection_file.csv
--infinity_marker_annotation test.csv
--random_state 7
--use_logicle_scaling True
--normalization_method "zscore"
--n_events_train 0
--n_events_validate 0
--ratio_for_validation 0.3
--separate_backbone_reference None
--n_events_combine 0
--n_final 0
--add_umap True
--find_clusters True
--find_markers True
--make_feature_plots True
--use_pca True
--n_pc 15
--n_pc_plot_qc 50
--save_h5ad True
--save_feather True
--save_file_handler True
--save_regression_models True
--verbosity 1
--n_cores 40
Thank you for your help
Kevin

Hello!
By "have all the observations after running the pipeline" I assume you mean the number of events in the output InfinityFlow Object .fcs file.
The --n_events_combine 0 parameter is meant to tell the program to use all of the events from all of the InfinityMarker files to pool together. So for each input .FCS file, all events will be taken, with their backbone features, and the InfinityMarker values will be imputated across all observations. So by including this, you should be getting all cells from every input file in the final FCS output file.
Are you seeing fewer events in the final InfinityFlow Object than expected when you run the command? - If that is the case I will try to debug issues that prevent the expected behavior
The --ratio_for_validation doesn't apply to the final output object. This parameter is used for the purpose of building/validating the regression models. It is the ratio of observations that are withheld from regression model fitting. The purpose of these events is to test how well the regression model could predict the true value. Because these events were not used to build the model, it provides an unbiased input from which we can take the predicted InfinityMarker value, and calculate how far off it is from the real recorded value. I would recommend not increasing this above 0.5 (50% of the cells withheld). This is because I would prefer to give the model more information to help make it more accurate. This is especially important if you have a lot of rare cell populations in your sample. You want to try to have enough cells to fit the model to describe all of the cell populations in your sample. I would say it is more common to use 20% of the data for validation and 80% for model fitting (--ratio_for_validation 0.2). However, since the original InfinityFlow package had used a default of 50% for each, I had used that in the documentations example.

Hello,
Thank you for the response!
I ran my analysis with the settings I shared and got a subset of cells as an output of the pipeline.
My original amount of cells combined is 24988815
When I ran with these settings, it gave 8036977 cells.
So with this, it feels as if it took the ratio of cells that I stated in  --ratio_for_validation  0.3

I see. The program is only pooling from the validation set then.
I can add in an option to take cells from both the validation and fitting data sets, but it will take a little time.
An alternative approach you can take now would be to pool all of your input data into one FCS file. Then use this concatenated file for the --separate_backbone_reference. I think most FCS analysis software (Eg. Flowjo) can concatenate FCS files, as long as the channel names are the same for the backbone markers.
Would you try this approach?

Hello,
It seems that I am running into a error if I try to use the reference.
pyInfinityFlow --data_dir /mnt/
--out_dir /mnt/
--backbone_annotation /mnt.csv
--infinity_marker_annotation .csv
--random_state 7
--use_logicle_scaling True
--normalization_method "zscore"
--n_events_train 0
--n_events_validate 0
--ratio_for_validation 0
--separate_backbone_reference /mnt/beegfs/userdata/k_mulder/PBMC_Legendscreen/run_3/concat_Fj_Legendscreen_3_1.fcs
--n_events_combine 0
--n_final 0
--add_umap True
--find_clusters True
--find_markers True
--make_feature_plots True
--use_pca True
--n_pc 15
--n_pc_plot_qc 50
--save_h5ad True
--save_feather True
--save_file_handler True
--save_regression_models True
--verbosity 1
--n_cores 20
This gives the following error:
Running InfinityFlow from single directory, with all input .fcs files using the same channels for backbone...
Traceback (most recent call last):
File "/mnt/beegfs/merfish/.conda/envs/pyInfinityFlow/bin/pyInfinityFlow", line 8, in
sys.exit(main())
File "/mnt/beegfs/merfish/.conda/envs/pyInfinityFlow/lib/python3.8/site-packages/pyInfinityFlow/pyInfinityFlow.py", line 204, in main
raise ValueError("separate_backbone_reference and n_events_combine can not be "
ValueError: separate_backbone_reference and n_events_combine can not be used together. One must be None.
And then replaced with none it gives the following error:
usage: pyInfinityFlow [-h] --data_dir DATA_DIR --out_dir OUTPUT_DIR
--backbone_annotation BACKBONE_ANNOTATION_FILE
--infinity_marker_annotation
INFINITY_MARKER_ANNOTATION_FILE
[--random_state RANDOM_STATE]
[--use_logicle_scaling USE_LOGICLE_SCALING]
[--normalization_method NORMALIZATION_METHOD]
[--n_events_train N_EVENTS_TRAIN]
[--n_events_validate N_EVENTS_VALIDATE]
[--ratio_for_validation RATIO_FOR_VALIDATION]
[--separate_backbone_reference SEPARATE_BACKBONE_REFERENCE]
[--n_events_combine N_EVENTS_COMBINE]
[--require_n_events_combine REQUIRE_N_EVENTS_COMBINE]
[--n_final N_FINAL] [--add_umap ADD_UMAP]
[--find_clusters FIND_CLUSTERS]
[--find_markers FIND_MARKERS]
[--make_feature_plots MAKE_FEATURE_PLOTS]
[--use_pca USE_PCA] [--n_pc N_PC]
[--n_pc_plot_qc N_PC_PLOT_QC] [--save_h5ad SAVE_H5AD]
[--save_feather SAVE_FEATHER]
[--save_file_handler SAVE_FILE_HANDLER]
[--save_regression_models SAVE_REGRESSION_MODELS]
[--verbosity VERBOSITY] [--n_cores N_CORES]
pyInfinityFlow: error: argument --n_events_combine: invalid int value: 'None'

For now, can you simply remove the argument --n_events_combine
It should default to None type if you don't try to specify any option.
I'll look into how the argument is being parsed currently so that it supports using 'None' as a string argument.

When I remove this comment --n_events_combine
It gives still the following error:
Running InfinityFlow from single directory, with all input .fcs files using the same channels for backbone...
Traceback (most recent call last):
File "/mnt/beegfs/merfish/.conda/envs/pyInfinityFlow/bin/pyInfinityFlow", line 8, in
sys.exit(main())
File "/mnt/beegfs/merfish/.conda/envs/pyInfinityFlow/lib/python3.8/site-packages/pyInfinityFlow/pyInfinityFlow.py", line 204, in main
raise ValueError("separate_backbone_reference and n_events_combine can not be "
ValueError: separate_backbone_reference and n_events_combine can not be used together. One must be None.
With the following arguments:
source activate pyInfinityFlow
pyInfinityFlow --data_dir /mnt/fcs/
--out_dir /output
--backbone_annotation /mnt/backbone_selection_file.csv
--infinity_marker_annotation /mnt/test.csv
--random_state 7
--use_logicle_scaling True
--normalization_method "zscore"
--n_events_train 0
--n_events_validate 0
--ratio_for_validation 0
--separate_backbone_reference /mnt/concat_Fj_Legendscreen_3_1.fcs
--n_final 0
--add_umap True
--find_clusters True
--find_markers True
--make_feature_plots True
--use_pca True
--n_pc 15
--n_pc_plot_qc 50
--save_h5ad True
--save_feather True
--save_file_handler True
--save_regression_models True
--verbosity 1
--n_cores 20

I noticed that the default value for --n_events_combine was adjusted and that caused this issue with the failed separate_backbone_reference.
I fixed this and reuploaded the package.
Can you please try to reinstall the package and see if running the above command works with this fix?

Hello,
I am not sure why but currently it does not want to detect the individual files.
/var/spool/slurmd/job4645016/slurm_script: line 7: activate: No such file or directory
Running InfinityFlow from single directory, with all input .fcs files using the same channels for backbone...
Isotype controls detected. Will attempt to use background correction...
Traceback (most recent call last):
File "/mnt/beegfs/merfish/.conda/envs/pyInfinityFlow/lib/python3.8/site-packages/pyInfinityFlow/InfinityFlow_Utilities.py", line 911, in check_infinity_flow_annotation_dataframes
raise ValueError("\tratio_for_validation must be between "
ValueError: 	ratio_for_validation must be between 0 and 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/beegfs/merfish/.conda/envs/pyInfinityFlow/bin/pyInfinityFlow", line 8, in
sys.exit(main())
File "/mnt/beegfs/merfish/.conda/envs/pyInfinityFlow/lib/python3.8/site-packages/pyInfinityFlow/pyInfinityFlow.py", line 219, in main
file_handler = check_infinity_flow_annotation_dataframes(backbone_annotation=backbone_annotation,
File "/mnt/beegfs/merfish/.conda/envs/pyInfinityFlow/lib/python3.8/site-packages/pyInfinityFlow/InfinityFlow_Utilities.py", line 1000, in check_infinity_flow_annotation_dataframes
raise ValueError("File reading error. Could not set up FileHandler for"
ValueError: File reading error. Could not set up FileHandler for  file A1_Blank.fcs.
The input is still the same as before.

Can you try to remove the --ratio_for_validation option, or set it to a low value > 0.0?
It will default to use 20% (O.2) of the data from each InfinityMarker file to use as validation for the regression model. A value of 0 is throwing the error. I would suggest to try 0.05 or 0.1 if you want to use lower than 0.2.


[BUG] Viewing harvester resources through the k8s container, the page crashes

Describe the bug
In a multicluster, viewing harvester resources via the k8s container page, the page crashes

To Reproduce
Steps to reproduce the behavior:

Creating multiple clusters， Importing harvester clusters
click Virtualization management
go to harvester cluster
Change harvester/harvesterhci.io.dashboard#vm => explorer in the url
click More Resources
click `kubevirt.io

Expected behavior

Harvester CRD in Explorer should display correctly

Support bundle

Environment:

Harvester ISO version:
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630):

Additional context
Add any other context about the problem here.

Verified fixed on Rancher v2.7-head (9bb8d56) with Harvester master-14bbee2c-head.
Close this issue.
Result
We can view the following Harvester resources in Rancher explorer

harvesterhci.io

Test Information

Test Environment: one and two nodes harvester on local kvm machine
Harvester version: master-1be9cb5e-head (1/23)
Rancher version: v2.7-head (9bb8d56)

Verify Steps

Create a ubuntu virtual machine
Set the following user data in cloud config

password: 123456
chpasswd: { expire: False }
ssh_pwauth: True

Open virtual machine in serials console
Login with default user ubuntu
Run command to output large amount of content to console
Scroll down console to the bottom of the window


[SPIKE] Survey porting options

Use Case
A number of python based ROS1 drivers exist for the roboclaw.
One of them needs to be chosen for porting as we're not developing from scratch.
Acceptance Criteria
GIVEN we have ported a roboclaw driver
WHEN we want to know why a particular ROS1 driver was chosen
THEN this ticket will briefly discuss the reviewed options
AND why that one was chosen.
Notes
Official driver https://github.com/sonyccd/roboclaw_ros
4-wheel version https://github.com/sheaffej/roboclaw_driver
SV-ROS version https://github.com/SV-ROS/roboclaw_driver

None of the ROS1 drivers seemed to work out of the box, so creating a ROS2 driver from the native library from basic micro was the best option.


KAFKA-4516: When a CachingStateStore is closed it should clear its associated NamedCache

Clear and remove the NamedCache from the ThreadCache when a CachingKeyValueStore or CachingWindowStore is closed.
Validate that the store is open when doing any queries or writes to Caching State Stores.

@mjsax @enothereska @guozhangwang

LGTM

Merged to trunk, thanks!


Update priorityClassName default values in READMEs

Updates READMEs for the elasticsearch, filebeat and kibana charts to clarify that the default value of priorityClassName is an empty string, re a discussion in #248.
I've made the change in one commit per chart, let me know if you'd rather want these squashed into a single commit.

[x] Chart version not bumped (the versions are all bumped and released at the same time)
[x] README.md updated with any new values or changes
[ ] Updated template tests in ${CHART}/tests/*.py
[ ] Updated integration tests in ${CHART}/examples/*/test/goss.yaml

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

Thank you for so quickly following up...it was almost too fast so I think you might be a robot ;)


set the args in the overloaded SetupEventstore method

the args format $" --config=\"./config.yaml\" {additionalArgs ?? ""}" is not set in the overloaded method SetupEventStore when its called externally.

Obsolete


Dataflow with no dedupe throws 409

With the recent changes to allow 409, collector always returns 201 even with duplicates. The right approach is to returns 409 when dataflow doesn't have duplicate detection configured.

Changes Unknown when pulling ed6fd0eebf997632272a5d8885410fe05f57370c on KRuelY:noDedupe into ** on cloudfoundry-incubator:master**.


Handle normalization of unicode identifiers

Also add a test-case. (Also add a test case for passing keyword
arguments, which already worked, but is good to have as a test
case).
PEP3131 implies that normalization of identifiers should be done before tokenization (i.e. the moment any unicode character is found the surrounding area of code should be normalized then checked to see if it's an identifier). Practically, I don't think this is what Python does. Instead it finds a bit of string that might be an identifier and validates that it is (https://github.com/python/cpython/blob/69f37bcb28d7cd78255828029f895958b5baf6ff/Parser/tokenizer.c#L1062). Normalization is done when the AST is generated (https://github.com/python/cpython/blob/51aac15f6d525595e200e3580409c4b8656e8a96/Python/ast.c#L610).
I've followed (what I believe is) Python's approach. This might make a different if some unicode characters that aren't valid start or continuation characters can get normalized to something that is. I don't think this happens though. And this patch should certainly take care of the majority of the cases.

Thanks! Could you please also add the example from the question here?
https://stackoverflow.com/questions/16467479/normalizing-unicode
Maybe there are also some combining characters that do not go together with a? (Because that's what you used to determine the continuation characters in the generator script.)
One cool feature in the test runner is that it allows writing tests in .py (not .pyx) files that are then also executed in normal Python for comparison. If they require a specific minimum Python version (such as 3.0 in this case), you can add a tag like # tag: pure3.0 at the top to disable them in pure Python 2.x. The compiled module will then still be tested in all Python versions. Thus, you can actually move the Cython specific parts of your tests into a separate .pyx test module and rename the main one to .py, so that we can get the confirmation that Cython behaves like Python for all those nice test cases.
That said, maybe an error case would also be helpful? Look in tests/errors/ (with # mode: error), those tests can check compile errors, (hopefully) including invalid identifier characters.

 Testing multiple cases tends to require a separate test module for each of them, rather than just separate functions or even code lines.

I've added some more testing for normalized symbols including the a-acute example. It's in a slightly odd srctree structure because constructing examples from characters that look identical seemed better to do with escape sequences and string formatting. I've also added a few tests of error cases, although I don't think they're hugely interesting.
I don't think there's an issue with combining characters that don't go with "a":

If you look at the CPython source (https://github.com/python/cpython/blob/2a16eea71f56c2d8f38c295c8ce71a9a9a140aff/Objects/unicodeobject.c#L12200) it goes through character by character so the choice of first character doesn't affect whether the second is valid.
In PEP 3131 https://www.python.org/dev/peps/pep-3131/#id16 XID_start and XID_continue are described as properties of the character. They consider cases where 1 character normalizes to multiple characters and exclude if the multiple characters are invalid. When 2 characters normalizes to 1 doesn't affect the XIDs.

With that said the tests do now include combining characters that don't normalize to anything shorter (the normalization only standardizes the order of the combining characters)

I've added one further test. This required a bit of tweaking of the test runner to be able to match the output. Therefore it might be reasonable to decide that this test isn't worth the amount of change it needed...

I'd rather keep the complexity low for this. Let's assume that the errors can be decoded with the declared source code encoding. In almost all cases, they will be ASCII-only.

OK. I've done it the way you suggest - decoding in ErrorWriter is needed.


Implement database update function

Implement a fuction update(database: str, source_files: str or source_dir: str) which will copy the source files to the target database and simultaneously update its index.
Not as urgent as #6. In the beginning, reindexing of the database could be enforced on every update.

Resolved in the commit above. Updating existing files is not handled and raises an exception.


enhanced testing thanks to the OpenBLAS-for-Windows

The appveyor service is a little slow, we have to wait for tests. Currently, I have no Windows machine, so the only way is to check things through appveyor-ci.

Closing, we need to have proper OpenBLAS library on MS Windows detection.


[WIP] ROSA Preflight checks DAG

Add options for running rosa installation on all rosa channels (stable, candidate & nightly)
Add option for selecting fixed OCP version on ROSA or latest image available
New DAG for ROSA with fixed version to 4.10.12, executed each 12 hours (for preflight validation)
Removed schedule for ROSA CCS DAGS

Codecov Report

Merging #192 (f9f8db5) into master (2d033e2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #192   +/-   ##
=======================================
  Coverage   93.29%   93.29%
=======================================
  Files          24       24
  Lines         999      999
=======================================
  Hits          932      932
  Misses         67       67

Flag
Coverage Δ

gha
93.29% <ø> (ø)

python-3.9
93.29% <ø> (ø)

unit
93.29% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d033e2...f9f8db5. Read the comment docs.

Installation example: http://morenod-managed-services-preflight-airflow.apps.sailplane.perf.lab.eng.rdu2.redhat.com/log?dag_id=4.10-rosa-sts-ovn-preflight&task_id=install&execution_date=2022-05-23T09%3A00%3A55.204697%2B00%3A00

Yay, so this logic worked out! 💯

@dry923 can you please review last changes? I have removed the DAG for preflights as it makes no sense because Im executing them on my airflow instance.
So now this PR is about allowing selecting the ROSA chanel group and the OCP version to be installed


[WIP] Add synthetic word generator

This PR is the opening for a synthetic Word Image generator as a supplement to the CharacterGenerator
If we want to add some more stuff like text rotation or distorsion while generation let me know !
suggestions are very welcome :hugs:
@fg-mindee
Can we provide some basic fonts in a resources folder ?
Todo:

[ ] UnicodeEncodeError: 'latin-1' codec can't encode character .... (catch ftm)
[ ] needs some more love :heart: and suggestions @fg-mindee
[ ] improve tests

Issues:
#262

Hi @felixdittrich92 :wave:
Thanks for the PR! Actually this is a long-term PR so it's usually better to discuss this a bit before opening a PR (otherwise we might end up going for a completely different implementation, which would have been a waste of your time :sweat_smile: )
cf. #262

@fg-mindee
ok i will close this ftm :)
and switch to the issue for discussions


update github repos to synerbi

Updates the URL pointing to CCPPETMR to SynerBI.

Together with https://github.com/SynerBI/CCPPETMR_VM/pull/151 I'm testing that it all works.

All fine.
Last Travis job "docker +SERVICE +GPU" fails with unrelated problem:
https://travis-ci.org/github/CCPPETMR/SIRF-SuperBuild/jobs/671456370#L4374
user_sirf-ubuntu.sh: line 13: cmake: command not found

@casperdcl a quick look?

https://travis-ci.org/github/SynerBI/SIRF-SuperBuild/builds/672550235 passed but https://travis-ci.org/github/SynerBI/SIRF-SuperBuild/builds/672660623 failed :/

Just a quick explicit confirmation... @KrisThielemans is it SynerBI or SyneRBI? It looks like the gh org was created with the former...

sigh. SyneRBI. Not sure what a github rename will do, as https://github.com/SyneRBI/ works as well. Just rename again?

Sure, I think renaming again should be fine. URLs are case-insensitive. The main thing that would change is the email notification subject lines.

I've renamed the org. So now we need to correct all @paskino 's renames (e.g. in the headers). @paskino, could you take care of this?

Sure, my bad, I'll do that.

I've updated the Superbuild headers, however:

can we make without the author? It'd make things way easier to maintain. Some headers don't have any header, let alone author. I suggest to keep an authors.md file listing all contributors.
I'll try to update all the references to SynerBI to SyneRBI, such the project name in the CMake files...
there are references to the www.synerbi.ac.uk here and there but it doesn't yet exists and the reference to the ccppetmr  website has been removed.

can we make without the author?

It's indeed a pain. However, I've noticed that some contributors really like to have their name listed in the file that is changed. On the other hand, others can't be bothered to update the header.
In any case, the author isn't the only a problem, more important is the copyright owner. That is often forgotten as well. However, I can't see how we can dispense with that. This is legal information, that otherwise would only be visible via git (if we assume that we never commit on behalf of someone else), and it wouldn't be visible on conda or so.
As we need to maintain the copyright message, we can just as well add the author as well...

Re the website, either you undo those changes, or we wait a bit for merging. I think I'd choose the latter...

Let's wait ;)

@paskino I already made those changes 9h ago :) did you mean to merge, btw?

@casperdcl I just added a few changes to comments, nothing functional, as per https://github.com/SyneRBI/SIRF-SuperBuild/pull/372#pullrequestreview-399727265
After this merging was the plan, afaik.


chore(deps): update eclipse-temurin docker tag to v17.0.8_7-jdk-jammy

This PR contains the following updates:

Package
Type
Update
Change

eclipse-temurin
final
patch
17.0.7_7-jdk-jammy -> 17.0.8_7-jdk-jammy

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

Thanks for the PR!
Any successful deployments (not always required) will be available below.
Backend
Frontend
Oracle-API
Once merged, code will be promoted and handed off to following workflow run.
Main Merge Workflow

Merging without Code Scanning API, due to be timing out.


servicebusserviceclient timeout does not work due to typo in option selection

Commit 46da8fabd799e0840bab9036b566a1010e4fc0b4 added support for the timeout query param to the servicebusserviceclient, but it incorrectly checks for options.timeoutIntervalInMs when deciding if a timeout was requested.  The correct option name is options.timeoutIntervalInS as seen on the next line of code.

Well spotted @malyn ... I'll get this fixed ASAP.


disable partial auth feature for relay

disable partial auth feature for relay

Closing this issue as the pr is merged


Fix several clippy::pedantic lints

I ran cargo clippy --fix -- -W clippy::pedantic and manually inspected the changes. These are the fixes.
There are a lot of fixes left, but this is the first batch. Also, while reading the code I found some bug fixes for future PR. (and performance improvements too). Sometimes, each programmer has their own code style so feel free to revert any changes this PR made.
This is a summary of the changes:

Use backticks around rust_xlsxwriter and other crate elements
Use underscores as separators in numbers and hex colors
Avoid star imports
Remove redundant () after names of functions and methods
Change "".to_string() and String::from("") to String::new()
Use From::from instead of as conversions whenever possible
Add #[must_use] to relevant functions
Change Iterator::cloned to Iterator::copied whenever possible
Use sort_unstable instead of sort when working with primitives
Change for _ in _.iter() to for _ in &_ (same with iter_mut)

As said in the related issue feel free to ask to revert any of the changes suggested. And this was just the first step to get familiarized with the code, I plan to submit less trivial PRs in the near future.

My bad, I didn't re run the tests after rebasing...

I added the missing &mut  and removed some backticks. It now passes cargo doc and cargo test.

It will pass everything now. I think.


Question about a new type of audio encoding.

Hi Martin, I recently noticed that you've added some support for audio encoding programs to the VVCEASY project on Github. I have a couple of questions to say:

can these new audio encoding programs surpass OPUS encoding in terms of compression efficiency?

is it possible to add Facebook's "EncodeC" encoding to FFMPEG?

https://github.com/facebookresearch/encodec

Do you plan to support these new encoding formats in your modified MPV player?

Hi, @jiangzemin-xudamin!
Thank you for asking three questions to me. I will answer your questions:

Well it may can surpass OPUS encoding by new audio encoding programs like xHE-AAC or other improved encoder programs.
It requires implementation to FFmpeg to write encodec decoder using C programming language, so that's hard to write on it for me, but I'm not good using C advanced programming skills and have to learn it by myself. You may need to create issue to encodec repository, if someone can make encodec implementation to FFmpeg for your request.
I'm not sure if I can plan new encoding formats to my modified MPV player, but I might find out later if I can still add for it.

Thank you! Feel free to reply me if you like! :)
Regards

Martin Eesmaa

Thanks for your answer, Martin
This is an audio test about MP3, OPUS, Lyra-v2, EnCodec, EVS, SoundStream:
https://ai.honu.io/papers/encodec/samples.html
In terms of sound quality at the same bitrate, my personal ranking is: EnCodec>SoundStream>EVS>Lyra-v2>OPUS>MP3.


Fix relation between AbstractSpaces and AbstractThemanticSurfaces

According to the Core module, AbstractSpace has a general zero-to-many relationship to AbstractThematicSurface that is navigable in both directions. In many but not all extension modules this relationship is restricted. For instance, an AbstractConstruction can only be bounded by AbstractConstructionSurface and the association is only navigable in this direction.
However, these restrictions are not very consistent. For example, BuildingRoom is directly derived from AbstractUnoccupiedSpace. Does this mean, that the boundary relationship is not restricted to AbstractConstructionSurface but that it can be bounded by every subclass of AbstractThematicSurface (note that this includes, for instance, LandUse and TINRelief)? This wouldn't really make sense, would it?
The relationsship between a space and its allowed thematic boundary surfaces should be much clearer and explicit. Maybe the general relationship in the Core module should be dropped and only be defined in the extension modules.
I am also not really convinced about the way ClosureSurface is used. It is defined in the Core module as subclass of AbstractThematicSurface. But since the boundary relation for AbstractConstruction is restricted to AbstractConstructionSurface it cannot be used directly. Therefore, a "has a" relation from AbstractConstructionSurface to ClosureSurface has been introduced. But this means, you cannot model a ClosureSurface on its own for construction spaces, but it always requires an embracing instance of a construction surface like roof or wall surface. Again, imho, this does not really make sense and is very different to CityGML 2.0

Just to have more examples, here is a non-exhaustive list of spaces whose boundary relationship is not restricted and which consequently could be bounded by any subclass of AbstractThematicSurface such as LandUse or TINRelief:

AbstractConstructiveElement
AbstractFillingElement
AbstractFurniture
AbstractInstallation

This issue was discussed at the SWG meeting in Banff, September 2019. Two options were presented to resolve the problem:

By providing OCL constraints in the UML model, the thematic surfaces that are allowed for a specific space (e.g. BuildingRoom) can be specified explicitly. The disadvantage with this solution is, however, that the space cannot be bounded by thematic surfaces that are defined as part of an ADE. The space can only be bounded by those surfaces that are now defined in the OCL constraint, it is not possible to include future thematic surfaces defined by an ADE in the OCL constraint now.
Alternatively, the allowed surfaces can be specified as normative requirement in the Conceptual Model specification in textual form. Here it can also be defined that the normative requirement is valid for surfaces defined within ADEs as well.

Due to the disadvantage of option 1 we discussed at the modelling subgroup meeting on 18 October 2019 that we would like to proceed with option 2.


StackLlaMa 2 dpo train with deepspeed oom

I can train stack llama2 with 8gpus with ddp. Which I have to use {"device_map": {"": Accelerator().local_process_index}} , detail info can be found here.
I want to use deepspeed stage 3 to train it because I will train 70b later. For large model, I can't train with ddp.
So I ran it with the deepspeed_zero3.yaml:
accelerate launch --config_file examples/accelerate_configs/deepspeed_zero3.yaml examples/research_projects/stack_llama_2/scripts/dpo_llama2.py \
    --model_name_or_path="sft/final_checkpoint" \
    --output_dir="dpo" \
    --report_to="tensorboard"

it failed with:

Traceback (most recent call last):
  File "/nas/lili/codes/pt/ft/trl/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py", line 140, in <module>
    model = AutoModelForCausalLM.from_pretrained(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
    return model_class.from_pretrained(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2992, in from_pretrained
    raise ValueError(
ValueError: DeepSpeed Zero-3 is not compatible with `low_cpu_mem_usage=True` or with passing a `device_map`.

As in [this issue]((https://github.com/huggingface/trl/issues/1348), I passed device_map to AutoModelForCausalLM.from_pretrained. But it seems DeepSpeed Zero-3 is not compatible with passing device_map.
So I remove this parameter. It ran oom this time:
Traceback (most recent call last):
  File "/nas/lili/codes/pt/ft/trl/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py", line 214, in <module>
    dpo_trainer = DPOTrainer(
  File "/nas/lili/codes/pt/ft/trl/trl/trainer/dpo_trainer.py", line 234, in __init__
    model = prepare_model_for_kbit_training(model, **prepare_model_kwargs)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/peft/utils/other.py", line 105, in prepare_model_for_kbit_training
    param.data = param.data.to(torch.float32)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB. GPU 0 has a total capacity of 39.39 GiB of which 469.00 MiB is free. Process 70022 has 4.62 GiB memory in use. Including non-PyTorch memory, this process has 4.61 GiB memory in use. Process 70021 has 4.62 GiB memory in use. Process 70018 has 4.62 GiB memory in use. Process 70019 has 4.62 GiB memory in use. Process 70015 has 5.59 GiB memory in use. Process 70017 has 4.62 GiB memory in use. Process 70016 has 5.59 GiB memory in use. Of the allocated memory 3.99 GiB is allocated by PyTorch, and 145.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Traceback (most recent call last):
  File "/nas/lili/codes/pt/ft/trl/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py", line 214, in <module>
    dpo_trainer = DPOTrainer(
  File "/nas/lili/codes/pt/ft/trl/trl/trainer/dpo_trainer.py", line 234, in __init__
    model = prepare_model_for_kbit_training(model, **prepare_model_kwargs)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/peft/utils/other.py", line 105, in prepare_model_for_kbit_training
    param.data = param.data.to(torch.float32)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB. GPU 0 has a total capacity of 39.39 GiB of which 477.00 MiB is free. Process 70022 has 4.62 GiB memory in use. Process 70020 has 4.61 GiB memory in use. Process 70021 has 4.62 GiB memory in use. Including non-PyTorch memory, this process has 4.61 GiB memory in use. Process 70019 has 4.62 GiB memory in use. Process 70015 has 5.59 GiB memory in use. Process 70017 has 4.62 GiB memory in use. Process 70016 has 5.59 GiB memory in use. Of the allocated memory 3.99 GiB is allocated by PyTorch, and 145.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Traceback (most recent call last):

I have 8 a100 40GB gpus. I think for llama2-7b, it's enough. And I checked the yaml config, it has "zero3_init_flag: true". So I think it will not load the whole model to a single gpu/device but load each shard its own parameters.
But in  peft/utils/other.py
    if not is_gptq_quantized:
        # cast all non INT8 parameters to fp32
        for param in model.parameters():
            if (param.dtype == torch.float16) or (param.dtype == torch.bfloat16):
                param.data = param.data.to(torch.float32)

it seems peft need cast bf16 parameter to fp32. When it ran, I saw there are 8 processes ran in gpu0 and the gpu0 memory is used up and it failed.
So I guess the peft don't support shard parameters to 8 gpus but load all to a single gpu.

see https://github.com/huggingface/peft/issues/1507


Unique names for CoreLibNative

CoreRT side of https://github.com/dotnet/corefx/issues/4818. Mirrors style and naming conventions of the corefx changes.

LGTM