Async chart not rendering

Hello, I have tried to turn on GAQ on the latest helm/superset release (0.6.1, 1.5.0, respectively). When I try to run a query from a chart, the chart spins until it times out. However, I can see the query execute on the data warehouse and the results are returned to redis. In fact, if I refresh the chart page after seeing the load_chart_data_into_cache task complete successfully on the worker, then the chart pulls from redis and loads as expected. I believe that I've configured superset as outlined here.
Async queries are running as expected in sql labs.
How to reproduce the bug
Deploy superset to k8s using helm with the following config overrides:
configOverrides:
  enable_gaq: |
    GLOBAL_ASYNC_QUERIES_JWT_SECRET = env('GLOBAL_ASYNC_QUERIES_JWT_SECRET')
    GLOBAL_ASYNC_QUERIES_REDIS_CONFIG = {
      "port": env('REDIS_PORT'),
      "host": env('REDIS_HOST'),
      "password": env('REDIS_PASSWORD'),
      "db": 0,
      "ssl": False
     }

    FEATURE_FLAGS = { 'GLOBAL_ASYNC_QUERIES': True }

Expected results
I expect the chart to render results after the query completes.
Actual results
The chart never renders.
Screenshots

Environment
(please complete the following information):

browser type and version: Firefox 99.0.1 (64-bit)
superset version: 1.5.0
python version: 3.8.12
node.js version: node -v
any feature flags active: Global async queries

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
Add any other context about the problem here.

+1, the async event in polling api is always empty([]). The charts never renders.

I'm seeing this same behavior as well, the celery worker seems to load the query fine
2022-07-27 16:15:45,322: INFO/MainProcess] Task load_chart_data_into_cache[4470045e-893b-4591-b9f0-e137680061f1] received
[2022-07-27 16:15:45,323: DEBUG/MainProcess] TaskPool: Apply <function fast_trace_task at 0x7fd54dd5b790> (args:('load_chart_data_into_cache', '4470045e-893b-4591-b9f0-e137680061f1', {'lang': 'py', 'task': 'load_chart_data_into_cache', 'id': '4470045e-893b-4591-b9f0-e137680061f1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, 21600], 'root_id': '4470045e-893b-4591-b9f0-e137680061f1', 'parent_id': None, 'argsrepr': "({'channel_id': '672a8009-1db5-45ca-b2eb-6cd2d8f2713b', 'job_id': 'eb001c6c-d2e3-43b4-9b43-af5a84f6fa26', 'user_id': 1, 'status': 'pending', 'errors': [], 'result_url': None}, {'datasource': {'id': 182, 'type': 'table'}, 'force': False, 'queries': [{...}], 'form_data': {'datasource': '182__table', 'viz_type': 'big_number_total', 'slice_id': 305, 'url_params': {...}, 'granularity_sqla': 'data_dt', 'time_grain_sqla': 'P1D', 'time_range': 'No filter', 'metric': {...}, 'adhoc_filters': [...], 'header_font_size': 0.4, 'subheader_font_size': 0.15, 'y_axis_format': 'SMART_NUMBER', 'time_format': 'smart_date', 'extra_form_data': {...}, 'force':... kwargs:{})
The field `timeseries_limit` is deprecated, please use `series_limit` instead.
[2022-07-27 16:15:45,341: WARNING/MainProcess] The field `timeseries_limit` is deprecated, please use `series_limit` instead.
[2022-07-27 16:15:45,645: DEBUG/MainProcess] Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
[2022-07-27 16:15:45,648: DEBUG/MainProcess] Making request: POST https://oauth2.googleapis.com/token
[2022-07-27 16:15:45,652: DEBUG/MainProcess] Starting new HTTPS connection (1): oauth2.googleapis.com:443
[2022-07-27 16:15:45,702: DEBUG/MainProcess] https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
[2022-07-27 16:15:45,706: DEBUG/MainProcess] Starting new HTTPS connection (1): bigquery.googleapis.com:443
[2022-07-27 16:15:46,143: DEBUG/MainProcess] https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects//jobs?prettyPrint=false HTTP/1.1" 200 None
[2022-07-27 16:15:46,338: DEBUG/MainProcess] https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects//queries/c24b2e77-f8e9-441e-9008-366628029d54?maxResults=0&location=US&prettyPrint=false HTTP/1.1" 200 None
[2022-07-27 16:15:48,358: INFO/MainProcess] Task load_chart_data_into_cache[4470045e-893b-4591-b9f0-e137680061f1] succeeded in 3.0331106479279697s:

But the web server keeps repeatedly hitting the async endpoint, and not getting any results.
Configs are
class CeleryConfig(object):
    broker_url = 'redis://{0}:{1}/0'.format(REDIS_HOST, REDIS_PORT)
    imports = (
        'superset.sql_lab',
        'superset.tasks',
    )
    result_backend = 'redis://{0}:{1}/0'.format(REDIS_HOST, REDIS_PORT)
    worker_log_level = 'DEBUG'
    worker_prefetch_multiplier = 1
    task_acks_late = False
    task_annotations = {
        'sql_lab.get_sql_results': {
            'rate_limit': '10000/s',
        },
        'email_reports.send': {
            'rate_limit': '1/s',
            'time_limit': 120,
            'soft_time_limit': 150,
            'ignore_result': True,
        },
    }

CELERY_CONFIG = CeleryConfig

# On Redis
from cachelib.redis import RedisCache
RESULTS_BACKEND = RedisCache(host=REDIS_HOST, port=REDIS_PORT, key_prefix='superset_results')

# import redis
CACHE_DEFAULT_TIMEOUT = 21600 # Default timeout

GLOBAL_ASYNC_QUERIES_JWT_SECRET = ''
GLOBAL_ASYNC_QUERIES_POLLING_DELAY = 4000

GLOBAL_ASYNC_QUERIES_REDIS_CONFIG = {
    "port": REDIS_PORT,
    "host": REDIS_HOST,
    "password": "",
    "db": 0,
    "ssl": False,
}
GLOBAL_ASYNC_QUERIES_TRANSPORT = "polling"

CACHE_CONFIG = {
    'CACHE_TYPE':'redis', # Use Redis
    'CACHE_REDIS_PORT': REDIS_PORT, # Configure port number
    'CACHE_REDIS_URL':'redis://{0}:{1}'.format(REDIS_HOST, REDIS_PORT)
}

DATA_CACHE_CONFIG = {
    'CACHE_TYPE':'redis', # Use Redis
    'CACHE_REDIS_PORT': REDIS_PORT, # Configure port number
    'CACHE_REDIS_URL':'redis://{0}:{1}'.format(REDIS_HOST, REDIS_PORT)
}

EXPLORE_FORM_DATA_CACHE_CONFIG = {
    'CACHE_TYPE':'redis', # Use Redis
    'CACHE_REDIS_PORT': REDIS_PORT, # Configure port number
    'CACHE_REDIS_URL':'redis://{0}:{1}'.format(REDIS_HOST, REDIS_PORT)
}

FILTER_STATE_CACHE_CONFIG = {
    'CACHE_TYPE':'redis', # Use Redis
    'CACHE_REDIS_PORT': REDIS_PORT, # Configure port number
    'CACHE_REDIS_URL':'redis://{0}:{1}'.format(REDIS_HOST, REDIS_PORT)
}

FEATURE_FLAGS = {
"ENABLE_TEMPLATE_PROCESSING": True,
"DASHBOARD_NATIVE_FILTERS": True,
"DASHBOARD_CROSS_FILTERS": True,
"VERSIONED_EXPORT": True,
"GLOBAL_ASYNC_QUERIES": True,
}

Celery worker running with "celery --app=superset.tasks.celery_app:app worker --pool=threads -c 20 --loglevel=DEBUG",
This is running superset 2.0.0, we may look into using websockets but at this point hard to invest the time if global async queries might not work.

Did anyone investigate this further? Having the same problem with 2.0.0.

Having the same issue, tried different versions, no luck

I'm seeing similar behavior, particularly in embeds, it seems to be that there is auth / permission issues with the POST to the chart endpoint to fetch the polling data. Captured here - https://github.com/apache/superset/discussions/18814

I have the same issue. I tried version 1.3.2, 1.5.2 and 2.0.0 but no luck

+1, any update?

+1

+1

I am getting results successfully no problem

I got same problem.. is there an update related this issue..?

I'm not sure if this is still an issue in current versions of Superset (3.x). If it is, we can re-open this, or feel free to open a new issue with updated context and a reproducible case. We're no longer supporting Superset 2.x or prior, and it's been a while since this thread saw any activity, so I'm closing this as stale.

@smisch would you be willing to open a docs PR? There's an "edit this page" button in the bottom right of the docs site now which I hope makes it easier. Or if anyone wants to spell out what the instructions should be here on this thread, I can open the PR.

@rusackas I'll check it out. Haven't been contributing on Github yet, but looks easy enough. Fork, clone, change, commit, push, propose as PR to you.

Yep! And I think the Github website will do like half of that pretty seamlessly/smoothly if you do it from the website. Of course, forking/cloning and making edits locally is fantastic if you want to contribute to additional pages or areas of the codebase!


Fix ctrl+c handling on windows

In windows, it is necessary for the console mode to include the
ENABLE_PROCESS_INPUT flag in order for ctrl+c to be treated as a signal
rather than a character. In jline 2, the ENABLE_PROCESS_INPUT flag was
preserved whenever the console mode was changed but in jline 3, it was
not. It is easy enough to manually set the flag after entering and
exiting raw mode and setting attributes (which are the apis that change
the console mode on windows).

Fixes #5940


ScrubLinesWithReplace does not inject the whole line into the delegate, just the property value

Preamble
Development Stack

Rider 2022.1.2
.net 6.0
Verfify.XUnit 17.10.2

Describe the bug
When calling ScrubLinesWithReplace(Func<string, string?> replaceLine) on VerifySettings and passing that into the Verify() method along with an object. When that object is serialized as JSON, only the property value is injected into the delegate/func for each line.
Example .recieved.txt
{
  Id: kxdi7ZanHj,
  ForeignObjectId: TaqKSdYXCs,
  Name: Name Of Person
}

Will call ScrubLinesWithReplace with the values kxdi7ZanHj, TaqKSdYXCs and Name Of Person. It does not include the property name.
This prevents me from Scrubbing that line, or replacing its value where I need to know the property name to correctly scrub the value.
What does work?
If you serialize the object to a string first, then this works as expected.
Minimal Repro
VerifyTests.JsonPropertyNames
Submit a PR that fixes the bug
If this is confirmed as a bug, I will consider a PR.

Hey, this is affecting us too.  On v16.9.0 the scrubber received the whole line, which made it easy to write a regex to scrub the value based on the property name.
On v17.10.2 (and possibly other v17 releases) the scrubber just receives the value, which as @kaddyuk mentions, makes it impossible to write a regex.
Would it be possible to consider a global flag to opt in or out of this behaviour please?

@paulsuart @dracan happy to accept a PR for that

@SimonCropp thanks for getting back.
What strikes me as odd is that this is a breaking change (for us, and the OP), but I wouldn't expect a breaking change between such minor releases.
This makes me think that there must be a different/better way of scrubbing property values, leaving the property names intact.
Do you have any advice?

@paulsuart isnt https://github.com/VerifyTests/Verify/blob/main/docs/serializer-settings.md#scrub-member-by-expressions sufficient ?

Thanks, I'll give this a go and get back to you.

@SimonCropp - I had to install a pre-release package to get this ScrubMember method, is that right?
I tried it out, and it doesn't seem to do anything at all.  Applying the setting globally, via settings or fluently doesn't seem to make any difference.

@paulsuart can u submit a test that fails?

Our tests use VerifyJson.  This method doesn't seem "compatible" with this method - even ScrubMember(memberName) doesn't seem to do anything for nested JSON prroperties.
I'm trying to work out how to use ScrubMember with VerifyJson, but again, if you have any advice...

Here's an example of what one of our tests looks like:
record MyType
{
    public string PropA { get; set; }
}

[Fact]
public Task WorkedOnVersion17Point5()
{
    var received = new MyType
    {
        PropA = "PropA"
    };
    var settings = new VerifySettings();

    VerifierSettings.ScrubLinesWithReplace(x =>
        Regex.Replace(x, @"PropA: ([^,]*)", "PropA: [SCRUBBED]"));

    // Imagine this Json comes from an HTTP endpoint
    var rawJson = JsonSerializer.Serialize(received);

    // {
    //     PropA: [SCRUBBED]
    // }
    return Verifier.VerifyJson(rawJson, settings);
}

try this
    [Fact]
    public Task WorkedOnVersion17Point5()
    {
        var received = new MyType
        {
            PropA = "PropA"
        };
        VerifierSettings.ScrubMember("PropA");

        // Imagine this Json comes from an HTTP endpoint
        var rawJson = JsonSerializer.Serialize(received);

        return Verifier.VerifyJson(rawJson);
    }

or with it scoped
    [Fact]
    public Task WorkedOnVersion17Point5()
    {
        var received = new MyType
        {
            PropA = "PropA"
        };

        var rawJson = JsonSerializer.Serialize(received);

        return Verifier.VerifyJson(rawJson)
            .ScrubMember("PropA");
    }

Going back to the ScrubLinesWithReplace change - it does feel that this makes that method a lot less useful. Scrubbing the value without the context of the property name is a bit too all-encompassing. It seemed way more useful as it was before the change.

or with it scoped
    [Fact]
    public Task WorkedOnVersion17Point5()
    {
        var received = new MyType
        {
            PropA = "PropA"
        };

        var rawJson = JsonSerializer.Serialize(received);

        return Verifier.VerifyJson(rawJson)
            .ScrubMember("PropA");
    }

@SimonCropp - thanks, this worked.  I could have sworn I tried this a couple of days ago and it didn't work - hence my earlier comment about it not seeming to do anything at all.  I think I can refactor our tests to use this approach.
I also echo @dracan's comment about the method feeling less useful.  Furthermore, it feels like the method is now mis-named as it doesn't necessarily scrub (whole) lines!
Anyway, thanks for all your effort on this project.  It really is appreciated 👍🏽

Furthermore, it feels like the method is now mis-named as it doesn't necessarily scrub (whole) lines!

it scrubs lines of text content. in the context of a string (using Verifier.Verify(rawJson)) it will work as you expected. In the context of a serialized object (which is what VerifyJson does since it creates a node model under the covers) it will process string members.
As i said, if u want an option to opt into have ScrubLinesWithReplace (scrubbers in general) operate on the full resulting string, I am happy to consider a PR

@paulsuart i would really like a failing test in a PR that demonstrates a scenario that you can only solve with the previous behaviour

Apologies for the delay - I only just saw this.
I had to remind myself of what we were doing when this thread was created.  I tried writing some tests, and the only scenario I can think of that can only be solved with the previous behaviour is probably too contrived to be worth the attention.
I am happy with the current behaviour.


Add --path flag

Instead of piping csv data into the binary, accept a --path argument and read the data from a file. This is possible now because we aren't insisting on being dependency-free.

I don't think this is necessary and changes the API & implementation significantly. Reading csv data from stdin should be suitable for now. Can re-open in the future if needed.


Tidy up PeerSet:

Move PeerSet to overlay/
Remove unused functions
Make some public members private
Rename some functions
Add comments

LGTM :+1:

Tiny comments about comments.
Otherwise, LGTM!

Just a minor point: while you're tidying this up, note that mAggressive is only used in InboundLedger and could be moved to that class instead. Not a dealbreaker for me.

I'm adding "Passed" since this technically passed review, modulo comments about comments.


Remove the base64 encoded scripts in the workflow files

To run the Python scripts within the reusable workflows we checkout the main branch of this repository. However, this means that if a PR updates a Python script the script that the PR CI tests is the version in the main branch and no the PR branch. Need to change the following in each workflow:
- uses: actions/checkout@v2
  with:
    repository: 'OpenAstronomy/github-actions-workflows'

Actually, the PR will test the Python script from the PR branch because 'OpenAstronomy/github-actions-workflows' is the default value and in that case the current ref is used.
However, if the reusable workflow is called by a different repository it will checkout the default branch of OpenAstronomy/github-actions-workflows and not e.g. refs/tags/v1 if the workflow was referenced as ...tox.yml@v1. See https://github.community/t/reusable-workflows-get-the-ref-inside-the-called-workflow/224109
We could ask the user to pass the ref they used as an input to the reusable workflows although that is not ideal. Hopefully there will be a better solution soon.

An ugly solution would be that when we are ready to tag, we add a commit that hard-codes the ref of the repo to check out, e.g. v1.0.2, and then tag that commit v1.0.2, then add a commit that removes the ref again. I think it would work, although is pretty ugly.

Another solution, not very elegant but which would at least work would be to embed the Python scripts inside the workflow YAML, for instance:
<contents of tox_matrix.py>
" > tox_matrix.py
python tox_matrix.py ...

Yes this is ugly but it might be that the Python script doesn't need to be updated too often so it might not be that bad.

See https://github.com/OpenAstronomy/github-actions-workflows/pull/20 for a proof of concept

@ConorMacBride should we close this?

This is no longer an issue so maybe just close for less noise. But we should switch to a solution that doesn't involve duplication if/when GitHub adds such a feature.

https://github.com/actions/toolkit/issues/1264 seems to suggest it's still not possible via the github context. I'm guessing the github.workflow_ref might refer to the workflow that calls the reusable workflow. Looking through some of the workarounds mentioned in the issue we should be able to use the GitHub API.
When I run the following in the GitHub CLI, I get d68193b68216da64eafaa618f53c59f5d271c56e, which is correct:
gh api repos/astropy/astropy/actions/runs/12186371008 \
    -q '.referenced_workflows |
        map(
            select(
                .path |
                startswith("OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@")
            )
        ) |
        .[0].sha'

In a GitHub Action we should be able to do: repos/${{ github.repository }}/actions/runs/${{ github.run_id }}.
However, if someone is calling the same reusable workflow twice in the same workflow file, we'll get multiple results and I don't think we can distinguish which is which. That would only be an issue if they are using multiple versions. The code above just takes the sha from the first match. We could detect multiple versions and just error out saying it's unsupported.

That's annoying. I quite often change one version (or repo) in a workflow while I am testing stuff but I guess I am not a usual usecase in doing that.
An alternative thought I had is if it would be possible to inline the script in plain text?


Input change via webinterface

Describe Your Problem:
Is there a possibility to use the webinterface to change a input?
I use the Yamaha Plugin that provides me a input list on the iPhone home app. But on the webinterface I see only a turn on/off switch.

Environment:

Node.js Version: v16.14.2
NPM Version: v8.5.0
Homebridge Version: v1.4.0
Homebridge Config UI X Version: v4.42.0
Operating System: Raspbian
Process Supervisor:  hb-service

any idea? Is it a problem of the plugin, or from the Config UI X?


[Typescript][Feature] Automatically add inferred types to code

From @Alber70g on February 21, 2018 11:14
I'm not sure if I should place this here, but since I cannot find the specific repo for Typescript in VSCode, it seems to be the correct place
When working with Typescript it's very nice that the compiler infers a lot of types for me. However, I usually add the return type to functions in order to preserve the signature when refactoring functions. This way I don't end up having a problem when I accidentally change the return type because I return something different.
Would it be possible to add the return type automatically, since the AST knows what type I'm returning.
This could be an option to do when doing format code and optional for const's let's functions and others that infer types.
Copied from original issue: Microsoft/vscode#44111

Moving upstream for more discussion.
This is somewhat similar to the existing infer from usage quick fix but that fix only applies to any. The request here is add explicit type annotations to places where we already correctly infer the type

Looks like a possible quick fix for tslint rule like https://palantir.github.io/tslint/rules/typedef/

This would indeed be useful occasionally. I think it should be a code action provided by TypeScript, not by a specific linter.

I think this would be highly useful and would probably use it several times per day if it existed. I am not always sure what type of value a piece of code will produce, and if I can have typescript generate them and insert them into the code it would be a huge help.
If I am not happy with the generated type, I can just modify it, but it's great to have an autogenerated starting point!

Here is a workaround in the meanwhile: https://marketplace.visualstudio.com/items?itemName=ebrithil30.vscode-ts-auto-return-type

This should also help to alleviate the problem of seeing what type a piece of code will produce. This VS Code extension inline renders the inferred types, so you can see them clearly inside the code when you want:
https://marketplace.visualstudio.com/items?itemName=wuwenzhou.auto-ts-inline-types
If you are not happy with typescript's automatically inferred type, then you can define an explicit type. Hope this helps.

"Currently, vscode de facto allows this in a very limited scope: when using "noImplicitAny": true and a function param is untyped, one can do a quick fix with cmd + . to infer the type."
Currently, this just copy pastes the type from the preview popup, which is shortened with ellipses. So you'll get something like
const createPlateUI: <T extends string = string>(overrideByKey?: Partial<Record<T, PlatePluginComponent<any>>> | undefined) => {     h1: FunctionComponent<any>;     ... 13 more ...;     underline: FunctionComponent<...>; }
Can confirm that TsAutoReturnType extension breaks a lot of the time.
I agree with the other commenters that this would be a useful feature in VSCode - could the issue be reopened?

Looks like this one will do the trick: https://marketplace.visualstudio.com/items?itemName=nick-lvov-dev.typescript-explicit-types

Looks like this one will do the trick: https://marketplace.visualstudio.com/items?itemName=nick-lvov-dev.typescript-explicit-types

If it just had some more commuinty around it it would be great!

Looks like this one will do the trick: https://marketplace.visualstudio.com/items?itemName=nick-lvov-dev.typescript-explicit-types

It only does it one at a time. It can't add types in a bulk. For adding inferred types in a bulk, https://github.com/brandhaug/add-function-return-types can be used. It's a freshly created tool that uses ts-morph under the hood.


Advanced configuration option

Is your feature request related to a problem? Please describe.
Adding more and more functionality is going to make the generator even more complex and give the users a lot of questions. Adding an option to go to run using advanced settings could make this easier
Describe the solution you'd like
Add a a question as one of the initial questions that asks the user if they want to run advanced setup.

Added in #176


refactor for docker container

[ ] refactor code to run in a docker container
[x] add dockerfile and docker-compose.yml

Depends on #10


docs: [IAC-2506] Update IaCM state migration repo reference

Description

Update IaCM state migration repo reference from private to public address.
JIRA

PRs must meet these requirements to be merged:

[x] Successful preview build.
[x] Code owner review.
[x] No merge conflicts.
[x] Release notes/new features docs: Feature/version released to at least one prod environment.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://66ebe149b95641564e389054--harness-developer.netlify.app


Refreshed azure-portal.png screenshot

The existing Azure Portal screenshot was about 2 years old and the Portal doesn't look like that anymore. So, I created a new, refreshed screenshot.

Ping...

@cephalin Is anyone maintaining this project anymore?

Closing as my PR was ignored by maintainers of this project.


in headless mode, don't bother trying to memoize user's alias and env

This is only helpful in electron or browser+proxy mode. We use these memos to amortize the expense of creating a login shell. Since headless is run-and-done, such memoization I think has no positive value.

A fix has been merged into the dev branch. https://github.com/starpit/kui/pull/24


snd: 20.3 -> 21.1

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/snd/versions.
meta.description for snd is: "Sound editor"
meta.homepage for snd is: "http://ccrma.stanford.edu/software/snd"
meta.changelog for snd is: ""
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 21.1 with grep in /nix/store/mykvvpwirh0jwz59mcd83dpq826chs2q-snd-21.1

directory tree listing: https://gist.github.com/af54ec795decf3c0da605eace3b99bb0

du listing: https://gist.github.com/096b3ba2ac9e0f2b91f0a3b4883aa437

Rebuild report (if merged into master) (click to expand)

3 total rebuild path(s)

1 package rebuild(s)

1 x86_64-linux rebuild(s)
1 i686-linux rebuild(s)
0 x86_64-darwin rebuild(s)
1 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
snd

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/mykvvpwirh0jwz59mcd83dpq826chs2q-snd-21.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A snd https://github.com/r-ryantm/nixpkgs/archive/158252e6ff371cc9f20ce43bf1ba7c9df1ccc1a2.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/mykvvpwirh0jwz59mcd83dpq826chs2q-snd-21.1
ls -la /nix/store/mykvvpwirh0jwz59mcd83dpq826chs2q-snd-21.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

snd

Maintainer pings

Closing in favour of #118933.


Translation fix for Chinese

VSCode Version: 1.5.2
OS Version: Windows 10 1607

Steps to Reproduce:

Ctrl-Shift-X to open extension manager
Click any of the extensions in the list

In this case "扩展名" should be "扩展". "扩展" means extension, plugin, etc. while "扩展名" means suffix of filenames e.g. ".txt", ".exe".
This seems to occur only in tabs showing detailed information of extensions.

mistranslated string in extensionsInput.nls.resx.lcl

Waiting for linguistic input.
File: src\vs\workbench\parts\extensions\electron-browser\extensionsInput.nls.resx
ID: ".resx", 0; "extensionsInputName"
SRC: Extension: {0}
TGT: 扩展名: {0}

Fixed as suggested.
File: src\vs\workbench\parts\extensions\electron-browser\extensionsInput.nls.resx
ID: ".resx", 0; "extensionsInputName"
SRC: Extension: {0}
TGT: 扩展: {0}


Indexed Merkle tree

closes https://github.com/o1-labs/o1js/issues/1655
This PR introduces IndexedMerkleMap, an all around better version of MerkleMap. See #1655 for a detailed description, including the API which this PR implements.
In short, there are two motivations to introduce a new Merkle storage primitive:

The old primitives didn't cleanly map to provable types, and are appallingly hard to use in circuits
The old Merkle map is a sparse tree of height 256, which meant a large number of in-circuit hashes.

IndexedMerkleMap uses about 4-8x fewer constraints than MerkleMap when used with height 31 (which supports 1 billion entries). Here are some constraint counts for different operations:
indexed merkle map (get) 1005

sparse merkle map (get) 4208

indexed merkle map (insert) 1382
indexed merkle map (update) 950
indexed merkle map (set) 1958

sparse merkle map (set) 8160

@dfstio I ran into the same and fixed it here: https://github.com/o1-labs/o1js/pull/1666/commits/33c216e69b7c57a895d850831f1ad00b52ddde3b

@dfstio

you're not supposed to use proveInclusion() at all. it's a helper method. the user-facing API is get(), set(), update(), insert()
see my last commit for how to use it in provable code: https://github.com/o1-labs/o1js/pull/1666/commits/f173c596bdfdd97e45eb7a897bc672d34ff3c881

Filling the map is now 3.5 times faster:

Btw, the set() API is definitely not optimized for filling out a map quickly :D
But rather, to prove insertion while keeping the number of constraints low

you're not supposed to use proveInclusion() at all. it's a helper method. the user-facing API is get(), set(), update(), insert()

The problem with get() is that I cannot pass the index to it. When I use proveInclusion() directly, it takes 175 constraints, and get(), as you have mentioned above, takes 1005 constraints.
My statistics for proving the inclusion of the array of 10 key-value pairs in the MerkleMap of 1000 entries with proveInclusion():
IndexedMerkleMap:
map fill: 16.250s
compiled: 2.068s
constraints 1746
prove: 12.787s

MerkleMap:
map fill: 57.924s
compiled: 35.682s
constraints 42071
prove: 23.363s

it's a fantastic decrease in the number of constraints))

The problem with get() is that I cannot pass the index to it

Can you elaborate on your use case? Because the index is supposed to be an internal detail. It's supposed to be a key value map, and you access elements by key.
Also I'm worried that if you just use proveInclusion(), you're not proving everything that's needed.
Nonetheless, your comments gave me a good idea: Add a cheaper version of get() which asserts that the key-value pair is included (the current version gracefully handles the case when it's not included, and returns an option)

When I use proveInclusion() directly, it takes 175 constraints

Also, this just seems wrong, or like you're passing in constants, or using a smaller height.
Proving inclusion means doing at least height - 1 hashes, which for height 31 means at least 30 * 11 = 330 constraints.

Can you elaborate on your use case? Because the index is supposed to be an internal detail. It's supposed to be a key value map, and you access elements by key.

I'm keeping NFT private key-value pairs as MerkleMap. I need to prove that those keys exist and are set to specific values. What I did before was create a new redacted MerkleMap that contains disclosed key values and then prove that the elements are included in both Merkle Maps by creating proofs for each key-value element and then merging them.
Given the new number of constraints, I can put many key-value proofs in one ZkProgram and then consume the proofs for different array sizes in SmartContract using the MerkleTree with verification keys and side-loading verification keys.
I prefer to get indexes in non-provable code and then pass them to the provable code to decrease the number of constraints.

Proving inclusion means doing at least height - 1 hashes, which for height 31 means at least 30 * 11 = 330 constraints.

I'm using height 11. I can put 1024 elements in the IndexedMerkleMap of this height. It is 10* 11 = 110 constraints, plus some checks on Fields add more, for a total of 175.
By the way, even ZkProgram with 0 constraints takes at least 8 seconds to create a proof; for some reason, it is much slower than SmartProgram with the same number of constraints.

Thanks for your feedback @dfstio. Based on it, I added a new version of get() which only proves inclusion and made it the default version. The old version, which also handled non-inclusion, is called getOption() now.
Here are the numbers for height 11, it matches with what you wrote:
indexed merkle map (get) 176
indexed merkle map (get option) 405
sparse merkle map (get) 4208

indexed merkle map (insert) 542
indexed merkle map (update) 350
indexed merkle map (set) 758
sparse merkle map (set) 8160

Nonetheless, your comments gave me a good idea: Add a cheaper version of get() which asserts that the key-value pair is included (the current version gracefully handles the case when it's not included, and returns an option)

It would be a great addition. Proving inclusion and exclusion (for nullifiers) are very important operations that should probably be reflected in the IndexedMerkleMap methods.
And given that data is public, I can do myself toJSON () and fromJSON () to/from base64.

The old version, which also handled non-inclusion, is called getOption() now.

Can the version that will only prove exclusion (non-inclusion) take less than 405 constraints, similar to the get() that takes half of it?

Can assertIncluded() return a value for the key?

Can assertIncluded() return a value for the key?

the version of assertIncluded() which returns a value already exists - it's get()!

How do we serialize the witness to calculate recursive proofs using many workers?
If I want to create a proof confirming that I've correctly added ten key-values to the IndexedMerkleMap and want to split the calculation between 10 separate workers running in parallel, each calculating the proof to be merged later for one key-value pair, I would need to be able to generate a serializable witness to be passed to the worker. Otherwise, I should serialize the whole map, which would take much longer.
Effectively, for this use case, computeRoot() should be split to map.getWitness(key) and computeRoot(witness), with the witness being easily serializable.
map.getWitness() should be called in the master worker for all ten key-value pairs, and computeRoot() should be called in the provable code for each of the ten workers. Each worker should not have the map, just a witness.
Example when it is needed: Serializing Proving

How do we serialize the witness to calculate recursive proofs using many workers?

Interesting challenge!

map.getWitness() should be called in the master worker for all ten key-value pairs, and computeRoot() should be called in the provable code for each of the ten workers. Each worker should not have the map, just a witness.

I thought about this a bit, and think it can be done in a way that is compatible with the current design of the IndexedMerkleTree data structure.
The idea is that the current implementation should work if you don't have the full tree, but just the subset that are touched by your updates. This is quite similar to what you propose, since in the end a collection of Merkle witnesses is also just a subset of the tree.
There are two internal data structures: nodes and sortedLeaves. Both should currently allow pruning to the values you actually need. For nodes, you'd need to store arrays of the same length, but mostly filled with empty slots, not sure how much memory that saves. In the case of sortedLeaves, only having a subset should just work.
So for parallel proving, we could:

Run all tree updates serially, without proving
Take a pruned snapshot every k updates
Send each pruned snapshot to a worker, which will perform a proof of k updates
Merge those proofs in a tree to get a single proof

Actually this is extremely close to what Mina does with transaction proofs :D
*

Note to self / reviewers: the implementation currently doesn't do proof of updates correctly. The problem is that it doesn't connect the Merkle path for the update with the path previously validated against the old commitment

For nodes, you'd need to store arrays of the same length, but mostly filled with empty slots, not sure how much memory that saves

I've done some testing with MerkleTree to evaluate serialized map size and serialized MerkleTree witness size, and the results are as follows:
random indexes in the tree:
height: 11, elements: 1000, tree size: 75,508 chars, witness size: 495 chars (0.66%)
height: 20, elements: 10000, tree size: 3,243,225 chars, witness size: 860 chars (0.03%)
height: 30, elements: 10000, tree size: 8,159,377 chars, witness size: 1,312 chars (0.02%)
height: 50, elements: 10000, tree size: 18,475,240 chars, witness size: 2,211 chars (0.012%)
height: 100, elements: 100000, tree size: 456,081,607 chars, witness size: 4,507 chars (0.0010%)
height: 255, elements: 25000, tree size: 405,075,106 chars, witness size: 11,600 chars (0.0029%)

ordered indexes in the tree:
height: 11, elements: 1000, tree size: 93,319 chars, witness size: 499 chars (0.53%)
height: 20, elements: 10000, tree size: 941,828 chars, witness size: 910 chars (0.096%)
height: 30, elements: 10000, tree size: 942,558 chars, witness size: 1,367 chars (0.15%)
height: 50, elements: 10000, tree size: 943,572 chars, witness size: 2,283 chars (0.24%)
height: 100, elements: 100000, tree size: 9,526,927 chars, witness size: 4,567 chars (0.048%)
height: 255, elements: 100000, tree size: 9,535,772 chars, witness size: 11,662 chars (0.12%)

The IndexedMerkleMap should be closer to ordered indexes, so by creating a witness or pruned snapshot we can decrease the serialized witness size by circa 1000x. Btw, it also shows the serialized map size savings IndexedMerkleMap will bring: it should be 170x (16k per element in MerkleMap vs 95 bytes per element with IndexedMerkleMap)

Take a pruned snapshot every k updates

We need to take a pruned snapshot BEFORE running the circuit without proofs and make sure that low leaves are also included.

Actually this is extremely close to what Mina does with transaction proofs, where snapshots of the ledger are updated :D

I believe that IndexedMerkleMap is extremely important for rollups on Mina protocol and will save a lot of money in proving costs


Support for the browsers native encoder/decoder api

I just heared that chrome added an API to support raw native h264 encode and decode.
https://caniuse.com/mdn-api_videoencoder
https://caniuse.com/mdn-api_videodecoder
https://caniuse.com/mdn-api_audioencoder
https://caniuse.com/mdn-api_audiodecoder
https://github.com/vjeux/mp4-h264-re-encode
I'm not too familiar with emscripten, but it looks like it is possible to share memory between javascript and c.
So my idea is to implement those decoder/encoder apis into ffmpeg as sort of an hardware acceleration.
This could resolve some of the "slow" tickets like: https://github.com/ffmpegwasm/ffmpeg.wasm/issues/326 https://github.com/ffmpegwasm/ffmpeg.wasm/issues/206
It could also make it viable to run ffmpeg single core and avoid the isolation requirements: https://github.com/ffmpegwasm/ffmpeg.wasm/issues/285
We'd still have the c implementation of said codecs as a fallback, the demuxers and muxers and we'd still have all the filters of ffmpeg. So I think it is still worth it to use ffmpeg.

Some part of me thinks this is outside of the scope of this project, as it is meant to be a port of FFMPEG into a Javascript environment -- not an "all-purpose best-built video encoder for the browser". I think your idea could lead to the potential for a larger library that wraps multiple solutions like ffmpeg.wasm, VideoEncoder API, MediaRecorder API, etc
However, I generally like the idea and where your head is at


Memory item name change

Currently, the memory item is represented as a solely as a number in the units of MB or GB. There should be the text "Mem: " before the number to signify to the user that this is the total memory usage of the process.

Closed by #99


Enhanced ScrollTop Button Functionality

Enhanced ScrollTop Button Functionality
Fix #1096

Sets the initial visibility of the scrollTop button to hidden, improving the overall user interface by reducing clutter.
The button becomes visible only after scrolling down more than 100 pixels, enhancing usability and encouraging seamless navigation back to the top.

@vivekvardhan2810
Pls assign me, give me labels like gssoc-extd, hacktoberfest-accepted, and level.

@ygowthamr please mention the issue number.

@vivekvardhan2810 done mentioned

@ygowthamr @shivhere007 thank you for the contribution to ML Fusion Lab.
your contribution gives a value to us.
now please do work on other projects.


[SSD] learning rate compliance

Forced learning rate to be a multiple base learning rate (2.5e-3)
depends on https://github.com/mlperf/training/pull/418

@googlebot I consent.

@abhinavvishnu you are correct, the learning rate for SSD should be unconstrained.
Closing this PR


Visualize Regions or Replicas

Feature Request
Is your feature request related to a problem? Please describe:
We commonly face these issues:

How many miss replicas are there? Do they distribute on a specific host?
Is the leader distribution of a table balanced for each store?
Is the read / write flow of a table balanced for each store?
What's the proportion of the hot / cold data for each table / database?
What is the distribution of replica sizes, or keys in replica? (so that one can decide a better region merge configuration)

Describe the feature you'd like:
Issues listed above can be all answered by the region info accessible via region command in pd-ctl. However pd-ctl only produce raw data. People usually need to rely on other tools (which uses region info data under the hood), like writing a script to analyze, or using SQL command to query distributions.
We can provide a visualization / analytics page in TiDB Dashboard for region info, so that it can be easy to know everything about region info. The detail functionality of the page is listed in this proposal: https://docs.google.com/document/d/1moQVhvIgqu_FWuv_UMB76AM5tETJkyeWnzI04wyevhk/edit#heading=h.8xipn1t7e2cb
The page can be implemented by using the awesome visualization library SandDance.
This feature is a subset of https://github.com/pingcap-incubator/tidb-dashboard/issues/548

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

Description：Visualize Regions or Replicas
Category：Feature、Usability
Value： Simplify the debug method of replica/region and provide more fine-grained observability of TiDB
Value Score: 3

The link about how to use pd-ctl


Migrate to v2 addon format

The addon is converted to latest v2 format
https://github.com/NullVoxPopuli/ember-addon-migrator codemod is used to update files structure.
Resolves https://github.com/qonto/ember-lottie/issues/43

Closed in favour of https://github.com/qonto/ember-lottie/pull/63


How to access the map after creation

No big deal if you guys are busy. I was wondering if there is a way to expose the <GoogleMap /> object?
In another google react map module it allows me to access it like this:
<GoogleMap
          ...
          onGoogleApiLoaded={({ map }) => drawPolygons(map, center.lat, center.lng)}
        >
</GoogleMap>

Couldn't find any way in the docs so....

Sorry, no. Only react ref:
https://tomchentw.github.io/react-google-maps/#!/SearchBox


Upgrade terraform-provider-google-beta to v6.13.0

This PR was generated via $ upgrade-provider pulumi/pulumi-gcp.

Upgrading terraform-provider-google-beta from 6.12.0  to 6.13.0.
Fixes #2719

Fixed via https://github.com/pulumi/pulumi-gcp/pull/2733.


False Advertising. There are dependencies to this project.

They just don't need to be included because they are wrapped up into a single file.
It would be as if I said "had no dependency on jquery" but included the version of jQuery I wanted with no conflict enabled and set to a locally scoped variable.
This project has two external dependencies: tiny-emitter and good-listener.

What I meant by "no dependencies" is that there's no framework dependency involved. No jQuery, React, Angular, Ember. If you take a look at the dependencies listed in package.json you'll see that they all were made by me, except for one (which I'll rewrite at some point). The only reason why they are in separate packages is because they are generic enough to help other people.

@zenorocha If possible, I would suggest create some action on these kind of feedback. In this case it might be useful to use "No vendor dependencies" or explaining what "no dependencies" mean in the docs. Probably most devs that read the docs are not aware that the "no dependencies" represent that there's no framework of library required for this, and most of them will not look for clarification in the issues.
The first thing one might think is that this kind of stuff is not important, but if we really want to make an impact and create something meaningful for everyone, sometimes it might be interesting to reduce the margin for interpretation in the docs.
This seems similar on two pieces of feedback I received a few weeks ago. That issue managed to receive a second feedback because the issue was kept open for sometime to see if someone was going to comment on it.
It was very useful for that project and I hope it can be useful here too 👍

@FagnerMartinsBrack I honestly think this is super irrelevant and has nothing to do with "impact" or "meaningfulness". There are tons of more important things to be done in this project (and in life).
Changing to "no frameworks" instead of "no dependencies".


Converting circular structure to JSON Error

https://docs.xpla.io/docs/develop/tools/xpla-js/get-started-with-xpla-js/
import fetch from "isomorphic-fetch";
import { MsgSend, MnemonicKey, Coins, LCDClient } from<EMAIL_ADDRESS>
// Fetch gas prices and convert to `Coin` format.
const gasPrices = await (
  await fetch("https://cube-fcd.xpla.dev/v1/txs/gas_prices", { redirect: 'follow' })
).json();
const gasPricesCoins = new Coins(gasPrices);

const lcd = new LCDClient({
  URL: "https://cube-lcd.xpla.dev/",
  chainID: "cube_47-5",
  gasPrices: gasPricesCoins,
  gasAdjustment: "1.5",
  gas: 10000000,
});

When I used this example code, I received the following error message:
[Nest] 24091  - 2023. 10. 12. 오후 6:34:32   ERROR [ExceptionsHandler] Converting circular structure to JSON
    --> starting at object with constructor 'LCDClient'
    |     property 'auth' -> object with constructor 'AuthAPI'
    --- property 'lcd' closes the circle

What is the probloem?

@LChanwoo Have you read this? https://stackoverflow.com/questions/64735881/typeerror-converting-circular-structure-to-json-starting-at-object-with-con
According to the article, response should be used as response.data when fetching url. So when you first try fetch, maybe you should use gasPrices.data, not gasPrices (before using the json function). Give that a try. I'm not good at nest.js, but I think this is a nest.js grammar problem. When I ran almost the same code in nodejs, there was no error.
This is node.js code.
const { MnemonicKey, LCDClient, Coins } = require("@xpla/xpla.js");
const fetch = require("isomorphic-fetch");

const main = async () => {
    const gasPrices = await (
        await fetch("https://cube-fcd.xpla.dev/v1/txs/gas_prices", { redirect: 'follow' })
    ).json();

    const gasPricesCoins = new Coins(gasPrices);

    const lcd = new LCDClient({
        URL: "https://cube-lcd.xpla.dev/",
        chainID: "cube_47-5",
        gasPrices: gasPricesCoins,
        gasAdjustment: "1.5",
        gas: 10000000,
    });

  const txInfo = await lcd.tx.txInfo("2426057C73D7A3499388A8FACF226F760B891BDE3C46149FE69AF5049E31D63D"); // sample tx data
  console.log(JSON.stringify(txInfo, null, 2));
}
main()


Language injections in Ruby Heredoc

Check for existing issues

[X] Completed

Describe the bug / provide steps to reproduce it
I've seen this done in other editors before, and it's nice when you need to write some other language inside a Ruby string. Given a heredoc with a terminator that matches a language name or file extension, some editors (like Sublime Text pictured here) will use language injection to change the syntax highlighting for the block.

In Zed the string is highlighted entirely as a Ruby string, which is not wrong, but not as helpful as it could be.

This is perhaps similar to #725, and is referenced in the Zed blog here as being something that Zed is capable of handling
Environment
Zed: v0.104.2 (preview)
OS: macOS 13.5.0
Memory: 64 GiB
Architecture: aarch64
If applicable, add mockups / screenshots to help explain present your vision of the feature
No response
If applicable, attach your ~/Library/Logs/Zed/Zed.log file to this issue.
If you only need the most recent lines, you can run the zed: open log command palette action to see the last 1000.
No response

We do have this functionality for markdown, just need to add it to our ruby language support. Thanks for the report!


 I will take some time to go through it. I’ll let you know if I have any questions.

@emmanuelkehinde Any updates?

@emmanuelkehinde Another mention, sorry.

@TacoTheDank Sorry for being away for a while. Some updates have however been made to the project that may require you to look into and possibly update the PR. Sorry about the inconvenience.

@emmanuelkehinde It's fine :). I have updated the PR (read over the first post).

@emmanuelkehinde It's fine :). I have updated the PR (read over the first post).

Thanks @TacoTheDank . I will take a look at it.

@TacoTheDank Thanks for the Pull Request. I really appreciate it.
Kindly check out the comments and let me know what you think. I would be doing thorough testing on my local with necessary credentials to confirm everything still works fine.

Hi @TacoTheDank
I'm sorry this PR took long. This is as a result of a pending decision to utilize KMM in the project. This has been done and the project now has a shared KMM code. So the PR might not be needed anymore.
But feel free to contribute to any of the current code. It would be appreciated.
Thanks for your patience.

Hi @TacoTheDank
I'm sorry this PR took long. This is as a result of a pending decision to utilize KMM in the project. This has been done and the project now has a shared KMM code. So the PR might not be needed anymore.
But feel free to contribute to any of the current code. It would be appreciated.
Thanks for your patience.

It's fine :) I've already updated the PR, so it's good to review.

Any reason why this app needs google services?


cFE Integration candidate: 2021-08-24

Describe the contribution
PR #1803
Add software bus tests

Fix #1680
Fix #1681
Fix #1682

PR #1756

Fix #1755, separate variable for OSAL status

PR #1809

Fix #1807, increase SB pool max size bucket

PR #1842

Fix #1840, Add Null check for CFE_ResourceId_FindNext

PR #1844

Fix #1840, Hotfix for strict resource IDs.

PR #1828
Improve tbl coverage tests

Fixes #472
Fixes #1792
Fixes #1793
Fixes #1795
Fixes #1791
Fixes #1789
Fixes #1788

PR #1833

Fix #1824, Add missing cases for msg id func tests

PR #1832

Fix #1831, Consolidate msg get/set doxygen group

PR #1836

Fix #473, Adding coverage tests to cFE TIME

PR #1848

Fix #1847, enable strict resource id w/OMIT_DEPRECATED

PR #1845

HOTFIX IC-20210819, type correction TBL coverage test

PR #1832

Fix #1831, Consolidate msg get/set doxygen group

PR #1806

Fix #1805, Add test for ES BackgroundWakeup

PR #1813

Fix #1786, Success Test for CFE_ES_RestartApp

PR #1814

Fix #1794, Subscribe to Message Limit Greater Than CFE_PLATFORM_SB_DEFAULT_MSG_LIMIT

PR #1811

Fix #1787, Success Test for CFE_ES_GetMemPoolStats

PR #1822

Fix #1821, Group MSG APIs documentation by header type

PR #1816

Fix #1815, add retroactive CFE status asserts

PR #1854

Fix #1851, remove unused CFE_TBL_ERR_BAD_APP_ID

PR #1855

Fix #1852, correct syslog message in pool create

PR #1853

Fix #1850, remove unused CFE_ES_POOL_BOUNDS_ERROR

PR #1859

Fix #1858, remove unused CFE_TBL_ERR_FILE_NOT_FOUND
Fix #1863, Remove comment referencing CFE_TBL_ERR_FILE_NOT_FOUND

PR #1856

Fix #1827 Check error ctr to TransmitMsg test

PR #1857

Fix #1771, End Child Task requirement remove error code

PR #1782

Fix #1679, Add functional tests for resource misc

Testing performed
See PRs
Expected behavior changes
See PRs
System(s) tested on
Ubuntu CI
Additional context
Part of https://github.com/nasa/cFS/pull/340
Third party code
None
Contributor Info - All information REQUIRED for consideration of pull request

Workflows passing in fork when using integration-candidate bundle
https://github.com/astrogeco/cFE/runs/3386692864


Update podman_tutorial.md

Fix the familiarizing-yourself-with-podman link.

LGTM

LGTM

@mheon
The previous commition was directly operated in github page, and I added the sign & re-push.
Please check it, thanks!

@warmchang Signoff looks good, thanks

bot, add author to whitelist

LGTM, good for merge once tests are green

@rh-atomic-bot r+


Versioning  active model

Ticket
https://netguru.atlassian.net/browse/PEOPLE-128
Description
Versioning via ActiveModel based on: http://strikingly.github.io/blog/2015/09/14/Simple-rails-history-pattern-ActiveRecord/
This is only backend solution.
At beginning this solutions seems to be better, than using gem, as:

we don't have to remember to update serialized data, when column names are changed / added. Gems provide versioning via serializing model into one field - so if we plan to change column name, or add new columns, we also have to update all already serialized columns
Easy manipulate data on SQL level - operations like grouping by date, can be done in SQL, when we keep old records in separate tables, instead of serializing them

However, right now it feels like a Goldberg machine

It provides unnecessary complexity, to a simple task, just to be able to handle corner cases, a bit more efficient.
Before merge checklist

[x] CircleCI is passing
[ ] Proper labels are set

We will use something similar to this one.

by far, lgtm 👍
( except specs :) )


Compare documents

Description
Closes #1521.

Readiness checklist

[ ] I added tests for new functionality or bugfixes.
[x] I ran task all, and it passed.
[x] I added/updated comments for both exported and unexported top-level declarations (functions, types, etc).
[ ] I checked comments rendering with task godocs.
[x] I ensured that the title is good enough for the changelog.
[x] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
[ ] I marked all done items in this checklist.

Need more work on this.

Build fails currently :thinking:
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/debian:11.5
#3 ...

#4 [auth] library/debian:pull token for registry-1.docker.io
#4 DONE 0.0s

#3 [internal] load metadata for docker.io/library/debian:11.5
#3 DONE 1.0s

#5 [1/1] FROM docker.io/library/debian:11.5@sha[25](https://github.com/FerretDB/FerretDB/actions/runs/3637778243/jobs/6139182202#step:6:26)6:a[28](https://github.com/FerretDB/FerretDB/actions/runs/3637778243/jobs/6139182202#step:6:29)8aa7ad0e4d443e86843972c25a02f99e9ad6ee589dd764895b2c3f5a8340b
#5 resolve docker.io/library/debian:11.5@sha256:a288aa7ad0e4d443e86843972c25a02f99e9ad6ee589dd764895b2c3f5a8340b done
#5 sha256:a4b14cf13428c71a5fce6d9380eb2ac1452a7ae4253e594dc90c8a884fdf3db2 5[29](https://github.com/FerretDB/FerretDB/actions/runs/3637778243/jobs/6139182202#step:6:30)B / 529B done
#5 sha256:291bf168077c554abde831d52178c76aa8d7dad0e55a2e37a68c2b55a5d92e64 1.46kB / 1.46kB done
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548[30](https://github.com/FerretDB/FerretDB/actions/runs/3637778243/jobs/6139182202#step:6:31)9fa240155067ab0e698 0B / 55.04MB 0.1s
#5 sha256:a288aa7ad0e4d443e86843972c25a02f99e9ad6ee589dd764895b2c3f5a8340b 1.85kB / 1.85kB done
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 7.34MB / 55.04MB 0.2s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 15.73MB / 55.04MB 0.3s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 24.12MB / 55.04MB 0.4s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 32.51MB / 55.04MB 0.5s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 55.04MB / 55.04MB 0.7s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 55.04MB / 55.04MB 0.7s done
#5 extracting sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698
#5 extracting sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 1.6s done
#5 DONE 2.4s

#6 exporting to image
#6 exporting layers done
#6 writing image sha256:5a652491d05f00366d8d337346f4a11f70aa635b1[31](https://github.com/FerretDB/FerretDB/actions/runs/3637778243/jobs/6139182202#step:6:32)b7cf9f7a7ab69d0344a05 done
#6 naming to docker.io/library/ferretdb-debian done
#6 DONE 0.0s
Selecting previously unselected package ferretdb.
(Reading database ... 6661 files and directories currently installed.)
Preparing to unpack /deb/ferretdb.deb ...
Unpacking ferretdb (0.7.0~79-g7aafd6f) ...
Setting up ferretdb (0.7.0~79-g7aafd6f) ...
ferretdb: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.[32](https://github.com/FerretDB/FerretDB/actions/runs/3637778243/jobs/6139182202#step:6:33)' not found (required by ferretdb)
ferretdb: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.[34](https://github.com/FerretDB/FerretDB/actions/runs/3637778243/jobs/6139182202#step:6:35)' not found (required by ferretdb)
task: Failed to run task "packages": task: Failed to run task "packages-deb": exit status 1
Error: Process completed with exit code 1.

Build and codev fail currently thinking
Packages / Build packages (pull_request) Failing after 3m
#5 [1/1] FROM docker.io/library/debian:11.5@sha[25]
...
ferretdb: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.[32](https://github.com/FerretDB/FerretDB/actions/runs/3637778243/jobs/6139182202#step:6:33)' not found (required by ferretdb)
ferretdb: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.[34](https://github.com/FerretDB/FerretDB/actions/runs/3637778243/jobs/6139182202#step:6:35)' not found (required by ferretdb)
task: Failed to run task "packages": task: Failed to run task "packages-deb": exit status 1
Error: Process completed with exit code 1.

codev
[2022-12-07T10:04:57.851Z] ['error'] There was an error running the uploader: Error uploading to [https://codecov.io:](https://codecov.io/) Error: There was an error fetching the storage URL during POST: 503 - upstream connect error or disconnect/reset before headers. reset reason: connection failure
[2022-12-07T10:04:57.852Z] ['verbose'] The error stack is: Error: Error uploading to [https://codecov.io:](https://codecov.io/) Error: There was an error fetching the storage URL during POST: 503 - upstream connect error or disconnect/reset before headers. reset reason: connection failure
    at main (/snapshot/repo/dist/src/index.js)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-12-07T10:04:57.852Z] ['verbose'] End of uploader: 337 milliseconds
Error: Codecov: Failed to properly upload: The process '/home/runner/work/_actions/codecov/codecov-action/v3/dist/codecov' failed with exit code 255

Fixed on main now thanks :pray:

Skipped integration tests failing on tigris.


Queryable projections: Inline user-implemented mapping expressions from different file

Is your feature request related to a problem? Please describe.
When having mappings defined in different classes/files, I get RMG068: Cannot inline user implemented queryable expression mapping. In contrast, when I move the two mapper definitions into one class, the mapping gets propertly inlined.
Describe the solution you'd like
A clear and concise description of what you want to happen including a proposal on the API surface and the matching generated code.

public class Car
{
    public string Model { get; set; }
}
public class CarDto
{
    public string ModelName { get; set; }
}

public class B
{
    public Car? Value { get; set; }
}
public class BDto
{
    public CarDto? Value { get; set; }
}

[Mapper]
public static partial class CarMapper
{
    [MapProperty(nameof(Car.Model), nameof(CarDto.ModelName))]
    public static partial CarDto MapCar(Car src);

    private static partial BDto Map(B src);
    public static partial IQueryable<BDto> ThisWorks(this IQueryable<B> src);
        [global::System.CodeDom.Compiler.GeneratedCode("Riok.Mapperly", "<IP_ADDRESS>")]
        public static partial global::System.Linq.IQueryable<global::Vexillum.WebApp.AutoMapper.BDto> ThisWorks(this global::System.Linq.IQueryable<global::Vexillum.WebApp.AutoMapper.B> src)
        {
#nullable disable
            return System.Linq.Queryable.Select(src, x => new global::Vexillum.WebApp.AutoMapper.BDto()
            {
                Value = new global::Vexillum.WebApp.AutoMapper.CarDto()
                {
                    ModelName = x.Value.Model,
                },
            });
#nullable enable
        }
}

[Mapper]
[UseStaticMapper(typeof(CarMapper))]
public static partial class BMapper
{
    // RMG068: Cannot inline user implemented queryable expression mapping
    private static partial BDto Map(B src);
    public static partial IQueryable<BDto> ThisShouldWork(this IQueryable<B> src);
        [global::System.CodeDom.Compiler.GeneratedCode("Riok.Mapperly", "<IP_ADDRESS>")]
        public static partial global::System.Linq.IQueryable<global::Vexillum.WebApp.AutoMapper.BDto> ThisShouldWork(this global::System.Linq.IQueryable<global::Vexillum.WebApp.AutoMapper.B> src)
        {
#nullable disable
            return System.Linq.Queryable.Select(src, x => new global::Vexillum.WebApp.AutoMapper.BDto()
            {
                Value = global::Vexillum.WebApp.AutoMapper.CarMapper.MapCar(x.Value),
            });
#nullable enable
        }

Describe alternatives you've considered
Copying all dependend configurations into one class works as ugly workaround.
Additional context
Mapperly v3.6.0

We decided to not support this in the first iteration of inlining, since the source syntax of external methods is not always available (e.g. if it is part of another assembly). However, this could be improved and inlining could always happen if the source syntax is available.

I have tried to provide a PR for this, but struggle as PartialImplementationPart is always null. Furthermore, even having the PartialImplementationPart would not be enough as Mapperly uses multi-statement bodies by default. Do you have any guidance how to get around this?
Failing Unit-Test:
namespace Riok.Mapperly.Tests.Mapping;

public class QueryableProjectionExternalTest
{
    [Fact]
    public Task UseStaticMapperInlinedExpression()
    {
        var source = TestSourceBuilder.CSharp(
            $$"""
            using System;
            using System.Linq;
            using System.Collections.Generic;
            using Riok.Mapperly.Abstractions;
            using Riok.Mapperly.Abstractions.ReferenceHandling;

            [Mapper]
            [UseStaticMapper(typeof(OtherMapper))]
            public partial class Mapper
            {
                private partial IQueryable<B> Map(IQueryable<A> source);
            }
            [Mapper]
            static partial class OtherMapper
            {
                public static partial D MapToD(C v);
            }
            class A { public string StringValue { get; set; } public C NestedValue { get; set; } }
            class B { public string StringValue { get; set; } public D NestedValue { get; set; } }
            class C { public string Value { get; set; } }
            class D { public string Value { get; set; } }
            """
        );

        return TestHelper.VerifyGenerator(source);
    }
}

P.S. While debugging Mapperly #1422 happened as a side-effect.

PartialImplementationPart is probably null because it is compiled in another assembly / compilation reference and not visible to the current compilation. This is where it get's complicated and this is also the reason why we didn't implement it in the first place. Not sure if it works with https://github.com/riok/mapperly/pull/1418 (compilation references)?


Update CHANGELOG about v0.13.1 release. (#1174)

(cherry picked from commit 256ea4da790e796c40ce0f1189d14f160a5312af)

Codecov Report

Merging #1177 into v0.13.x will not change coverage.
The diff coverage is 76%.

@@            Coverage Diff             @@
##             v0.13.x    #1177   +/-   ##
==========================================
  Coverage      82.01%   82.01%
  Complexity      1226     1226
==========================================
  Files            192      192
  Lines           5933     5933
  Branches         551      551
==========================================
  Hits            4866     4866
  Misses           920      920
  Partials         147      147

Impacted Files
Coverage Δ
Complexity Δ

...porter/stats/prometheus/PrometheusExportUtils.java
95.74% <100%> (ø)
16 <6> (ø)
:arrow_down:

.../opencensus/contrib/zpages/StatszZPageHandler.java
89.52% <100%> (ø)
45 <0> (ø)
:arrow_down:

...ter/stats/prometheus/PrometheusStatsCollector.java
64.91% <40%> (ø)
7 <0> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c3f383e...b80234a. Read the comment docs.


calamares: 3.2.0 -> 3.2.1

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools.
This update was made based on information from https://repology.org/metapackage/calamares/versions.
These checks were done:

built on NixOS
Warning: no invocation of /nix/store/krqvinap3lgbn4ka1qnshi9mxvlj9syl-calamares-3.2.1/bin/calamares had a zero exit code or showed the expected version
0 of 1 passed binary check by having a zero exit code.
0 of 1 passed binary check by having the new version present in output.
found 3.2.1 with grep in /nix/store/krqvinap3lgbn4ka1qnshi9mxvlj9syl-calamares-3.2.1
directory tree listing: https://gist.github.com/9765935d9d60806e75255e4a33bbba21
du listing: https://gist.github.com/c356abacb504a32f4376ca8fd2e58638

meta.description for calamares is: '"Distribution-independent installer framework"'.
cc @manveru for testing.
Experimental: this build is cached with Cachix. To use the cache follow these experimental instructions:
One time setup in nixpkgs Git checkout:
cachix use r-ryantm
git remote add r-ryantm https://github.com/r-ryantm/nixpkgs.git

Test this build:
git fetch r-ryantm && git checkout a793b31caa60493614b7e3f9860c1f936c8c0592 && nix-shell --pure -I nixpkgs=. -p calamares

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: calamares
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Attempted: calamares
Partial log (click to expand)
shrinking /nix/store/krqvinap3lgbn4ka1qnshi9mxvlj9syl-calamares-3.2.1/lib/calamares/modules/users/libcalamares_viewmodule_users.so
shrinking /nix/store/krqvinap3lgbn4ka1qnshi9mxvlj9syl-calamares-3.2.1/lib/calamares/modules/netinstall/libcalamares_viewmodule_netinstall.so
shrinking /nix/store/krqvinap3lgbn4ka1qnshi9mxvlj9syl-calamares-3.2.1/lib/libcalamaresui.so.3.2.1
shrinking /nix/store/krqvinap3lgbn4ka1qnshi9mxvlj9syl-calamares-3.2.1/lib/libcalamares.so.3.2.1
gzipping man pages under /nix/store/krqvinap3lgbn4ka1qnshi9mxvlj9syl-calamares-3.2.1/share/man/
strip is /nix/store/4qvrxzxa535y8304mk195x50b6p9607d-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/krqvinap3lgbn4ka1qnshi9mxvlj9syl-calamares-3.2.1/lib  /nix/store/krqvinap3lgbn4ka1qnshi9mxvlj9syl-calamares-3.2.1/bin
patching script interpreter paths in /nix/store/krqvinap3lgbn4ka1qnshi9mxvlj9syl-calamares-3.2.1
checking for references to /build in /nix/store/krqvinap3lgbn4ka1qnshi9mxvlj9syl-calamares-3.2.1...
postPatchMkspecs

Failure on aarch64-linux (full log)
Attempted: calamares
Partial log (click to expand)
[6876/16620] CXX obj/third_party/webrtc/rtc_base/rtc_base_approved_generic/bytebuffer.o
[6877/16620] CXX obj/third_party/webrtc/rtc_base/rtc_base_approved_generic/bufferqueue.o
[6878/16620] CXX obj/third_party/webrtc/rtc_base/rtc_base_approved_generic/copyonwritebuffer.o
[6879/16620] CXX obj/third_party/webrtc/pc/rtc_pc_base/mediamonitor.o
[6880/16620] CXX obj/third_party/webrtc/pc/peerconnection/rtpreceiver.o
[6881/16620] CXX obj/third_party/webrtc/modules/video_processing/video_processing/skin_detection.o
[6882/16620] CXX obj/third_party/webrtc/modules/video_coding/video_coding/video_sender.o
building of '/nix/store/dpmlnvn2i8w2mw96can6k08pdbp4444c-qtwebengine-5.11.0.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/5bgj2y6ywkr13k7xkylmvngxrklnkvis-calamares-3.2.1.drv': 1 dependencies couldn't be built
error: build of '/nix/store/5bgj2y6ywkr13k7xkylmvngxrklnkvis-calamares-3.2.1.drv' failed


Recorder component unable to reconnect to MySQL after lost connection

Home Assistant release with the issue:
0.91.2
Last working Home Assistant release (if known):
I only noticed the issue starting with 0.91.x.
Operating environment (Hass.io/Docker/Windows/etc.):
Docker on x86
Component/platform:
Recorder
Description of problem:
When HA is running about a day or so, I noticed that the sensor graphs in the UI started to flat out although the read values were correct.
Further investigation showed that the recorder component seems to lose the connection to the MySQL(MariaDB) server at one point and then is unable to reconnect due to some transaction that needs to be rolled back first (according to the logs).
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
recorder:
  purge_keep_days: 14
  purge_interval: 1
  db_url: !secret mysql_url

Traceback (if applicable):
2019-04-14 09:39:34 ERROR (SyncWorker_5) [homeassistant.components.recorder.util] Error executing query: (MySQLdb._exceptions.OperationalError) (2006, 'MySQL server has gone away')
2019-04-14 09:39:34 ERROR (SyncWorker_5) [homeassistant.components.recorder.util] Error executing query: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
2019-04-14 09:39:34 ERROR (SyncWorker_5) [homeassistant.components.recorder.util] Error executing query: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
2019-04-14 09:39:34 ERROR (SyncWorker_5) [homeassistant.components.recorder.util] Error executing query: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back

Additional information:
I don't see any issues in the MySQL server logs; the server is running for weeks now (since last system reboot) and I see active HA sessions in the show processlist; output.
See this thread in the community forum for more details:
https://community.home-assistant.io/t/sensor-cards-not-showing-graph-anymore/110507

I see this too under 0.92.x. I was using MariaDB10 on Synology and when initially set up it flew. Lately, I noticed some of my template sensors reporting weird values and after dropping the database, everything came back to life. Sadly, this only last for <12hrs when the logbook fails to recall anything and the history just sits there forever.
Switched to local MariaDB add-on under HassOS (Intel NUC) yesterday and again, flew for the first 12hrs and has now stopped.

Do you have anything like this in your log?
17-03-12 15:31:09 ERROR (Thread-9) [homeassistant.components.recorder] Error during query: 'latin-1' codec can't encode character '\u2013' in position 7: ordinal not in range(256)
I was having the same issue you are seeing and it came out of nowhere.  I added ?charset=utf8 to the end of my db_url and restarted.  The error no longer shows up in the log and my recorder no longer stops recording data.

I already has that suffix in both instances.
3307/hassdb?charset=utf8 (Synology)
/homeassistant?charset=utf8 (local add-on)
For now, I've switched back to the default internal database although this is not ideal as it grows to 1.8GB pretty quickly which causes Dropbox backup management issues for me.

I did see those "ordinal not in range" errors, too. I appended the suggested charset option to my db_url yesterday and so far it looks like that fixed the issue for me.
I'll wait a little longer, but usually the graphs started to flat out within 24h for me.

This for me happens when there is a location update coming from the IOS app. I am not sure what "state" has a dash in it but for me it is coming from IOS. These are the last events recorder sees before it drops connection.

FYI, my issues turned out to be the bug when stream: and shell_commands: are both in use. I disabled my shell commands and HA has settled down and is nice and stable again. I'm now using the MariaDB add-on.

on 0.91.4, I occasionally getting this error. I am not using HASS.IO. I am using manually installed Docker Homeassistant. MariaDB is also manually installed Docker Mariadb.
2019-06-05 15:03:46 ERROR (Recorder) [homeassistant.components.recorder.util] Error executing query: (MySQLdb._exceptions.OperationalError) (2006, 'MySQL server has gone away')

I changed DB connection to TCP/IP to Socket, but did not help.
I changed wiat_time to 8 hours in my.cnf, but did not help much.

when stream: and shell_commands: are both in use

I am not sure what these mean, though I don't think I use them, since I am not on Hass.io.

[mysqld]
wait_timeout		= 86400
It was originally 600. (I am using linuxserver/mariadb image form Docker hub)
Bellow is my speculation.
In HA, I only record a few switches and scripts so, It would be common for my server not producing any db inserts within 10 minutes window. After the period, I image HA looses SQL connection, hence the error.
Increasing the said entry to over 20000 seems to mitigate the error occurrence. I don't have any idea about the negative side-effect, I mean whether it exists or not.


fix(region): validate invalid character for password

这个 PR 实现什么功能/修复什么问题:
校验密码中不支持的字符

[x] 功能、bugfix描述
[x] 冒烟测试
[ ] 单元测试编写

是否需要 backport 到之前的 release 分支:

release/3.5

/area region
/cc @zexi

Codecov Report

Merging #9205 (41944b0) into master (7f3c0d5) will increase coverage by 11.73%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           master    #9205       +/-   ##
===========================================
+ Coverage    5.83%   17.57%   +11.73%
===========================================
  Files         838       90      -748
  Lines      130487     9043   -121444
===========================================
- Hits         7611     1589     -6022
+ Misses     122194     7279   -114915
+ Partials      682      175      -507

Flag
Coverage Δ

aFlag
17.57% <100.00%> (+11.73%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/util/seclib2/seclib.go
88.23% <100.00%> (+2.18%)
:arrow_up:

pkg/compute/guestdrivers/aws.go

pkg/compute/models/dns_zones.go

pkg/compute/models/dbinstanceresource.go

pkg/compute/models/schedtagjoint.go

pkg/compute/models/instance_snapshots.go

pkg/multicloud/ucloud/securitygroup.go

pkg/baremetal/tasks/basedeploy.go

pkg/keystone/models/expandidps.go

pkg/baremetal/tasks/base.go

... and 729 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f3c0d5...41944b0. Read the comment docs.

/hold

/hold cancel

/lgtm
/approve


Decouple Chain from networking

What is wrong?
The evm.p2p.lightchain.LightChain extends the Chain class.  This is looking to be problematic as I implement the various processes in trinity.  The JSON-RPC server needs access to the Chain object.   In order to do that as things are written, we'd need to setup a multiprocessing.Manager in the networking process to expose the various chain methods for the JSON-RPC process.
This is problematic because it allows for requests to the JSON-RPC server to compete with the execution of the networking processes.
How can it be fixed
I would like to create a new class type to contain the networking specific code that currently lives in the LightChain class.  Maybe something like ChainNetwork or NetworkManager.  This class can then interact with the Chain using a proxy object, much the same way that it interacts with the ChainDB object.

Alternatively, the Chain object could live within the database process itself, giving it first class access to the ChainDB.  This cuts down one layer of interprocess communication which should be a good thing.

So, IIUC the Chain would have direct access to ChainDB, and what is currently the database process would become a chain process? Or would we still need a process to access the ChainDB directly? I think it'd be good if we could have just the chain process accessing ChainDB

After some experimentation yesterday, I think I've changed my stance on this, but it's still unclear what the path forward is.
Complication 1
Consider the following split.

Chain object in process A
Networking in process B
JSON-RPC in process C

Separating Chain and networking keeps the networking processes from getting blocked by the Chain object.  Similarly, JSON-RPC needs it's own process to ensure similar guarantees to keep it from blocking the other processes.
The JSON-RPC server needs access to the Chain object to serve requests.  Naive approach would be to use the multiprocessing Manager pattern to serve it over a socket like we're currently doing with the databases.  This however doesn't work, because it doesn't allow us to serve JSON-RPC requests when we run an LES node because many of the JSON-RPC requests will require us to go out over the networking process to retrieve responses.
Conclusion: Whatever Chain object the JSON-RPC server uses must be able to delegate certain requests to the networking process.  This indicates that having the Chain and networking code combined as they are now might be the right choice.

Complication 2
Assuming that we keep the Chain and networking processes together as they are now, we need a solution for the following two things.

Avoid calls to Chain methods which would block and subsequently time out the networking processes.  Without separating them, I don't know how we can accomplish this beyond self enforced discipline which is error prone.
Expose access to the Chain to the JSON-RPC process.  Current known approach using multiprocessing Manager pattern doesn't appear viable because it will allow heavy request load from the JSON-RPC server to overload the networking process, causing timeouts.

replaced by https://github.com/ethereum/py-evm/issues/501


do not bootstrap with an empty schema

Adds an additional check before bootstrapping. This ensures that a node will not try to bootstrap if its schema is empty.

I wonder if we should have a sort of "TERMINATED_TO_PREVENT_BAD_STATE" or something "Mode", that sidecar/manager can then observe, report, and potential monitor for?
and then in this case instead of looping in the while for forever we bail after like a minute or something.
could then re-use said Mode in other codepaths in StorageService that we don't think our deployment models should ever reach/hit; @zpear and I started to discuss something similar today

The type of mode you're talking about we can do easily already by adding a new non transient error (we already do this for bootstraps that start with data already on disk to avoid corruption).
Separately, I just found https://issues.apache.org/jira/browse/CASSANDRA-15158 which is worth a read


[SPARK-40141][CORE] Remove unnecessary TaskContext addTaskXxxListener overloads

What changes were proposed in this pull request?
TaskContext currently defines two sets of functions for registering listeners:
def addTaskCompletionListener(listener: TaskCompletionListener): TaskContext
def addTaskCompletionListener[U](f: (TaskContext) => U): TaskContext

def addTaskFailureListener(listener: TaskFailureListener): TaskContext
def addTaskFailureListener(f: (TaskContext, Throwable) => Unit): TaskContext

Before JDK8 and scala-2.12, the overloads were a convenient way to register a new listener without having to instantiate a new class. However, with the introduction of functional interfaces in JDK8+, and subsequent SAM support in scala-2.12, the two function signatures are now equivalent because a function whose signature matches the only method of a functional interface can be used in place of that interface.
Result: cryptic ambiguous overload errors when trying to use the function-only overload, which prompted a scala bug report (which was never addressed), as well as an attempted workaround that makes addTaskCompletionListener gratuitously generic, so that the compiler no longer considers e.g. addTaskCompletionListener[Unit] as equivalent to the overload that accepts a TaskFailureListener. The latter workaround was never applied to addTaskFailureListener for some reason.
Now that scala-2.12 on JDK8 is the minimum supported version, we can dispense with the overloads and rely entirely on language SAM support to work as expected. The vast majority of call sites can now use the function form instead of the class form, which simplifies the code considerably.
While we're at it, standardize the call sites. One-liners use the functional form:
addTaskCompletionListener(_ => ...)

while multi-liners use the block form:
addTaskCompletionListener { _ =>
  ...
}

Why are the changes needed?
Scala SAM feature conflicts with the existing overloads. The task listener interface becomes simpler and easier to use if we align with the language.
Does this PR introduce any user-facing change?
Developers who rely on this developer API will need to remove the gratuitous [Unit] when registering functions as listeners.
How was this patch tested?
All use sites in the spark code base have been updated to use the new mechanism. The fact that they continue to compile is the strongest evidence that the change worked. The tests that exercise the changed code sites also verify correctness.

This reminds me about so many pair methods for both Scala and Java friendly... Even there are several such methods in DataFrame. When we migrated to Scala 2.12, we figured out such issue, and our decision was let them leave to not break anything. But I agree it would be nice if we could reconsider removal of one of pair methods which can be taken care by Scala.

Another thing to check is that whether this changes enforce end users to rebuild their app jar or not. If we all consider that end users should rebuild their app jar based on the Spark version then that is OK, but we probably may not want to enforce this in every minor version.

Can one of the admins verify this patch?

Hi, @ryan-johnson-databricks . Apache Spark uses the PR contributor's GitHub Action resources instead of Apache Spark GitHub Action resources. Please enable GitHub Action in your repository. Currently, it seems to be disabled in your repo.

I did enable it -- and it even worked briefly -- but something seems to have gone wrong. I have an open ticket w/ github support but they've not yet responded.

This is developer API so I'm fine with this cleanup. Can you push an empty commit to retrigger the Github Action tests?

For Spark's own internal development, I'm wondering whether this change will introduce source-compatibility concerns in our own patch backports: If I write a bugfix patch using the new lambda syntax then cherry-pick that patch to older branches then I'll run into compile failures. Of course, the option to be compatible exists but a developer might forget to use it (esp. since IDEs are likely to suggest a replacement to the lambda syntax).

I'm not sure that concern is very troublesome?

The task listener code is very slow-moving -- code involving a task listener tends to not change after being added, other than a half dozen refactors that move/indent it. Full analysis below.
In the last four years, 11 new listeners were added by a half dozen code authors. Each case would have required the author to figure out that [Unit] was needed. Meanwhile, there has only been ONE bugfix "backport" since [Unit] was added in 2018 -- and that one backport was for one of the 11 new-code changes, from master back to a recent branch cut, to fix a regression found during release testing. So today the new dev cost of keeping [Unit] is 10x higher than the backport cost of removing it.
it's pretty easy to add [Unit] back in if needed?

(***) By "very slow moving" I mean:

There are no prod uses of addTaskFailureListener in the spark code base today -- which probably explains why nobody realized it needed to become polymorphic before now.
Out of 45 prod files that use addTaskCompletionListener[Unit], 28 have no changes since the Aug 2018 refactor that added [Unit] as part of scala-2.12 effort. Of the 17 remaining files (full list below) only one change was a bug fix that would have needed a backport. That change added a new listener (2022-03-09) and merged to master less than a week before spark-3.3 branch cut (2022-03-15); a perf regression was during release testing and fixed by revert (2022-04-26).

List of recent changes to addTaskCompletionListener[Unit]:

d3d22928d4f in Jun 2022, ArrowConverters.scala (adding a null-check to the task context a listener gets added to, as part of a refactor)
20ffbf7b308 in Apr 2022, DataSourceRDD.scala (refactor-induced indentation change)
6b5a1f9df28 in Apr 2022, ShuffledHashJoinExec.scala (revert a Mar 2022 change to code first added in Aug 2020)
8714eefe6f9 in Aug 2021, Columnar.scala (refactor-induced indentation change)
3257a30e539 in Jun 2021, RocksDB.scala (implement new RocksDB connector)
cc9a158712d in Jun 2021, SortMergeJoinExec.scala (added a new completion listener to update a spill size metric)
f11950f08f6 in Mar 2021, ExternalSorter.scala (refactor that moved existing code to a new location and also reformatted it)
d871b54a4e9 in Jan 2021, ObjectAggregationIterator.scala (added a new completion listener for capturing metrics)
21413b7dd4e in Nov 2020, streaming/state/package.scala (new code)
713124d5e32 in Aug 2020, InMemoryRelation.scala (refactor that moved an existing completion listener)
d7b268ab326 in Dec 2019, CollectMetricsExec.scala (added a new completion listener that collects metrics)
05988b256e8 in Sep 2019, BaseArrowPythonRunner.scala (refactor that moved existing code to a new file, otherwise unchanged)
3663dbe5418 in Jul 2019, AvroPartitionReaderFactory.scala (new code, avro DSv2 support)
23ebd389b5c in Jun 2019, ParquetPartitionReaderFactory.scala (new code, parquet DSv2 support)
d50603a37c4 in Apr 2019, TextPartitionReaderFactory.scala (new code, text DSv2 support)
8126d09fb5b in Feb 2019, ArrowRunner.scala (new code)
1280bfd7564 in Jan 2019, PipedRDD.scala (bug fix that required adding a new completion listener)

Since this is a source level incompatibility (as per @JoshRosen's analysis), and given this has been around for a while as a DeveloperApi, I would mark it as deprecated - and relook at removing it in the next major release.
We can ofcourse cleanup our own use

Since this is a source level incompatibility (as per @JoshRosen's analysis), and given this has been around for a while as a DeveloperApi, I would mark it as deprecated - and relook at removing it in the next major release. We can ofcourse cleanup our own use

We can't "just" mark it as deprecated and also clean up our own call sites, because this is an ambiguous overload. If we mark the polymorphic function overload as deprecated, that just forces everyone to either ignore the warning, or to create an actual listener object until we get around to removing the deprecated overload.
Neither of those seems to provide much benefit?

Yep deprecation doesn't help. The caller can use casts to disambiguate it, but that's ugly. I wouldn't object strongly to remove this before 4.0 as it's a developer API, but by the same token, it's a developer API. Is it worth the binary-compatibility breakage vs just having devs use casts?


CSharp: Make string not nullable when used for date

https://github.com/NJsonSchema/NJsonSchema/commit/b7569b8991af133558f4a6f0488227ffa5ac7cd6


Intel MPI version for HB120rs_V2

Hi,
Could you please provide information on which version of Intel MPI is compatible with HB120rs_V2?
Thanks.
/D

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fbe00d45-6f18-5537-1ccc-2e86927ed114
Version Independent ID: bb5f5c1a-01ec-b61b-db1b-3cbf30a0da0a
Content: Set up Message Passing Interface for HPC - Azure Virtual Machines
Content Source: articles/virtual-machines/workloads/hpc/setup-mpi.md
Service: virtual-machines
GitHub Login: @vermagit
Microsoft Alias: amverma

@dops0 Thank you for your feedback! We will review and provide an update as appropriate.

@dops0
Thanks for the feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.

@dops0
All MPI implementations are supported on the SR-IOV enabled HB, HC, HB_v2 VM sizes, as is Intel MPI.
I suggest Intel MPI 2019.6 as one of the older versions in 2019 had a bug, and is more performant than some older versions (2018).

#please-close


Bug: Forecast takes long time to complete

Detailed Description
Forecaster should take around ~ 1 min to complete the run.
On 24th October it was taking a lot longer.
Note: Database was overloaded, so this has a knock on effect.
Context
Hard to develop on the FE, when database is being very slow
PD: killed ECS tasks on 24th October to reset the system
This task took 20 mins before being killed
The forecast updates the 'forecast_lvalue_latest' but this can take a while. The code that does the updaing is here
Possible Implementation
TODO

Idea: When getting forecast_value_latest object, only get the objects from at most a day a go. This will help reduce the query time.


Put code at the start of the sms text

Both Google and Trello does this. It allows for peeking the text and swipe it away rather than clicking on it and viewing the text. Saves the user a click and a few secs.
Google's: "xxxxxx is your Google confirmation code."
Trello: "xxxxxx is your Trello verification code"

Sure, that seems fine. Thx.


fix: remove update-notifier

It's better to catch outdated dependencies before deployment, with tools such as dependabot, instead of only finding about it when you're about to run the application.
The motivation for having update-notifier isn't strong as per https://github.com/probot/probot/issues/1102#issuecomment-570598406 and it pulls in needless dependencies as showcased in https://github.com/probot/probot/issues/1703, therefore it's better to remove it.
BREAKING CHANGE: update-notifier will no longer be available for the CLI
close #1703

Thanks! I wouldn't consider this a breaking change though, no public APIs are changed

I'll edit the commit message then


Add Kabronero

What changes does this PR do?
Add "Kabronero" to the list.

Thanks! 🎉

HI!
I see that my commit was approved but then when I go to the original repository it seems that my name is not here. Am I doing something wrong?
Thanks so much!

The thing is, your commit has been approved but not merged yet.
Wait till it gets merged and you will see your name in the list!

The thing is, your commit has been approved but not merged yet.
Wait till it gets merged and you will see your name in the list!

Oh, got it. Thanks for your reply!!!


[request] Add bot's territory marker

It would be nice to see on map territory where bot will be walking. Something like in PokemonGo-Map

run this command on root folder: git submodule foreach git pull origin master and you will get this feature.

ask the pokemongo-map people for such feature as they are already doing location to web


Add support for xz compressed metadata

go-apt-mirror is unable to mirror buster-backports as an example.
http://deb.debian.org/debian/dists/buster-backports/main/binary-all/

You're welcome!


Without require: false, rubocop-rspec will cause crash

Hopefully it will save time. Without require: false to this gem in Gemfile, the following crash will occur. Happened to both ruby 2.6.0 and 2.6.2.
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-darwin18]

-- Crash Report log information --------------------------------------------
   See Crash Report log file under the one of following:
     * ~/Library/Logs/DiagnosticReports
     * /Library/Logs/DiagnosticReports
   for more details.
Don't forget to include the above Crash Report log file in bug reports.

-- Control frame information -----------------------------------------------
c:0003 p:---- s:0011 e:000010 CFUNC  :sleep
c:0002 p:0025 s:0006 e:000005 BLOCK  /Users/gbudiman/.rvm/rubies/ruby-2.6.0/lib/ruby/2.6.0/timeout.rb:86 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/Users/gbudiman/.rvm/rubies/ruby-2.6.0/lib/ruby/2.6.0/timeout.rb:86:in `block (2 levels) in timeout'
/Users/gbudiman/.rvm/rubies/ruby-2.6.0/lib/ruby/2.6.0/timeout.rb:86:in `sleep'```

Is this it? Can you provide more context? Is this all of the output? What command are you running?
Can you create a minimal all that allows to reproduce the issue and share it?

Also had this bug. Running rspec tests produce the error reported above. There are cases wherein the bug doesn't occur but only on rare occasions.
I'm running on MacOs.
Process:               ruby [24888]
Path:                  /Users/USER/*/ruby
Identifier:            ruby
Version:               0
Code Type:             X86-64 (Native)
Parent Process:        ruby [6044]
Responsible:           ruby [24888]

Date/Time:             2019-06-10 17:46:18.554 +0800
OS Version:            Mac OS X 10.14.5 (18F132)
System Integrity Protection: enabled
Crashed Thread:        0  Dispatch queue: com.apple.main-thread
Exception Type:        EXC_BAD_ACCESS (SIGABRT)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000104e3ea3a
Exception Note:        EXC_CORPSE_NOTIFY

VM Regions Near 0x104e3ea3a:
    __LINKEDIT<PHONE_NUMBER>e3c000-0000000104e3e000 [    8K] r--/rwx SM=COW  /Users/USER/*/digest.bundle
-->
    MALLOC_LARGE<PHONE_NUMBER>e56000-0000000104e7a000 [  144K] rw-/rwx SM=PRV

Application Specific Information:
crashed on child side of fork pre-exec

Again, could you create a minimal app that allows to reproduce the issue and share it? It’s hard to debug this issue just from an OS error message

Can you provide some reproducible scenario, @gbudiman @sAnti09 ? Something involving ruby version installation and a repository that crashes would do.


Purge newrelic support

Codecov Report

Merging #1018 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1018      +/-   ##
==========================================
- Coverage    90.6%   90.59%   -0.02%
==========================================
  Files          50       50
  Lines        6762     6749      -13
==========================================
- Hits         6127     6114      -13
  Misses        635      635

LGTM