docs: Clarify that trust of X-Fowarded-Proto is also necessary.

Previously, X-Forwarded-Proto did not need to be set, and failure to set loadbalancer.ips would merely result in bad IP-address rate-limiting and incorrect access logs; after 0935d388f053, however, failure to do either of those, if Zulip is deployed with http_only, will lead to infinite redirect loops after login.  These are accompanied by a misleading error, from Tornado, of:
Forbidden (Origin checking failed - https://zulip.example.com does not match any trusted origins.): /json/events

This is most common with Docker deployments, where deployments use another docker container, such as nginx or Traefik, to do SSL termination.  See zulip/docker-zulip#403.
Update the documentation to reinforce that loadbalancer.ips also controls trust of X-Forwarded-Proto, and that failure to set it will cause the application to not function correctly.

Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

Merged, thanks @alexmv! We should backport to 7.x and probably aim to do a 7.2 release somewhat soon so that the documentation update actually takes effect in ReadTheDocs.
(Or start pushing a stable branch to make ReadTheDocs do builds without a release?)

Backported as e921c7dafe15c008989d3d340e10eee8c89d049f.


Add a tag to the current release (CON-344)

I'm attempting to programmatically track some GitHub repositories, including this one.  In order to track releases over time, the repository needs a release version tag.  Would it be possible to add a version tag?

@haxorthematrix we are in process of forking out our first release sometime this week or earlier next week. So, yeah, will release a tag soon.

@haxorthematrix Interim update - We have now created release/v1.0 branch for Matter v1.0. We will create a tag in the coming days. Also, the main branch will soon be updated to support Matter v1.1 (connectedhomeip submodule update)


ZEPPELIN-1817 - Added documentation to Zeppelin's shiro.ini to provide help users configure security

What is this PR for?
Add documentation to Zeppelin's shiro.ini to provide help users configure security
The added documentation walks through different security stages
What type of PR is it?
Documentation
Todos

None

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1817,

How should this be tested?
This pull request add only documentation, to see new doc, see shiro.ini.template
Questions:

Does the licenses files need update?
N
Is there breaking changes for older versions?
N
Does this needs documentation?
N

LGTM!

@vinayshukla Looks good 👍

Thanks @vinayshukla for the contribution.
We have a document about configuring shiro authentication on website.
What do you think keeping information in the single document(website), instead of keeping two different document for the single feature?

Great improvement, thank you @vinayshukla !
But indeed, it would be really nice to have documentation available on the website as well, as @Leemoonsoo suggests.

@Leemoonsoo  The CI failure is unrelated, the 2 files changed are template and markdown. Is there a way to re-trigger the CI?

@vinayshukla I think what @Leemoonsoo suggested was having only one documentation about this info.
And IMO having detailed information to describe about "Authentication in Zeppelin", but it's not good  idea put this all contents under "Overview".

when I set to property zeppelin.anonymous.allowed to true in conf/zeppelin-site.xml  and set cfg in the shiro like this:

I also need  login using username/password

@qinchaofeng If you want to turn off shiro(login as anonymous), you should set shiro.ini like below
/** = anon
#/** = authc

@AhyoungRyu  thinks, and the question is  I set shiro.ini like this:

and set to property zeppelin.anonymous.allowed to true in conf/zeppelin-site.xml
I also need to login using username/password


feat: implement new hook to set addresses

Description
Add support for new endpoints on Katana to set addresses for Solis.

For this endpoint to be available, solis must be run with --dev argument.
Then you can curl like this:
curl -X POST \
     -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","id":"1","method":"katana_setSolisAddresses","params":{"addresses": {"orderbook_arkchain":"0x1234", "executor_starknet":0x8888}}}' \
     http://localhost:7777

Amazing thanks @glihm

i would add a secret from the env, so we can use that outside of dev env with a basic auth ie


dataset._level() combined with ['counts_sum', 'c%_sum'] in views, results in duplicated values in tables


BigQueryException: Failed to create view with recursive CTE

Steps to reproduce
        bigQuery.create(DatasetInfo.of("temp"));
        bigQuery.create(TableInfo.of(TableId.of("temp", "countries"), StandardTableDefinition.of(
            Schema.of(Field.newBuilder("name", StandardSQLTypeName.STRING).build())
        )));
        bigQuery.create(TableInfo.of(TableId.of("temp", "my_view"), ViewDefinition.of(
            "with recursive all_countries as (select * from temp.countries)\n" +
                "select * from all_countries"
        )));

Will lead to an exception
com.google.cloud.bigquery.BigQueryException: failed to create view CREATE VIEW `test-project.temp.my_view` AS (with recursive all_countries as (select * from temp.countries)
select * from all_countries): failed to analyze: INVALID_ARGUMENT: RECURSIVE is not supported in the WITH clause [at 1:45]
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:187)
	at com.google.cloud.bigquery.BigQueryImpl$2.call(BigQueryImpl.java:300)
	at com.google.cloud.bigquery.BigQueryImpl$2.call(BigQueryImpl.java:297)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:296)

But if you execute the same SQL in BigQuery, everything will be successful
CREATE VIEW `temp.my_view` AS (
with recursive all_countries as (select * from temp.countries)
select * from all_countries
);

Recursive CTEs are currently not supported.
https://github.com/goccy/go-zetasqlite#query


correct the swapping of min/max args in drag_int and drag_float

Fix for https://github.com/swistakm/pyimgui/issues/53
Tested on mac os high sierra with python3.6 and SDL2.

Great, looks good. I will also revert the order of max/min params in Python functions in case someone uses the calls without keywords.

Released as 0.1.0


QueryResultTypeWalker: fix nullability checks over unknown type

Separated from https://github.com/phpstan/phpstan-doctrine/pull/506
Without this fix, the new testcase fails to add nullability and the inferred type does not accept the real result (invalid type was inferred). More mixed tests with all possible AST nodes is in the origin PR.

Thank you.


Need to reconfigure database connection configuration after upgrade

What problem is this solving
Need to reconfigure database connection configuration after upgrade
Proposed solution
Not need to reconfigure database connection configuration after upgrade
Describe alternatives you've considered
No response

What do you mean by "reconfigure"?
The settings should be preserved, they are stored in ~/.csdm and the last upgrade didn't change any settings.

After installation/upgrade, it require me to input all db related configuration again
The config shown in UI is incorrect, from port to username to password

Settings were removed during updates.
Unfortunately it should happen again when upgrading from 3.0.1 to 3.0.2 but will not happen again after that.
Thank you!


link fix

There was an error while validating job d9f5120d-4603-46ee-9246-787c85bd62d8 Message:Error clonning repositoryDetails:System.ArgumentException: Error clonning repository ---> System.Exception: Error when trying to initialize shared repository ---> LibGit2Sharp.NotFoundException: Branch 509 not found. Available branches:
at Microsoft.DPS.Services.LibGit2SharpSourceControlProvider.GetBranch(Repository repo, String name)
at Microsoft.DPS.Services.LibGit2SharpSourceControlProvider.<>c__DisplayClass3e.b__3d()
at Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.b__0()
at Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func1 func)    at Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.RetryPolicy.ExecuteAction(Action action)    at Microsoft.DPS.Services.LibGit2SharpSourceControlProvider.InitializeSharedRepository(String url, IDictionary2 branches, String sharedRepoPath)
at Microsoft.DPS.Services.LibGit2SharpSourceControlProvider.VerifySharedRepository(String url, IDictionary2 branches, String tempPath)    --- End of inner exception stack trace ---    at Microsoft.DPS.Services.LibGit2SharpSourceControlProvider.VerifySharedRepository(String url, IDictionary2 branches, String tempPath)
at Microsoft.DPS.Services.LibGit2SharpSourceControlProvider.CloneRepository(String url, IDictionary2 branches)    --- End of inner exception stack trace ---    at Microsoft.DPS.Services.LibGit2SharpSourceControlProvider.CloneRepository(String url, IDictionary2 branches)
at Microsoft.DPS.Services.LibGit2SharpSourceControlProvider.CloneRepository()
at Microsoft.DPS.Services.LibGit2SharpSourceControlProvider.GetTreeAsync(String path, Boolean includeSubDirectories, String excludePath, IList1 extensionWhiteList)    at Microsoft.DPS.Commands.IdentifyDocumentsToPublish.ListSourceControlFilesCommand.Do(IDictionary2 context)
at Microsoft.LightweightJobEngine.Core.Language.JobWorkflow1.RunCommands(Guid jobId, IDictionary2 context, CancellationTokenSource cancelSignal)


dual pipe entity

Works like the pipe entity but you could enter or exit it. simple enough to explain.

This is already possible with the normal pipe entity, just place a pipe and an exit next to each other


Cropping and setting JPG quality

In addition to resizing, cropping and setting JPG quality would be very nice to have for images. Resizing alone is seldom used in real world applications, and if you're cropping and setting quality on photos in your desktop image editor, you may as well taylor the sizes right there and then.

Couldn't agree more; cropping in particular would actually make the resize function really useful!
I found this gist which should make matters easy: https://gist.github.com/maxivak/3924976

I see in that commit quality was taken out of the README (I didn't see any corresponding code when I skimmed it). Will quality be implemented later?

Did cropping get added back in the end?
I've tried a few variations of
{% image test.jpg [resize:100x100^ crop:100x100] %}
Trying to make it essentially crop zoom from any aspect ratio so the image ends up 100x100 with no stretching :)
(like background-size:cover in css)

Nobody can help you without a version, please give me a version.

Ah sorry! I found a workaround in the end actually, having the image resize, then set that as a background-image and use it that way, i.e.:


cli-test-utils: print_stream helper function

Adds a new print_stream function, closing https://github.com/paritytech/substrate/issues/13985
Updates fn common::start_node to return a tokio::process::Child so it can be used with print_stream
Updates fn common::start_node to return the ws_url of the node it spawned, so its stderr doesn't need to be consumed to derive it later
Updates wait_n_finalized_blocks to retry the WS connection a few times before giving up. The new async Child is not always available to connect to WS immediately after spawns

When I run the following on this branch:
cargo test --features try-runtime -p try-runtime-cli -- --nocapture

I get:
   Compiling try-runtime-cli v0.10.0-dev (/Users/bruno/src/temp/delete/substrate/utils/frame/try-runtime/cli)
    Finished test [unoptimized + debuginfo] target(s) in 3m 25s
     Running unittests src/lib.rs (target/debug/deps/try_runtime_cli-4c611e9fb746222d)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/follow_chain.rs (target/debug/deps/follow_chain-46b53fb1de696150)

running 1 test
test tests::follow_chain_works has been running for over 60 seconds
thread 'tests::follow_chain_works' panicked at 'called `Result::unwrap()` on an `Err` value: "Could not create rpc client after 30 seconds"', utils/frame/try-runtime/cli/tests/follow_chain.rs:48:63
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test tests::follow_chain_works ... FAILED

failures:

failures:
    tests::follow_chain_works

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 360.20s

error: test failed, to rerun pass `-p try-runtime-cli --test follow_chain`

I've tried increasing the time limit here:
https://github.com/paritytech/substrate/blob/438e887909f81b270b050b886d703810139e78fc/utils/frame/try-runtime/cli/tests/follow_chain.rs#L34
To 15 * 60 (I have an M1) and that resulted in the same error.

@brunopgalvao which branch are you using?

@brunopgalvao which branch are you using?

liam-cli-test-utils-print-stream

Seems this is not necessary, if you don't specify .stdout(...) when creating the child, it seems to auto direct output to your terminal.
cc @brunopgalvao


region at the start of a normal comment triggers error about endregion being missing

Environment data

Language Server version: v2023.1.30
OS and version: macOS
Python version (& distribution if applicable, e.g. Anaconda): -

Code Snippet

            for diagnostic, candidate in diagnostics:
                # Checking against points is inclusive unlike checking whether region intersects another
                # region which is exclusive (at region end) and we want an inclusive behavior in this case.

Repro Steps

Have a normal comment in the file that starts with a "region" word

Expected behavior
No error about missing "endregion" pair
Actual behavior
Getting error about "endregion" missing
Logs

Not relevant

@debonte dup?

Not a dupe. This is really about multi-line comments that happen to have one line starting with "region".
@rchl, thanks for filing this.

I don't see a way to improve detection of #region comments that would help in this scenario without breaking more common region scenarios and also making Pylance behave significantly differently from other Python editing experiences (ex. PyCharm). For example:

Require some specific formatting (ex. #region: with a colon).
Assume that #region at the start of a line other than the first line in a multi-line comment is not intended to be a region.

I think the best solution is to provide a way to disable the unmatched region diagnostics. This seems especially desirable in Pyright which doesn't support region folding. At least Pylance users have folding support as an incentive to get their #region comments recognized correctly.
@erictraut, any concerns with adding a new DiagnosticRule for this? Maybe disabled by default in Pyright and on by default in Pylance? I think Rich added a mechanism for reportShadowedImports that makes that possible.
@rchl, would you be satisfied with this solution?
Workarounds in the meantime to silence a particular instance of this diagnostic:

Rejustify the text in the comment so "region" isn't at the start of a line.
Add an extra # before the line in question (## region).
#type: ignore, but that seems ridiculous in the midst of a multi-line comment/paragraph.

This doesn't strike me as something that justifies a new diagnostic rule. If we were to add a new diagnostic rule, it should definitely not have a different default because pyright and pylance. We don't have any precedent for that. The defaults are determined by the "none", "basic" or "strict" rule sets.
My suggestion here is to make the parsing a bit more strict, along the lines of what you're suggesting above. Do you see any good reason why we'd want to interpret something as a # region comment if it is followed by something other than whitespace or a colon? For example, I would think that it's unlikely that # region blah blah blah was meant to be interpreted as a region demarkation comment.

It's actually quite common (maybe even the majority of cases) from what I've seen searching through GitHub.

https://github.com/curtisjamesholt/BY-GEN-public/blob/006811b227316145eb0d195e4f1511b5f671dd17/__init__.py?q=%23region+language%3APython#L98
https://github.com/apple/unityplugins/blob/648ce6f5254424ca17759a26f9056cdf7ddcbf2a/build.py?q=%23region+language%3APython#L115
https://github.com/open-mmlab/mmskeleton/blob/b4c076baa9e02e69b5876c49fa7c509866d902c7/deprecated/origin_stgcn_repo/processor/io.py?q=%23region+language%3APython#L95
https://github.com/turanszkij/WickedEngine/blob/30c47c83ccc84bf658c6b24aab149a1ecc81d032/Editor/Linux/Installer/Installer.py?q=%23region+language%3APython#L3

it should definitely not have a different default [between] pyright and pylance. We don't have any precedent for that.

I think this is true for reportShadowedImports? https://github.com/microsoft/pyrx/blob/de4a60e8f212bb665a337632bedb14fde86d3c2c/packages/pylance-internal/src/server.ts#L721

It's actually quite common

Ah, that's unfortunate. Is a colon (or no following text) used in the majority of cases? Would it make sense to say "if you don't use a colon, we will not treat it as a region demarkation"? That would eliminate false positive errors at the expense of forcing some users to add colons if they want region demarkations to work. That sounds preferable to me than adding a switch that we need to maintain and that user need to toggle to eliminate false positive errors in what is otherwise perfectly legal comments that happen to contain the word "region".

I think this is true for reportShadowedImports?

Ah, I forgot about that. I think this is an unfortunate precedent. We receive frequent questions from users about how to get the same behaviors between pyright and pylance. The more we diverge, the harder this becomes. So I think we should maintain a high bar for any divergence. IMO, this doesn't meet that bar.

Don't other languages (like TS or C#) support region demarkation comments? How do they deal with this case? Perhaps there's some precedent we should be following here.

Is a colon (or no following text) used in the majority of cases?

No. Searching on cs.github.com, I'm seeing about 1K with a colon, 1K with just whitespace, and 40+K with text and no colon.

Don't other languages (like TS or C#) support region demarkation comments?

C# doesn't use comments. It uses the #region and #endregion preprocessor directives.
TypeScript uses //#region and //#endregion comments. Looking at various issues related to this, it appears that they considered also supporting //region and //endregion, but they ended up requiring the #. Maybe because they ran into this issue? I didn't find a GitHub comment that explained this decision though.

I'm reading those comments and I'm wondering whether we're missing an elephant in the room. My case was about comment that starts with # region while all the examples that use it for the purpose of region folding that I'm seeing are using #region so without the space after #. Only considering the latter case for region folding should eliminate large amount of false positives, I would think.

@rchl, cs.github.com finds 6k files without a space (#region) and 47k files with a space (# region).
To double check, i browsed through the files that had spaces and the ones I saw all appeared to be defining regions; they weren't comments that happened to contain the word "region" as in your case.

Is there anyway to disable this rule? Quite annoying.

This also triggers on a commented-out line like # region.some_method_on_the_region when there is a variable named region.
At the very least there should be a setting to opt-out from region comment tracking. If you're doing geospatial work with entities called region, you're likely not using region/endregion anyways.

I think at very least the start shouldn't be trimmed here: https://github.com/microsoft/pyright/blob/9a53b9962c763191690e48e92322d853df37ed0d/packages/pyright-internal/src/analyzer/regions.ts#L57. I think having #region having a special meaning is probably not too bad, but having # region having a special meaning is very annoying when "region" is an important noun in your code base.

This issue has been fixed in prerelease version 2024.2.101, which we've just released. You can find the changelog here: CHANGELOG.md


Installing Gekko using Docker:  UI is unable to open a websocket connection

I'm submitting a ...
[x] bug report
[ ] question about the decisions made in the repository
Action taken (what you did)
Installing Git Master Gekko using Docker #
$ docker-compose build
$ docker-compose up -d
Expected result (what you hoped would happen)
gekko instance running on localhost:3000.
Actual result (unexpected outcome)

Disconnected
Something happened to either Gekko or the connection. Please check the terminal where Gekko is running or your network connection.
This message is shown when the UI is unable to open a websocket connection with the Gekko Server.

Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, etc)
Maybe this is something very basic, the web UI loads up in the background but the above error pops up saying the websocket is unable to connect.
Attaching Docker Logs

$ docker logs -f gekko_gekko_1

Serving Gekko UI on http://localhost:3000/

Something went wrong when trying to open your web browser. UI is running on http://localhost:3000/.
  <-- GET /favicon.ico
  --> GET /favicon.ico 404 4ms -
  <-- GET /favicon.ico
  --> GET /favicon.ico 404 1ms -

~/nodejs-projects/gekko/web/vue/dist @ slab [gamerjoe] $ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
78738f209005        gekko_gekko         "/usr/src/app/docker…"   About an hour ago   Up About an hour    <IP_ADDRESS>:3000->3000/tcp   gekko_gekko_1
1aadc1bbc61a        redis:latest        "docker-entrypoint.s…"   About an hour ago   Up About an hour    6379/tcp                 gekko_redis_1

~/nodejs-projects/gekko/web/vue/dist @ slab [gamerjoe] $ docker-compose ps
WARNING: The PORT variable is not set. Defaulting to a blank string.
ERROR: The Compose file './../../../docker-compose.yml' is invalid because:
services.gekko.ports is invalid: Invalid port ":", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]

Any pointers on how to solve this would be helpful. I'm running docker on Mac OS X.

were you able to find a solution ? as i'm having the same issue when trying to deploy my gekko docker image to kubernetes

Don't use localhost prefere real IP and set a real Port, i think is "The PORT variable is not set" set it  to 3000.


Fraud and Scams - There needs to be a distinction between long and short cons

Short and long cons are distinct.
They should probably be distinguished between.
See Hustle (BBC UK)

Thank you very much!
The confidence tricks should have been classified as either short cons or long cons in https://github.com/delight-im/Knowledge/commit/bfb6d8d1e23d32eac58c20030168d60ccce3ce3c so that they are in (hopefully correct) distinct groups now: https://github.com/delight-im/Knowledge/blob/bfb6d8d1e23d32eac58c20030168d60ccce3ce3c/Fraud and Scams.md

+1


Instrumentation fails for the "calendar" module from the standard library

Code:
import atheris

with atheris.instrument_imports():
    import calendar

Output:
INFO: Instrumenting calendar
Traceback (most recent call last):
  File "/home/cepe/repos/croniter-fuzzing/atheris_test.py", line 4, in <module>
    import calendar
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "/home/cepe/venvs/croniter-fuzzing/lib/python3.10/site-packages/atheris/import_hook.py", line 196, in get_code
    return patch_code(code, self._trace_dataflow)
  File "/home/cepe/venvs/croniter-fuzzing/lib/python3.10/site-packages/atheris/instrument_bytecode.py", line 756, in patch_code
    inst.consts[i] = patch_code(inst.consts[i], trace_dataflow, nested=True)
  File "/home/cepe/venvs/croniter-fuzzing/lib/python3.10/site-packages/atheris/instrument_bytecode.py", line 758, in patch_code
    return inst.to_code()
  File "/home/cepe/venvs/croniter-fuzzing/lib/python3.10/site-packages/atheris/instrument_bytecode.py", line 457, in to_code
    self._check_state()
  File "/home/cepe/venvs/croniter-fuzzing/lib/python3.10/site-packages/atheris/instrument_bytecode.py", line 372, in _check_state
    listing[i].check_state()
  File "/home/cepe/venvs/croniter-fuzzing/lib/python3.10/site-packages/atheris/instrument_bytecode.py", line 208, in check_state
    assert jump_arg_bytes(self.arg) == self.reference
AssertionError

Oh, thanks for the heads-up, we'll look into it.

Looks like this is specific to the 3.10 upgrade

I believe I've identified the issue. After instrumenting code, Atheris needs to fix-up instruction offsets. In one code path in adjust(), the offsets are adjusted like this:
          self.reference += size  # type: ignore[operator]
          self.arg += size

Calling self.arg += size in 3.10 is incorrect, because the argument to jump instructions changed from a byte offset to an instruction offset.
The correct code is:
          self.reference += size  # type: ignore[operator]
          self.arg = add_bytes_to_jump_arg(self.arg, size)

This wasn't detected during testing because this particular code path is only triggered in the unlikely event that (1) the argument is an offset for an absolute jump instruction, and (2) the argument was previously <256 but is now >=256.
Here's a minimum reproducer of this issue (instrument the code)
import atheris

@atheris.instrument_func
def _ord2ymd(n):
    n -= 1
    n400, n = divmod(n, 1)
    year = n400 * 400 + 1

    n100, n = divmod(n, 1)

    n4, n = divmod(n, 1)

    n1, n = divmod(n, 1)

    leapyear = n1 == 3 and (n4 != 24 or n100 == 3)
    assert leapyear == baz(year)
    month = (n + 50) >> 5
    preceding = baz[month] + (month > 2 and leapyear)
    if preceding > n:
        month -= 1
        preceding -= bar[month] + (month == 2 and leapyear)
    n -= preceding
    assert 0 <= n < foo()

We'll validate that this fix is indeed correct, and if so cut a release, most likely tomorrow.


Bugifix/issue 12 wrong textures exported

Fixes #12

Coverage decreased (-0.3%) to 50.313% when pulling f28ec6d62246bf2b8f97a016fd3783e53f1678f3 on bugifix/ISSUE-12-wrong-textures-exported into 1b4669074bf7afdf2b54bdfc77f3af05f9d7d3b8 on develop.


New formatter "constraint"

Closes #157

Coverage remained the same at 100.0% when pulling 666ddf4ca697da5cfda29a06b8dee27be46da94c on warpy:fix-157 into f42d0ac4b94614e7e0c8ace427dd9ef3b6f979f6 on ansman:master.


Loading component

Currently it is possible to compose components in a way that we show "loading" text when apollo is querying data, based on the "loading" flag.
What would be great it to have a possibility to specify a second component in the connect method, and when this component is specified it would render that component instead of the original one.
Example:
connect({ mapQueriesToProps, mapStateToProps })(MyComponent, LoadingComponent)

Would you consider this?

@helfer I actually tried that, but I failed as I could not access the loading flag. Do you have any snippet I could use? Thanks.
Currently I solve it in components as you guys say, but it is a lot of repeated boilerplate for 20+ components in my app.

Does something along these lines not work?
function makeLoadingComponent(loadingComponent, component){
  return function({ data, ...args }){
    if(data.loading) {
      return loadingComponent({ data, ...args });
    } else {
      return component({ data, ...args });
    }
  }
}

@helfer looks fine! Thanks!


Add PR AUC array function

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added arrayPrAUC function, which calculates the AUC (Area Under the Curve) for the Precision Recall curve.
Documentation entry for user-facing changes

[x] Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

[ ]  Allow: All Required Checks
[ ]  Allow: Stateless tests
[ ]  Allow: Stateful tests
[ ]  Allow: Integration Tests
[ ]  Allow: Performance tests
[ ]  Allow: All Builds
[ ]  Allow: batch 1, 2 for multi-batch jobs
[ ]  Allow: batch 3, 4, 5, 6 for multi-batch jobs

[ ]  Exclude: Style check
[ ]  Exclude: Fast test
[ ]  Exclude: All with ASAN
[ ]  Exclude: All with TSAN, MSAN, UBSAN, Coverage
[ ]  Exclude: All with aarch64, release, debug

[ ]  Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
[ ]  Exclude: AST fuzzers

[ ]  Do not test
[ ]  Woolen Wolfdog
[ ]  Upload binaries for special builds
[ ]  Disable merge-commit
[ ]  Disable CI cache

The results obtained by the arrayPrAUC were validated comparing with scikit-learn's average_precision_score.

Outputs used on reference file can be checked with the collapsed code here.
  Code:
from sklearn.metrics import average_precision_score

def pr_auc(scores, labels):
  return average_precision_score(labels, scores)

scores_and_labels = [
  [[0.1, 0.4, 0.35, 0.8], [0, 0, 1, 1]],
  [[0.1, 0.4, 0.4, 0.35, 0.8], [0, 0, 1, 1, 1]],
  [[0.1, 0.35, 0.4, 0.8], [1, 0, 1, 0]],
  [[0.1, 0.35, 0.4, 0.4, 0.8], [1, 0, 1, 0, 0]],
  [[0, 3, 5, 6, 7.5, 8], [1, 0, 1, 0, 0, 0]],
  [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [1, 0, 1, 0, 0, 0, 1, 0, 0, 1]],
  [[0, 1, 1, 2, 2, 2, 3, 3, 3, 3], [1, 0, 1, 0, 0, 0, 1, 0, 0, 1]],
  [[1], [1]],
  [[1], [0]],
  [[0], [0]],
  [[0], [1]],
  [[1, 1], [1, 1]],
  [[1, 1], [0, 0]],
  [[1, 1], [0, 1]],
  [[0, 1], [0, 1]],
  [[1, 0], [0, 1]],
  [[0, 0, 1], [0, 1, 1]],
  [[0, 1, 1], [0, 1, 1]],
  [[0, 1, 1], [0, 0, 1]]
]

for scores, labels in scores_and_labels:
  print(str(pr_auc(scores, labels))[:12])

And here's the output:

I also ran the ClickHouse locally and compared the results on a dataset with 3M rows:

PR-AUC (ClickHouse): 0.5503175607143832

PR-AUC (scikit-learn): 0.5503175607143778

The difference is small 5.4e-15, so I think it should be related to multiple float divisions used on the Precision calculation.  For the same dataset, I got a 2e-16 difference between arrayAUC (ClickHouse) and roc_auc_score (scikit-learn).


fix(generic): ensure config proxy doesn't prevent access to built-ins

ISSUES CLOSED: #336

[x] I have read the contribution documentation for this project.
[x] I agree to follow the code of conduct that this project follows, as appropriate.
[x] The changes are appropriately documented (if applicable).
[x] The changes have sufficient test coverage (if applicable).
[x] The testsuite passes successfully on my local machine (if applicable).

Summarize your changes:
Prevent hiding built-ins when proxying config. Previously a check to hasOwnProperty fail (since this is a built-in) and no value would be returned.

It's worth noting returning the real hasOwnProperty will likely not respect the env var stuff you're doing. We could alternatively trap this via getOwnPropertyDescriptor, but I think you'll just hit this again the next time someone attempts to access a built in.

I've added that support as well


Add new project64 function that removes raw fp64 math from layer shaders

@ibgreen , could you specify which tests have been run for this PR in the description?

I have tested with layer-browser and the normal CI test suite.
But I couldn't test the point cloud layer since the example uses METER_OFFSET mode which disables fp64.
At this point I mostly want to float the idea of adding a function of this type. If we think it is a good idea, I think @jianhuang01 should probably be the one to actually complete the PR polishing the function and updating all layers.

@ibgreen Thanks for the PR, definitely a good direction.

Closing this as discussed in meeting, will use #1049 to track/resume this work.

I will move the new functions into a new optional shader module on top of project64 and put that into experimental-layers.
This way mesh layer and other experimental layers can use this while discussion continue on whether this should be part of the core project64 modules.


OCPBUGS-35072: docs: graceful recovery documentation

This PR adds a graceful recovery documentation for the 2 main failure cases in LVMS

Disk Failure of a multi-disk volume group
Node Failure of a multi-node deviceClass

/jira refresh

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 70.38%. Comparing base (1357036) to head (93dd2d8).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #692      +/-   ##
==========================================
- Coverage   70.66%   70.38%   -0.28%
==========================================
  Files          47       47
  Lines        3245     3245
==========================================
- Hits         2293     2284       -9
- Misses        785      790       +5
- Partials      167      171       +4

see 4 files with indirect coverage changes

/lgtm
/hold for further reviews

/lgtm
/unhold


Update deployment-script-bicep.md

Remove statement about deployment scripts only working with public networking

@mumian can you take a look at this one?
cc @subha-sa

@mumian

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


Parse Error : ';'

Hello and thank you for your useful plugin.
it works for me for most of serialized array except some where i got error :
server-1 { name: 'Parse Error', server-1   message: 'Unknown key type \';\' at position 239' }
the serialized array is : a:7:{s:13:"varPertinence";a:7:{i:0;s:8:"CLINIQUE";i:1;s:1:"7";i:2;s:1:"8";i:3;s:1:"9";i:4;s:2:"10";i:5;s:2:"11";i:6;s:2:"12";}s:10:"varSegment";N;s:12:"varSSegment1";N;s:12:"varSSegment2";N;s:7:"varVoie";N;s:4:"ingr";s:15:"levure de bière";s:2:"mc";N;}
and this another array that is working :
a:7:{s:13:"varPertinence";a:1:{i:0;s:2:"13";}s:10:"varSegment";a:1:{i:0;s:2:"25";}s:12:"varSSegment1";N;s:12:"varSSegment2";N;s:7:"varVoie";N;s:4:"ingr";s:10:"Vitamine k";s:2:"mc";N;}
server-1 { varPertinence: [ '13' ],
server-1   varSegment: [ '25' ],
server-1   varSSegment1: null,
server-1   varSSegment2: null,
server-1   varVoie: null,
server-1   ingr: 'Vitamine k',
server-1   mc: null }
could you tell me what wrong please, im trying to understand that but it take me long time and no results.
Thank you so much

i found the probleme it's the uft-8 format i have the letter : "è" i forgot to encode the serialized array.
thank you anyway :)


Grabber not downloading all the images or not accounting for them.

When I try to download from either Safebooru, Danbooru or Gelbooru it never downloads all the files. In the example I provided there should be 107k files but only about 20k are accounted for. All I do to cause this is I grab any tag with significant amounts of pictures (Azur Lane, Kantai Collection, Touhou, Pokemon, etc), order Grabber to download all, then hit the download button. The download finishes before it should and large numbers of pics are unaccounted for.
Expected behavior
I expect when I order a download to see all the pictures accounted for at the end, either as already existing, downloaded, an error, or otherwise accounted for. I have no way to properly tell if all those images were downloaded or not.
Context

Providing your settings.ini file will greatly help maintainers reproduce your problem if it may be caused by your configuration.
----I couldn't do this because the file type is not supported.
If the bug occurred during a download, make sure to provide said download as an .igl file that Grabber can generate by clicking the Save button of the Downloads tab.
---This file type also isn't supported.

System information
Windows 10
Grabber version: 7.10.1
main.log

main.log

@kagami777 I'm running into the same issue. Have you found any solution?


[Test] Add end-to-end tests

Description
This PR adds end-to-end tests to the CI.
Checklist

[x] PR's title starts with a category (e.g. [Bugfix], [Model], [Tutorial], etc)
[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage
[x] Code is well-documented

The docker should be updated accordingly to depict the latest changes for Megatron kernels.

Docker updated. CI passed.

Thanks @chhzh123


viewControllersToPresent not working

This doesn't appear to be working as a way to setup my hierarchy of viewControllers. Adding them as an array does not push them onto the navigation stack.

viewControllersToPresent is used to set the desired flow of view controllers that the BoardingNavigationController will show, you should be able to call setViewControllers(_:,animated:) to pre-populate the navigation stack.


Default description to synopsis

Hackage requires a description for every package.  If description is not specified, we could fallback to synopsis.

:+1:

Hackage apparently no longer requires a description

Unfortunately cabal-install still warns as of version <IP_ADDRESS>.

If Hackage no longer requires that, then I would skip this for now and hope that cabal gets fixed.
From my perspective it is a valid decision not to add a description and I don't want to make it unnecessarily complicated with hpack.

Please raise your voice if you have a different opinion, otherwise I will close this ticket and the corresponding pull request.


drivers: adc: adc_stm32: enable internal voltage reference source

Like other STM32 series the STM32L4x SoCs have an internal voltage
reference source that need to be enabled.
Signed-off-by: Aurelien Jarno<EMAIL_ADDRESS>

If this is all that was missing to support the source, please add the ref_internal setting to the adc_driver_api structure so systems that use the reference know what its voltage is.

Hmm I am not sure that the existing ADC API is able to correctly represent the STM32 ADC internal reference system. From what I understand ref_internal is the reference voltage of the ADC, said differently to know the measure voltage, one has to multiply ref_internal by (adc_value / (1 << resolution)).
On the STM32, the reference voltage (in the Zephyr sense) is the voltage on the VDDA pin, which can take any value as long as it is stable. There is however a stable internal voltage source at typically 1.212V (that's the value on the STM32L4), which is disabled by default. This voltage can be measured on the channel 0, once enabled. More importantly, this voltage source is characterized at manufacturing time and the value stored in ROM. Using the value from the stm32cube headers, the voltage VREFINT_CAL_VREF (typically 3V) is applied to VDDA during manufacturing, the value of channel 0 is read through the ADC and stored in ROM at address VREFINT_CAL_ADDR.
Therefore in order to get the value of ref_internal (in the Zephyr sense), one has to enable the voltage source (which is the purpose of this patch) and read the channel 0. Then we get ref_internal = (VREFINT_CAL_VREF * *(int)VREFINT_CAL_ADDR) / chan0_val.


Make MDB fetch happen before OMDB fetch

Think I found a way to do this, it seems to be from the async function in ajax
https://stackoverflow.com/questions/18366118/when-to-use-async-false-and-async-true-in-ajax-function-in-jquery


Image doesn't exist

I tried to do this command:
az acr import
--name $REGISTRY_NAME
--source mcr.microsoft.com/hello-world:1.1.2
But it couldn't find the image with that tag.
Invalid message NotFound Not Found {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest tagged by "1.1.2" is not found","detail":{"Tag":"1.1.2"}}]}

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a4b6715d-d24c-3000-c5c5-a20eca369a2f
Version Independent ID: a5416d3d-5196-ae30-aaef-10378a7b16d6
Content: Quickstart - Create connected registry using the CLI - Azure Container Registry
Content Source: articles/container-registry/quickstart-connected-registry-cli.md
Service: container-registry
GitHub Login: @toddysm
Microsoft Alias: memladen

Thanks for bringing this to our attention. We are actively investigating and will get back to you soon.

@buzzfrog I tried to reproduce on my side and it is working. Can you please try again ensuring you are using the right tag.
I had another customer who faced the same issue and in that case the tag which was being referred didn’t exist & a different tag specified did the work.
Let me know if you are still seeing the issue and I can further investigate over email by getting more details from you like Subscription Id to check from the backend. Thanks.

To test I use docker pull command:
docker pull mcr.microsoft.com/hello-world:1.1.2
Response
Error response from daemon: manifest for mcr.microsoft.com/hello-world:1.1.2 not found: manifest unknown: manifest tagged by "1.1.2" is not found
docker pull mcr.microsoft.com/hello-world:latest
works

Thank you for your dedication to our documentation.
Unfortunately, we have been unable to review this issue in a timely manner. We sincerely apologize for the delayed response. We are closing this issue and creating an internal work item to incorporate your suggestion as time allows. If you feel that the problem persists, please respond to this issue with additional information.
Please continue to provide feedback about the documentation. We appreciate your contributions to our community.
#please-close


[FLINK-11801] [table-common] Port SqlParserException to flink-table-common

What is the purpose of the change
Port SqlParserException to flink-table-common
SqlParserException is used in FlinkPlannerImpl for sql parsing. We will have another FlinkPlannerImpl in flink-table-planner-blink.
Because SqlParserException is an API level exception, we should move it to flink-table-common to avoid coping it to flink-table-planner-blink.
Brief change log

Port SqlParserException to flink-table-common

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

@flinkbot attention @twalthr

@KurtYoung I would move SqlParserException to flink-table-api-java for now. And only move things to common that are really required across many different Maven modules. As the description of the PR states, it is an API exception.

@twalthr  that makes sense to me. Will move it to flink-table-api-java.


fix: several hooks in the test will be rearranged in the alphabetical order when parsing yaml(json) which will cause the test to be failed.#770

fix: several hooks in the test will be rearranged in the alphabetical order when parsing yaml(json) which will cause the test to be failed.#770

@zlinghuvip123 Thanks. This will also be released in 2.4.0


Add options/API description and delete links

Hello!
It's possible add option/API in Readme adn delete url, because currently original site is dead/stollen?There is nothing just  location.href points to another web site.
Here is options/API from archiveOrg (slightly formated)
Customizing
===========

1. Options
----------

All of the options below are available to customize Owl Carousel.

Variable               Default             Type            Description
--------------------------------------------------------------------------------------------------------------------------------
items                  5                   int             This variable allows you to set the maximum amount of items displayed
                                                           at a time with the widest browser width

itemsDesktop           [1199,4]            array           This allows you to preset the number of slides visible with a
                                                           particular browser width. The format is [x,y] whereby x=browser width
                                                           and y=number of slides displayed. For example [1199,4] means that
                                                           if(window<=1199){ show 4 slides per page} Alternatively use itemsDesktop:
                                                           false to override these settings.

itemsDesktopSmall      [979,3]             array           As above.
itemsTablet            [768,2]             array           As above.
itemsTabletSmall       false               array           As above. Default value is disabled.
itemsMobile            [479,1]             array           As above
itemsCustom            false               array           This allow you to add custom variations of items depending from the width
                                                           If this option is set, itemsDeskop, itemsDesktopSmall, itemsTablet,
                                                           itemsMobile etc. are disabled For better preview, order the arrays
                                                           by screen size, but it's not mandatory Don't forget to include the lowest
                                                           available screen size, otherwise it will take the default one for screens
                                                           lower than lowest available.

                                                           Example:
                                                           [[0, 2], [400, 4], [700, 6], [1000, 8], [1200, 10], [1600, 16]]
                                                           For more information about structure of the internal arrays see itemsDesktop.

singleItem              false              boolean         Display only one item.
itemsScaleUp            false              boolean         Option to not stretch items when it is less than the supplied items.
slideSpeed              200                int             Slide speed in milliseconds
paginationSpeed         800                int             Pagination speed in milliseconds
rewindSpeed             1000               int             Rewind speed in milliseconds
autoPlay                false              int/boolean     Change to any integrer for example autoPlay : 5000 to play every 5 seconds.
                                                           If you set autoPlay: true default speed will be 5 seconds.

stopOnHover             false              boolean         Stop autoplay on mouse hover
navigation              false              boolean         Display "next" and "prev" buttons.
navigationText          ["prev","next"]    array           You can cusomize your own text for navigation. To get empty buttons use
                                                           navigationText : false. Also HTML can be used here

rewindNav               true               boolean         Slide to first item. Use rewindSpeed to change animation speed.
scrollPerPage           false              boolean         Scroll per page not per item. This affect next/prev buttons and mouse/touch dragging.
pagination              true               boolean         Show pagination.
paginationNumbers       false              boolean         Show numbers inside pagination buttons
responsive              true               boolean         You can use Owl Carousel on desktop-only websites too!
                                                           Just change that to "false" to disable resposive capabilities

responsiveRefreshRate   200                int             Check window width changes every 200ms for responsive actions
responsiveBaseWidth     window             jQuery selector Owl Carousel check window for browser width changes. You can use any other jQuery element
                                                           to check width changes for example ".owl-demo". Owl will change only if ".owl-demo" get new width.

baseClass               "owl-carousel"     string          Automaticly added class for base CSS styles. Best not to change it if you don't need to.
theme                   "owl-theme"        string          Default Owl CSS styles for navigation and buttons. Change it to match your own theme
lazyLoad                false              boolean         Delays loading of images. Images outside of viewport won't be loaded before user scrolls to them.
                                                           Great for mobile devices to speed up page loadings. IMG need special markup class="lazyOwl"
                                                           and data-src="your img path". See example.

lazyFollow              true               boolean         When pagination used, it skips loading the images from pages that got skipped.
                                                           It only loads the images that get displayed in viewport. If set to false, all images
                                                           get loaded when pagination used.  Use false to remove that effect.
autoHeight              false              boolean         Add height to owl-wrapper-outer so you can use diffrent heights on slides.
                                                           Use it only for one item per page setting.

jsonPath                false              string          Allows you to load directly from a jSon file. The JSON structure you use needs to match
                                                           the owl JSON structure used here. To use custom JSON structure see jsonSuccess option.

jsonSuccess             false              function        Success callback for $.getJSON build in into carousel. with custom JSON structure here.
dragBeforeAnimFinish    true               boolean         Ignore whether a transition is done or not (only dragging).
mouseDrag               true               boolean         Turn off/on mouse events.
touchDrag               true               boolean         Turn off/on touch events.
addClassActive          false              boolean         Add "active" classes on visible items. Works with any numbers of items on screen.
transitionStyle         false              string          Add CSS3 transition style. Works only with one item on screen.

2. Callbacks
------------

Variable        Default Type        Description

beforeUpdate    false   function    Before responsive update callback
afterUpdate     false   function    After responsive update callback
beforeInit      false   function    Before initialization callback
afterInit       false   function    After initialization callback
beforeMove      false   function    Before move callback
afterMove       false   function    After move callback
afterAction     false   function    After startup, move and update callback
startDragging   false   function    Call this function while dragging.
afterLazyLoad   false   function    Call this function after lazyLoad.

3. Defaults
-----------

Carousel default settings

$("#owl-example").owlCarousel({

    // Most important owl features
    items : 5,
    itemsCustom : false,
    itemsDesktop : [1199,4],
    itemsDesktopSmall : [980,3],
    itemsTablet: [768,2],
    itemsTabletSmall: false,
    itemsMobile : [479,1],
    singleItem : false,
    itemsScaleUp : false,

    //Basic Speeds
    slideSpeed : 200,
    paginationSpeed : 800,
    rewindSpeed : 1000,

    //Autoplay
    autoPlay : false,
    stopOnHover : false,

    // Navigation
    navigation : false,
    navigationText : ["prev","next"],
    rewindNav : true,
    scrollPerPage : false,

    //Pagination
    pagination : true,
    paginationNumbers: false,

    // Responsive
    responsive: true,
    responsiveRefreshRate : 200,
    responsiveBaseWidth: window,

    // CSS Styles
    baseClass : "owl-carousel",
    theme : "owl-theme",

    //Lazy load
    lazyLoad : false,
    lazyFollow : true,
    lazyEffect : "fade",

    //Auto height
    autoHeight : false,

    //JSON
    jsonPath : false,
    jsonSuccess : false,

    //Mouse Events
    dragBeforeAnimFinish : true,
    mouseDrag : true,
    touchDrag : true,

    //Transitions
    transitionStyle : false,

    // Other
    addClassActive : false,

    //Callbacks
    beforeUpdate : false,
    afterUpdate : false,
    beforeInit: false,
    afterInit: false,
    beforeMove: false,
    afterMove: false,
    afterAction: false,
    startDragging : false
    afterLazyLoad : false

})

4. Custom Events
----------------

Owl Carousel handles a few basic events. Mainly use them for custom navigation.

"owl.prev" //Go to previous
"owl.next" //Go to next
"owl.play" //Autoplay, also this event accept autoPlay speed as second parameter
"owl.stop" //Stop
"owl.goTo" //goTo provided item
"owl.jumpTo" //jumpTo provided item. Without slide animation.
5. Owl Data methods

To use Owl Carousel $.data use delegate function.

 This is an unmaintained archival copy of the original code. We plan to move away from Owl Carousel and we have no plans to continue development of the project.


Hovering 'Continue thread' should display a hand cursor

Just a small thing:
'Continue thread   > '   doesn't have a hover state, Chrome and Firefox on Mac.
Clicking on it works, though.

I think this is fixed when I switch to Threaded mode.


createQueryStream loads a big set of data into memory before streaming

The bigQuery.createQueryStream seems to load an entire set of data into memory before the stream starts actually piping data into the next streams.
Environment details

OS: MacOS 12.1
Node.js version: 14.18.1
npm version: 6.14.15
@google-cloud/bigquery version: 5.10.0

Steps to reproduce
Using this test script I can see over 300mb of data is loaded into memory before the stream starts piping to the next streams. And I am only selecting one column, so this is a lot of records in that case.
If I log each entry in the transform stream it also seems to come into batches. It pauses for a while and suddenly starts piping again. This makes me think internally a whole page is loaded into memory and then piped to the readable stream, but this might not be the issue.
const stream = bigQuery
  .dataset("dataset")
  .createQueryStream("SELECT email FROM table");

let checked = false;
const tr = new Transform({
  objectMode: true,
  transform: (chunk, enc, next) => {
    if (!checked) {
      console.dir("START PIPING");
      console.dir(process.memoryUsage());
      console.dir(
        "DIFFERENCE = " +
          (process.memoryUsage().heapUsed - heapUsed) / (1024 * 1024) +
          " MB"
      );
      checked = true;
    }
    next(null, JSON.stringify(chunk) + "\n");
  },
});

const write = fs.createWriteStream("/dev/null");

console.dir("BEFORE");
console.dir(process.memoryUsage());
const { heapUsed } = process.memoryUsage();

tr.pipe(write);
stream.pipe(tr);

@koenvanzuijlen This is the case because createQueryStream() is a streamified version of the query() method, which retrieves pages of results using job.getQueryResults().

@steffnay Is there an option available to stream query results directly without loading a lot of results into memory first?

No, unfortunately, we only have the option of streamifying the job.getQueryResults method.

Is there a way to limit the size of this chunk of data? Because it is causing an OOM exception for us on some tables. Can we use maxResults for that or will it limit the total results?

The actual createQueryStream method doesn’t have this parameter available, as it’s packaging both query creation and use of the streamify API. One option would be to end the stream after a certain amount of data has been emitted and make separate calls of that method until all data has been read. (See an example). For further flexibility, you can also write the logic by hand using various query and results methods with streaming. Feel free to open up a follow up issue if a sample seems helpful! As to incorporating more configurability into the library itself for creating streamified queries, it’ll depend on requests!

Also having the same problem. Loading all the results in memory isn't exactly what one would expect from streaming.


Did you manage to get plausible results?

Hi, so far I am limited in compute, therefore I am waiting to soon get my hands on more powerful GPUs. Right now I only have RTX 4090, which is not enough

Hi, the paper says that the step size of the teacher model is set to 1. I think this is unreasonable. I tried to use ddpm CIFAR10 to conduct ADD experiments. When the teacher model step size is 1, sampling is performed, and the result is a picture of completely random noise.

answered here https://github.com/Stability-AI/generative-models/issues/238

Hi, the paper says that the step size of the teacher model is set to 1. I think this is unreasonable. I tried to use ddpm CIFAR10 to conduct ADD experiments. When the teacher model step size is 1, sampling is performed, and the result is a picture of completely random noise.

We have a working implementation and training code for Latent Adversarial Diffusion Distillation, and achieved good results with multiple base models and resolutions: https://github.com/AMD-AIG-AIMA/AMD-Diffusion-Distillation


Sync with original template

Closes/fixes/resolves issue(s)?

closes #26
closes #27

What was added/changed/fixed?

Docs update
Pre & post bootstrap and startup
Automate release

Related issue(s)? [Optional]
Others [Optional]
Checklist (after created PR)

[x] Added reviewers
[x] Added assignee(s)
[x] Added label(s)
[x] Added to project
[x] Added to milestone

I see that format.sh probably can be removed. However I'll approve and you can choose what you'll do.
Edit: Aslo see that versions.tf is not a think anymore in the template? Should we do something about this as well?

I have removed template_example.
I will remove versions.tf from the root of the repository (I think it was trying to fix https://github.com/fredrikhgrelland/vagrant-hashistack-template/issues/14). I would recommend focusing on the issue first before doing such a try

I see that format.sh probably can be removed. However I'll approve and you can choose what you'll do.
Edit: Aslo see that versions.tf is not a think anymore in the template? Should we do something about this as well?

I have removed template_example.
I will remove versions.tf from the root of the repository (I think it was trying to fix fredrikhgrelland/vagrant-hashistack-template#14). I would recommend focusing on the issue first before doing such a try

Okey :+1: But the format.sh file is in root not in template_example: https://github.com/fredrikhgrelland/terraform-nomad-hive/blob/sync/format.sh


Update harfbuzzjs version and remove manual layout subset enable

Just released a new version of harfbuzzjs to make this happen :) But sorry for being late just for some minutes before your new version bump :/

Pull Request Test Coverage Report for Build 806682217

0 of 0   changed or added relevant lines in 0 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-5.9%) to 94.118%

Files with Coverage Reduction
New Missed Lines
%

index.js
2
94.12%

Totals

Change from base Build 806669503:
-5.9%

Covered Lines:
30

Relevant Lines:
32

💛  - Coveralls

Amazing, thanks! And great work on the harfbuzz side 😍
No worries, version numbers are free.

Released in 1.2.2

Thank you, yes it turned out it needed some finalization before enabling layout subset by default so I'd assume some improvements also has happened there.


Use pydata_google_auth instead of colab for Engine auth

Problems
Using google-colab as the dependency is problematic because:

On jupyter locally we have issues because of exactly google-colab: #3318
it has all sorts of issues with python 3.8
it's heavyweight

Recommendation
I'd suggest using instead pydata_google_auth for auth. It is light weight and works on both colab (installed by default) and jupyter notebooks (we can install it as part of the notebook).
# The Google Cloud Project id to use.
project_id = 'your project' #@param {type:"string"}

def setup_auth():
  """Runs the user through the Google OAuth process """
  try:
     import pydata_google_auth
  except:
     !pip install pydata_google_auth
  import pydata_google_auth
  return pydata_google_auth.get_user_credentials(
    ['https://www.googleapis.com/auth/cloud-platform'],
  )

print("Getting OAuth2 credentials.")
print("Press enter after entering the verification code.")
credentials = setup_auth()
print("Authentication complete.")

try:
    import cirq
except:
    !pip install cirq

print("Cirq is installed.")

import cirq

# Create an Engine object to use, providing the project id and the args
# used for authentication (produced by running the authentication above).
engine = cirq.google.Engine(project_id=project_id, service_args = {'credentials': credentials} )

Caveat
This creates a global var called "credentials" which is an OAuth2 token. This might be undesirable, we can try to hide it and work with the GOOGLE_APPLICATION_CREDENTIALS env var - which is a bit better as the user's code won't have direct access to the creds - it needs to read the creds file which the env var is pointing to.
Originally posted by @balopat in https://github.com/quantumlib/Cirq/pull/3365#r497614032

cc @dstrain115 wdyt?

I think this is fine.  It is a BSD-3 clause license, which I think is okay, but it might be worth checking out to verify that this is an acceptable license.

The problem I have with pydata_google_auth is that it uses PyData for the consent screen which is unfamiliar for the users.

Colab uses Google Cloud SDK which is also different from the Quantum Computing Service. @dstrain115 have we ever thought of creating an oauth consent page for QCS? I guess I can look into that.

Can we just warn the user that they will have to authenticate via pydata?

Thanks for the follow-up idea - I should have captured this - based on our chat on Friday I will do the following:
I will make the colab auth conditioned on colab being detected. Otherwise it will assume "advanced" usage and won't help with the interactive login. I'll make this very clear in the comments in the cells. If the jupyter user is not logged in, they will get DefaultCredentialsError. We can also catch that error within the cell and add some more context and instructions if we feel that it's needed.
Yes we could just warn the user about the pydata consent screen - but I'm not too hot on pydata anymore. It introduces yet another third-party library as dependency and it would require passing the credentials explicitly to cirq.google.Engine.
I'm going to close this issue for now - we can reconsider it in case we/users don't like the new version.


postinstallscript cannot open DISPLAY

I am not entirely sure how postinstalscript is executed by cfncluster. For me postinstall script is complaining that it cannot open DISPLAY.
So I have a VNC server running on master node on :50. This is done through userData.
I can manually log into the master node and do
DISPLAY=:50 xterm
and it works just fine.
But if I use following post install script then I get an error saying cannot open display %s.
Error is not fatal. cfncluster goes to completion and I have proper cluster working.
Any pointers/help is highly appreciated!
#!/bin/bash

. /etc/cfncluster/cfnconfig

case ${cfn_node_type} in
MasterServer)
    echo "I am the master" >> /tmp/master.txt

DISPLAY=:50 xlogo

    ;;
ComputeFleet)
    echo "I am a compute node" >> /tmp/compute.txt
    ;;
*)
    ;;
esac

I made some changes and now I am starting vnc server in post install script itself.
Something very strange is happening. If I look at the cfn-init.log then the logs are stuck at following lines:
2018-07-16 10:26:04,125 [DEBUG] Running command runpostinstall
2018-07-16 10:26:04,125 [DEBUG] No test for command runpostinstall
Master comes up but compute nodes never come up.
Then I log into master node and I manually execute fetch_and_run. No errors are reported .
Then I see cfn-init.log start to show more logs and all compute nodes also comeup. I get DISPLAY working and everything else is working as expected.
Any idea what is happening here?

Can you please tell me how is the successful completion of fetch_and_run reported back so that rest of rest of cfnclsuter can proceed?
Where should I start looking? This is very puzzling. I am doing lots of echo's inside post install script ... all of them are showing up properly including the one in last line. Still cfncluster creation does not proceed further and is stuck. If I run the fetch_and_run manually then it starts to move forward successfully.
Seems to me issues related to whether fetch_and_run is being run from a log-in shell versus run by cfnclsuter (possible in dettached shell ??).

Hi,
the fetch_and_run execute the post-install script which should terminate with an exit code of 0 in order to signal a success back to the fetch_and_run.
Are you sure that your post-install script is terminating?

@lukeseawalker,
Yes, at least I can confirm that by running it manually. There is an explicit exit 0 at the end as well.
Few strange things,
1/ if I execute fetch and run manually then cluster formation proceeds. Fetch and run is pulling exact same script from s3. Nt sure why it gets stuck when started automatically by cfncluster.
2/ If I comment out the vnc server command then also things work fine.
3/ vnc server does get started properly even in usual process when cfn is stuck at postinstall step. All commands after that are also executed.
Does putting a daemon process is background in the script can cause issues?

Hi,
I'd compare the environment between the case "started by cfncluster" vs "started manually".
Is the X server running before starting the vnc server?
I'd try to execute the vnc server command in a subshell ( vncserver [...] ) or even with sh -c "vncserver [...]" to see if there are any difference.
I'd try to start the vnc server as a service (e.g. init.d script) instead using directly the command.

@lukeseawalker , can you tell me parent of the fetch_and_run ... which program calls that script?

It's run from the UserData in the cloudformation init: https://github.com/awslabs/cfncluster/blob/develop/cloudformation/cfncluster.cfn.json#L2798
It's run as root and needs to terminate before the instance is added to the scheduler (and the cloudformation stack goes to CREATE_COMPLETE).


Support WooCommerce

Do you have support for WooCommerce (WordPress)?

this plugin support any website that are using cloudflare proxy

Please
Read the functionalities, do not use only with adding the code (UA)
https://es.wordpress.org/plugins/woocommerce-google-analytics-integration/


Daisy-chaining support

We had a discussion in https://github.com/ev3dev/ev3dev/issues/92 a while back about support for daisy-chaining in ev3dev. The conclusion seems to have been that, if it were implemented, it would be implemented as custom logic in the individual client libraries. I received some questions about daisy-chaining with Python on ev3dev so I figured I'd open this issue to discuss. I don't expect much of anything to happen here short-term as we are in the middle of the updates for new APIs, new micropython, and Stretch.
There are two ways I could see this working. First, as I explained (although not very clearly) in the issue I mentioned above, I think it would be great to have a full client library which implements the protocol LEGO already uses for daisy-chaining via the same interfaces as they use, allowing you to control a stock EV3 from an ev3dev app without any software modifications to the second (or further) EV3s. However, this makes it odd if we have ev3dev-style APIs in the library which aren't supported via daisy-chaining. So, it might make more sense to implement our own generic server and support for one or more communication methods. This latter option would also enable interfacing with non-EV3 ev3dev platforms.
I could see some form of this being based on the new brickd service -- it could serve as the remote control server and our Python library would have to do relatively little work itself. That's something for @dlech to consider (although I might be able to make time to work on an implementation of that over the holidays if it were something we saw as valuable).

I mentioned SSDP in that discussion, I have a python implementation of that working.  It could be used to discover the other EV3s on the network via IP multicast.  Just one small piece of the puzzle

FYI, ev3dev-stretch already has avahi and ssdp daemons that allow autodiscovery.

I am doing a project that involves two daisy chained EV3 bricks. Is there any way to write a program in Python that allows daisy chaining with the currently available ev3dev software? If this is possible I would appreciate to know references to the relevant documentation. Thanks.

@LorenzoBrualla check out https://sites.google.com/site/ev3python/learn_ev3_python/rpyc

Is LEGO’s protocol for EV3 to EV3 communication documented somewhere?

There is an SDK with docs on the official MINDSTORMS website.

A ton of awesome info (and code snippets) here on the UDP/TCP port numbers involved with setting up a socket with the EV3 to send the packets described in the pdf above:
http://ev3directcommands.blogspot.com/2016/01/no-title-specified-page-table-border_94.html
Read some more...this person appears to have implemented most (maybe all) of the protocol and they did it in python and it is open source!!
https://github.com/ChristophGaukel/ev3-python3

Note on the existing library: We have to be careful with it, as it's GPL-licensed... if we want to use that code I believe (?) we can at least sub-license that portion as GPL without it infecting the rest of our stuff, but I'm not a lawyer and haven't looked it up.

In general, I think either approach could work depending on how much implementation and maintenance effort they would turn out to need. Personally, I would really like to be able to support the entirety of our friendly API via daisy-chaining (e.g., maybe have special values you can pass for the port(s) which include remote device information). The main challenges I see with that approach are...

Mapping of LMS2012 opcodes 1 to the operations we support. For example, they see running a single motor as different from running a pair... obviously, you can emulate one with the other, but that might not have clean results -- we'd have to test it. So now perhaps the Move* classes also need to have special logic to handle remote operations. But I digress2.
Getting feedback on operation completion (as you mentioned)

If the control protocol doesn't provide any feedback, the dream of an opaque remote control layer is pretty much a non-starter... our APIs just wouldn't work. In that case. we'd need to have the design discussion for what a new control interface would look like -- I would suggest we seriously consider just recommending that external module itself.

1: It's worth noting that, as far as I can tell from my 5 minutes of research, the wire protocol is just a carrier for sending standard lms2012 opcodes. David has an index of them here as part of lmsasm. So it's pretty easy to see what operations are available and how they work. I haven't looked into response values at all though.
2:

Scope creep (also called requirement creep, or kitchen sink syndrome) in project management refers to changes, continuous or uncontrolled growth in a project’s scope, at any point after the project begins.

I hear you on scope creep...we have the potential to create something that is pain to maintain.  Thinking outloud on some of the various scenarios here:

No.
protocol
notes

(1)
EV3-to-EV3 client
Could say "use the  ChristophGaukel" repo. How useful will this be though? Are ev3dev users really still running the EV3 software on some bricks?

(2)
EV3-to-EV3 server
Would allow the iPad MINDSTORMS app to control an ev3dev device (I think) which would be cool

(3)
ev3dev-to-ev3dev client & server
We could define our own protocol that provides our full API on a remote ev3dev device

(1) The more I think about this scenario the less interesting it seems.  Personally I haven't run the EV3 software since the day I discovered ev3dev :)
(3) might not be a crazy amount of work, you basically have to make the low level places where we interact with the filesystem use the remote filesystem.  Not sure how all of our event-driven code where we wait for a change to a particular file would work but worst case you could drop into loop and use a polling model.  Would be API change be as simple as providing a way to indicate that a motor, sensor, etc is attached to a remote device?

FYI, iOS to ev3dev via Bluetooth is not currently possible for technical reasons (and possibly legal reasons). See https://github.com/ev3dev/ev3dev/issues/500.

Personally, I don't particularly care about scenario (2). If you want to use the official LEGO control software, use the official LEGO OS too.
Regarding (1) or (3), I think we should pick one and support it -- either an ev3dev device controlling another ev3dev device or an ev3dev device controlling a stock EV3. The latter is attractive to me as it requires minimal configuration; personally, I find administering ev3dev devices to be a pain, especially when you have to confgure networking. The stock EV3 software makes it easy to connect a USB cable or pair via Bluetooth and everything "just works" without touching the slave device at all.
I think implementing a custom client/server pair for ev3dev devices is probably technically easier (for example, it would be almost trivial to implement a thin file system wrapper which took attribute read/write operations from a socket and handled them locally -- see my footnote). But that means that you have to flash and configure ev3dev on each device, persuade the various IP networking powers-that-be to play nice, and figure out how to address each device with your particular topology. This is already relatively difficult with only two devices... then what happens when you add another EV3? Or want it connected to the Wi-Fi network? I glossed over the possibility of not using IP, but I doubt that would be much easier.

Footnote on ev3dev-to-ev3dev control: I think it would make a lot of sense to find/implement a remote control server and configure it as an installable daemon. Perhaps even add it to Brickman or ev3dev-config so it's easy to enable. If we did it right, there's no reason the client side would be limited to our Python implementation either -- anyone could write an interface for it. And you could presumably run your Python code on a PC and use the remote control capacity to operate your tethered robot with minimal modification. I'm not aware of anything that exactly fits our use case already (in particular, we need poll events for things like state and latency is a real issue) but perhaps others are. Again, there's some technical investigation that would need to be done here -- e.g., how much latency is there over various transport mechanisms and how does that affect us?

For me (2) would only be interesting if I could use my ipad to control ev3dev and that sounds like a no-go :(
For (1) vs (3) to me (3) is more interesting mainly because I can't come up with a scenario where people are going to use ev3dev on one device and EV3 software on another.
ev3dev setup hurdles...my guess is among usb, bluetooth and wifi that wifi is the most commonly used so maybe we brainstorm some on how to make that easier?  I only ever use wifi...I didn't think it was too crazy but my background is in networking so I am probably not a good reference point on this.  SSDP could be used to discover other ev3dev devices on the same network easily enough.
Of the thin file system wrapper vs remote control server the former sounds much easier and seems like it would solve (3).

For (1) vs (3) to me (3) is more interesting mainly because I can't come up with a scenario where people are going to use ev3dev on one device and EV3 software on another.

I would only install ev3dev on the other devices if necessary. As the user, if I want to have my code run on one device and then have it propagate control commands to the others, why would I care what software is running on the other devices as long as it works? The default OS requires no work to setup, whereas ev3dev does.
Nonetheless, it doesn't seem technically workable to implement support for a stock EV3 slave like we'd want, so the point is moot.

ev3dev setup hurdles...

I agree that Wi-Fi is generally the simplest to set up, but it's worth considering:

Not everywhere has accessible or consistent Wi-Fi

e.g. I'm currently living in on-campus housing for school, and it's much easier to connect via USB rather than dealing with subnet isolation and various security rules on the provided Wi-Fi network.

You must connect on every device inividually and give them unique names

If I were building a robot which had multiple EV3s built together, it would be nice to just string USB cables together and have the data "flow" refardless of whether the individual bricks are conneced to Wi-Fi. But perhaps that's a separate scenario which could be implemented after-the-fact if we could figure out how to consistently do, say, serial via USB or made it easy to set up IP tethering.

So, my vote would be to implement a network server which pretty much just forwarded commands to sysfs and then add the appropriate layer into this library so it can connect to it. @dlech do you have any comments on this that you'd like to get in before we go further with this idea?

What is the advantage of implementing something new vs. using RPyC for ev3dev-to-ev3dev or the existing python mentioned above library for ev3dev-to-official?

RPyC is worth testing -- I've never used it before so I don't know how easy it is to work with or, more importantly, how efficient it is.
I think the existing library is still freely available for anyone who'd like to use it. The interface is completely different from ours, but I don't see a reason it wouldn't work. Personally, my goal with officially-supported daisy chaining would be to provide a uniform interface to make it easy to use multiple EV3-like devices as extensions of a single master.

Nonetheless, it doesn't seem technically workable to implement support for a stock EV3 slave like we'd want, so the point is moot

yeah to do (1) in a way that would allow you to use our current API to control a remote ev3lego (an EV3 running stock LEGO os) just seems like a mountain of work with lots of challenges.
So at this point I think we have tossed (1) and (2) which leaves us with (3)  from an ev3dev device control another ev3dev device but that is doable today via rpyc.  Maybe it is time to just declare that we support daisy chaining via rpyc? We have docs on it here:
https://ev3dev-lang.readthedocs.io/projects/python-ev3dev/en/stable/rpyc.html
We could spend some cycles to make it easier to use:

see if ev3dev can install the python3-rpyc debian package by default
if so, create a service that runs via systemd that runs rpyc_classic.py

That would eliminate 2/3rds of the instructions on that page.

I just noticed https://github.com/ev3dev/ev3dev-lang-python/issues/677 so rpyc is already in ev3dev by default.

Another option here is MQTT. I’m trying to solicit some feedback (on a FB group) on it vs RPyC.

My understanding is that MQTT is a message protocol whereas RPyC is a softtware implementation for remote Python interactions. So the former would require additional implementation work, I think, but an implementation based on it might be more performant.
If we wanted to "properly" support this, we'd have to:

Test it (duh)
Figure out how easy/diffficult it is to configure nettworking. Wi-Fi is naturally easy, but how about direct Bluetooth connection or USB? I suspect those are harder. It would be nice to have a script or Brickman option for it.
We probably shouldn't auto-start the service, because every cycle is precious on these EV3 CPUs. But adding it as an enable-able service via Brickman would make sense.
Ideally, write some tutorials.

To me RPyC feels a little too magical, there is a lot going on behind the scenes. MQTT seems much closer to a standard client/server model so me personally would probably use MQTT. I can see how the choice could easily vary depending on the user and the robot they are building.
How about we:

close this issue as it is related to implementing the EV3g protocol which we are not going to do.
close the issue for implementing an EV3g protocol server
use the RPyC docs issue that David filed to make RPyC as easy to use as possible. I have a service file created that we could install so the user could easily enable it as a service.  Agreed on not enabling this by default given our limited CPU.
open an issue for making MQTT as easy to use as possible. I imagine a lot of this would be documentation.
review networking docs to see if anything can be improved there. This falls under ev3dev though not ev3dev-lang-python.
declare that daisy chaining via python is supported via RPyC or MQTT.

Thoughts?

Most of that seems reasonable to me. However, I think we're conflating the roles of MQTT and RPyC a bit:

MQTT is a generic communication protocol. You put bytes in, bytes come out the other end. It's then up to you to write your own code for doing something interesting with those bytes.
RPyC provides a language-integrated interface which lets you operate on the remote end with calls that appear local. Depending on which features you use, you can either simply initiate a remote connection and import+use ev3dev2 like you would locally (keeping track of which objects are remote and which are local) or you can implement a server module which exposes specific methods and then use those methods. The former should work already as-is, while the latter would mean implementing something like a RemoteEv3devConnection class which lets you open motors, sensors, etc.

RPyC would work as-is, in that you can simply instantiate a LargeMotor, for example, and it will control the remote motor.
So, to me, RPyC is the clearly preferred approach due to its pre-integrated nature for the type of things we want to do, with a few potentially major caveats:

It's probably somewhat easy to get confused about what objects are remote and what objects are local. I haven't yet tried it though.
Speed and performance. There are a lot of potential issues here...

Sequential file writes might be spaced out too much, causing synchronization issues and potentially making it unusable
There might be lots of delay between trying to start an operation and having it actually happen
It might be too much extra Python dynamic "stuff" for the poor EV3 processor to handle

It doesn't support Micropython and such support would be impossible.

That last caveat is perhaps the biggest.
The alternative, which would be using MQTT or another messaging protocol, would avoid many of those issues. The difference is that now there's no pleasant interface in Python and either we'd have to build a new one into the library or each user has to write their own.

I think these are all good points that we should mention in docs. At some point maybe we’ll be familiar enough with both to write a decent “RPyC vs MQTT” doc to help folks figure out which to use.

Closing this ticket as we have decided to not implement the EV3g protocol.


Add timezone to time match

Hi!
I'm an Italian user of your module. It's very useful for football crazy like me :D
The hour shown on the view, for me, it was wrong because here in Italy we have a different timezone.
To resolve this "problem", I add in your module a small workaround to set and show the right timezone.
Here's the code:
config.js
config: {
               timeZone: "Europe/Rome",

MMM-SoccerLiveScore.js
Line: 17

timeZone: 'Europe/Rome',

Line: 204

time.innerHTML = moment(activeLeagueStandings[i].time * 1000).tz(this.config.timeZone).format('DD/MM - HH:mm');

I hope that it'll be useful for other people.
Thanks for this module.
Marco.

Hi marco,
thanks for the info. Just make a push request and I'll merge it.
Best
Luke


Feature/civ 3388

Before creating a pull request make sure that:

[X] commit messages are meaningful and follow good commit message guidelines
[X] README and other documentation has been updated / added (if needed)
[X] tests have been updated / new tests has been added (if needed)

Please remove this line and everything above and fill the following sections:
JIRA link (if applicable)
https://tools.hmcts.net/jira/browse/CIV-3388
Related PR
https://github.com/hmcts/civil-general-apps-ccd-definition/pull/265
Change description
Update caseData with new fields for event details and additional information (events "Refer to judge" and "Refer to Legal Advisor")
Create handler for user events Refer to judge" and "Refer to Legal Advisor"
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[X] No

Summary page

Smoke Test

API

Functional multiparty

Functional 1v1


crash after upgrading to macOS 12.7.5 (21H1222)

Describe the bug
crash when i trying to start it after upgrade OS to 12.7.5 (21H1222)
To Reproduce
Click orbstack icon or through shell command orb
Expected behavior
do not crash
Diagnostic report (REQUIRED)
OrbStack info:
Version: 1.6.0
Commit: 73817253cc0680b865fe411b21e187325a28d758 (v1.6.0)
System info:
macOS: 12.7.5 (21H1222)
CPU: amd64, 8 cores
CPU model: Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
Model: MacBookPro11,4
Memory: 16 GiB
Full report: https://orbstack.dev/_admin/diag/orbstack-diagreport_2024-05-25T09-29-41.756694Z.zip

Screenshots and additional context (optional)
No response

Intel CPUs older than Skylake are no longer officially supported, but this will likely work in the next version.


[add] OAuth2 IdP

Coverage decreased (-2.9%) to 81.76% when pulling 379afd48b4f84621f7a65f52346b6ac0b70d9c0c on pr-oauth2-idp into 4d33929fe676454bf3204c52360ff048b8f8b211 on master.


Authentication now failing

This was working fine for the past few days. At 5am today, authentication started failing:
(1586491200197 Fri, 10 Apr 2020 04:00:00 GMT
Already logged in
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-10?slotGroup=1 [Apr 10 - 16]
No slots
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-17?slotGroup=1 [Apr 17 - 23]
No slots
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-24?slotGroup=1 [Apr 24 - 30]
No slots
Logging in with new user session
error: Auth failed. Please check details are correct in config.ini
1586491500196 Fri, 10 Apr 2020 04:05:00 GMT
Logging in with new user session
error: Auth failed. Please check details are correct in config.ini
1586491800150 Fri, 10 Apr 2020 04:10:00 GMT)

I can log in manually ok, so are they now detecting headless connections and rejecting them?

I haven't been running delivery-slot-bot for the last few days, but for me it's logging in fine at the moment.

It's still working for me too...
1586514001881 Fri, 10 Apr 2020 10:20:01 GMT
Logging in with new user session
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-10?slotGroup=1 [Apr 10 - 16]
No slots
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-17?slotGroup=1 [Apr 17 - 23]
No slots
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-24?slotGroup=1 [Apr 24 - 30]
No slots
1586514302074 Fri, 10 Apr 2020 10:25:02 GMT
Logging in with new user session
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-10?slotGroup=1 [Apr 10 - 16]
No slots
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-17?slotGroup=1 [Apr 17 - 23]
No slots
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-24?slotGroup=1 [Apr 24 - 30]
No slots

I was thinking we should perhaps add a random 0-60 second delay before the script runs so we don't all hit exactly at the same time (not that there is a huge number of people using it at the moment).
You could try resetting your password to see if that helps?

'*/6 * * * *' for me, then!

For me it was working fine until 9:35pm (GMT+1) last night. Then stopped working...
$ node ./src/index.js
1586464202176 Thu, 09 Apr 2020 20:30:02 GMT
Logging in with new user session
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-09?slotGroup=1 [Apr 09 - 15]
No slots
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-16?slotGroup=1 [Apr 16 - 22]
No slots
Opening https://www.tesco.com/groceries/en-GB/slots/delivery/2020-04-23?slotGroup=1 [Apr 23 - 29]
No slots
Done in 14.48s.
yarn run v1.22.4
$ node ./src/index.js
1586464501923 Thu, 09 Apr 2020 20:35:01 GMT
Logging in with new user session
error: Auth failed. Please check details are correct in config.ini
Done in 2.42s.
yarn run v1.22.4

I have modified the code to take a screenshot after login and it looks like they are getting wise to us scraping.

Any ideas of how to work around this?

If I change the useragent, it works again.
Modify tesco.js with:
70a71
>     await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36');

Thanks! That seems to have fixed it for now :thumbsup:

If you're not already using it, I'd suggest using the internal cron scheduler, (via yarn start cron, or via docker-compose) as it doesn't log in every run.

I've had a look at the default user agent, and can see a reason to change it

I'll close this issue as these changes seem to have fixed it.


Feature/implement flask admin

fixes #134

@aaron-suarez this PR contains the admin routes and the access control but some UI related things like - login page UI, logout page routing is remaining.

@fumblehool Thanks so much for your contribution! I haven't pulled it down to run it just yet, but I'd like to give it a really thorough review because it changes the models.
I think that this is also going to require a migration file to update the database, right? Can we get that added and the conflicts resolved and then I'll give it a really good review? Thanks!

@aaron-suarez  I've pushed the migrations + fixed the lint issues.
Please have a look.

@aaron-suarez have you tried it locally?

So locally, I got it running in the flask shell, but I'm a bit unclear as to the interface for creating a new user and assigning roles. Would you mind commenting with an example of how to add an admin user so I can do that locally and ensure that everything works as expected?
I'm sure it's in the Flask-Login docs somewhere, feel free to just link to the relevant page in the docs if that's easier.
Thanks again for this contribution, it's looking good so far, just gotta go through the flow to make sure everything works 😄

Also, please rebase to resolve conflicts

I ran it and navigated to /login and I got this error
<IP_ADDRESS> - - [29/Oct/2020 19:03:30] "GET /login HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1945, in full_dispatch_request
    self.try_trigger_before_first_request_functions()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1993, in try_trigger_before_first_request_functions
    func()
  File "/src/run.py", line 47, in before_first_request
    user_datastore.create_user(admin_email, password=encrypted_password)
TypeError: create_user() takes 1 positional argument but 2 were given

Let me know when you're ready for another review (because I assume you're still trying to work this out). If you need help, feel free to hit me up on slack.

@aaron-suarez I've fixed the issues -

I missed the positional params for user_creation
Fixed a bug in password hashing, on every run a different secret_key and `hash was used.

In order for the tests to pass, we'll probably need a default 'SECRET_KEY' and 'SECURITY_PASSWORD_SALT'

Other than that, it looks like it works at a first pass. I'll do some more extensive testing later, but I really appreciate the effort here!

@aaron-suarez I've added default values for secret_key and security_hash. It should work now.
Once the tests are updated, we can change/remove the default values for security reasons.

@aaron-suarez I've resolved the comment. Please review, Thanks :smile:


Misleading convert error when no concrete functions are given

System information

OS Platform and Distribution (e.g., Linux Ubuntu 18.04):
TensorFlow installed from source:
TensorFlow version 2.1.0:

Command used to run the converter or code if you’re using the Python API
If possible, please share a link to Colab/Jupyter/any notebook.
converter = tf.lite.TFLiteConverter.from_saved_model(model_path)
tflite_model = converter.convert()

The output from the converter invocation
ValueError: This converter can only convert a single ConcreteFunction. Converting multiple functions is under development.

Failure details
When attempting to convert a model in which no ConcreteFunctions have been defined, the error message implies that there are multiple.
As someone new coming to TF and TFLite, I found this very confusing as to where my concrete functions were being defined.
Inspecting the code from https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/lite/python/lite.py in the convert method at line 417 the Value Error is thrown when anything but 1 concrete function is defined in the model (rightly so) but the error message implies that more than one have been defined.
if len(self._funcs) != 1:
      raise ValueError("This converter can only convert a single "
                       "ConcreteFunction. Converting multiple functions is "
                       "under development.")

@PRDrum5
Can you please share the simple standalone code to reproduce the issue in our environment. It helps us in localizing the issue faster. Thanks!

Sure
Here I've made a simple model but not defined the input_signature for the tf.function for the __call__ of the model. In doing so there is no  concrete functions at all. But the error message which is returned implies that there are multiple and that this is the issue.
My suggestion is that the error thrown should be clear that in this scinario the user has not defined any concrete functions rather that suggesting that there exists more than one which is not the case.
import tensorflow as tf
from tensorflow.keras.layers import Dense

# Define very simple classification model
class Model(tf.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.d1 = Dense(2, activation='relu')
        self.d2 = Dense(2, activation='softmax')

    @tf.function
    def __call__(self, x):
        print("Tracing the model")
        x = self.d1(x)
        return self.d2(x)

model = Model()

example_data = tf.constant([[1.0, 2.0]])
preds = model(example_data)
tf.print(preds)

# Save the model
tf.saved_model.save(model, './model_example')

# Load the saved model and convert to TFLite
converter = tf.lite.TFLiteConverter.from_saved_model('./model_example')
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)

I have tried on colab with TF version 2.1.0,2.2.0-dev20200227  and was able to reproduce the issue.Please, find the gist here. Thanks!

@gargn for this case is it possible to provide the signature w/ the from_saved_model method? Or do they have to load the model and then explicitly call from_concrete_functions?


remove broken link from rationale text

(issue #3793)

@felipesanches you may to cherry-pick a01a8bdae80aab1b467609f2386b19bedba28ca6 to get unblocked.


 Happy to be educated here if I'm mistaken or to reword.

I think you've hit on a bit of a deeper problem within the documentation, which is the difference between resolving and committing offsets. The latter is a Kafka primitive users see in various kafka implementations and behave like advertised: it only controls from where processing starts, it doesn't advance processing. The former, resolving, is a more complicated term, as it controls both the consumed offset pointer, but also influences the committed offset.
@tulios I recall us discussing this a while back, wondering whether checkpoint wouldn't make for a less ambiguous alternative to resolve.

Yes, checkpoint is definitely a better term. We can add a new section describing more in depth what resolve is and how it works.