Show partial shortcut while editing.

And automatically accept/validate shortcuts when valid.

When receiving a modifier only event, use the event Modifiers to show the current partial shortcuts.

we need to handle keyrelease events otherwise the following sequence is incorrect.

Action          –  Shortcut shown
---------------------------------
Press Ctrl      –  Ctrl
Also Press Alt  –  Ctrl-Alt
Release Ctrl    –  Ctrl-Alt

Though now that we handle release, the user cant press a shortcut and then validate with enter as as soon as the user release the shortcut the QLineEdit will become empty.
Thus validate as soon a shortcut is valid, and contain a non-modifier key.
This in turn make it impossible to assign a modifiers only keybinding, I assume this is quite rare, but we can use on of the keys that are invalid as shortcut to validate modifiers only shortcuts. I choose Qt.Return, aka Enter.
I'm not quite confident on the Return/Enter, and the way to assign modifiers only shortcuts.
There is also this whole Meta/Alt/Ctrl are not the same on MacOS. I did not extract the logic into naprai/utils/interactions as those are Qt Specific, moreover the values of QtEvent.Modifiers are not the same than Qt.Keys.
You will note that there is one weirdness in the order of modifiers when typing these, but I believe this will be fixed upstream.
Closes #6047
Type of change

[X] New feature (non-breaking change which adds functionality)

How has this been tested?
This needs to be tested on more platforms.

In this video:

playing with modifiers to show they update.
Assinignig Ctrl-Shift-Meta-Cmd-T as a modifier.
Then using Alt-Cmd-Return to assign Alt-Cmd modifiers only shortcut.

https://github.com/napari/napari/assets/335567/77e8940c-a74a-4522-94d3-627991e7c005

@jni I think you were asking for this also, and were excited about shortcuts editor showing modifiers.

Note that with https://github.com/napari/docs/pull/200 we should be able to drop modifiers only shortcuts.

This looks great, but in your example assigning alt-meta doesn't actually toggle theme?
Personally, I don't think that should be valid—modifiers should modify something: a key or a mouse action—so it's not a big deal, but still a bit odd.

This looks great, but in your example assigning alt-meta doesn't actually toggle theme?
Personally, I don't think that should be valid—modifiers should modify something: a key or a mouse action—so it's not a big deal, but still a bit odd.

I completely agree, but this pull request is not to discuss whether modifiers only keybindings are good to have. I agree they are not and nap7 also says so. It's here just to fix the bug that modifier.
In the default shortcuts there is at least one "Reset Scroll" that is just a modifier (Ctrl), so my goal is just to let people enter shortcuts that are already assigned, and thus contain only modifiers.

I understand. But being able to assign alt-meta but then it not working is a bit strange
main doesn't allow alt-meta, just alt or meta, but either actually work to toggle the theme.

Was there any actual decisions ? I'm happy to remove the code that allows to bind only modifiers.

Very wierd, I still can't get alt-meta I can't get to work on my mac.
But control-shift does work—it's just touchy on the timing, because it's basically Control as the modifier, shift as the keybind.
Anyhow, single/solo modifiers have been around since forevers, so unless the NAP kills them, they should definitely remain IMO. The double modifiers-only that are now possible are new—0.4.18 doesn't let them work—and a bit weird, but do seem to mostly work?
Still I'd consider preventing them, because they arn't currently possible which would leave this as a UI change PR only.

I remove the explicit handling and messaging around Return and Modifiers only.

So I'm confused by the current implementation.
While the info text at the bottom no longer mentions confirming via return, double modifiers like alt-meta on macOS can still be bound (but don't actually work) using return.
Likewise single modifiers can still be bound using return—these have been around for a long time in napari and include some defaults, so barring a NAP I wouldn't remove—and they do work.

rebased on master also as there was some weird things going on with Ctrl/Cmd on mac that would sometime display improperly. After rebase this seem fine.

thanks !


update cloud.google.com/go and google.golang.org/api

go.mod referes cloud.google.com/go v0.36.0, but it refers
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
and git.apache.org is unreachable (see https://github.com/golang/go/issues/34016)
please update cloud.google.com/go version.

also google.golang.org/api v0.1.0 ?
<EMAIL_ADDRESS><EMAIL_ADDRESS>

I'm sorry, I don't fully understand this question. Are you running into an error? If so, could you post the full error log?

@jadekler, the turndown of git.apache.org broke some builds for users who don't already have the dependency cached and aren't using a module mirror.
The bad dependency has already been removed from cloud.google.com/go and google.golang.org/api (by @odeke-em in https://code-review.googlesource.com/c/gocloud/+/40311, and by you in https://code-review.googlesource.com/c/google-api-go-client/+/40330 respectively).
@ukai, it should suffice for you to upgrade your dependencies on cloud.google.com/go and google.golang.org/api to the latest releases.

@bcmills it's not for google-api-go-client, but google-cloud-go-testing
https://github.com/googleapis/google-cloud-go-testing/blob/master/go.mod
module github.com/googleapis/google-cloud-go-testing

require (
	cloud.google.com/go v0.36.0
	golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f
	golang.org/x/tools v0.0.0-20190305160728-f8c04913dfb7
	google.golang.org/api v0.1.0
	honnef.co/go/tools v0.0.0-20190215041234-466a0476246c
)

https://github.com/googleapis/google-cloud-go-testing/pull/27

thanks! #27 fixes this.


More tests

Explanation
Added further database unit test to validate structure has not changed in code.
Rationale
Test driven development is important. 👍
Tests

@yevgenybulochnik & @jrlegrand - can this please be reviewed and merged if appropriate.  Thanks.


Unnecessary rerenders cause poor performance

Our application has many droppable and draggable components displayed at once and is suffering from poor performance when dragging. When the dragged element crosses over a droppable, it causes all of our draggable and droppable components to rerender for seemingly no reason.
Here's a code sandbox that demonstrates the issue. Each Draggable/Droppable displays the number of times it has rendered. You'll notice that when dragging one of the red boxes, the render count of the other Draggable and Droppable components increases even though their props have not changed.
Code Sandbox
The performance of the code sandbox is actually fine, but the unnecessary rerenders cause performance issues in our app because the draggables are expensive to rerender (something we could probably fix).
I'm not sure if this is expected behavior or a bug. If it does seem to be a fixable problem I would be happy to attempt a PR if you provide some direction.
P.S. Great library!

Nice Project!
This is caused by the react context mechanism
https://github.com/clauderic/dnd-kit/blob/master/packages/core/src/components/DndContext/DndContext.tsx#L629
I know there are several ways to avoid rerender

Further subdivide the context, honestly it is difficult
Use unstable API readContext or useContextSelector, avoid rerender through changeBits
3.Like redux, use context as an unchanging subpub center,
Use forceUpdate to notify the component update

Great response! I was not able to find anything readContext. useContextSelector feels like the cleanest option to me. For reference, useContextSelector has a userland implementation and and an experimental PR in the React repository.
@clauderic, curious what your thoughts are on this. Would you accept a PR that added a dependency on the use-context-selector npm package? Only 1 kb minified and gzipped
The other option would be to keep dndkit as it is and rely on users to optimize their draggable and droppable components. Then when useContextSelector is (hopefully) added to React, we could update dndkit to use it.

@srmagura
You can google react + changeBits + readContext
const currentDispatcher = React.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED.ReactCurrentDispatcher;

function readContext(Context, observedBits) {
  var dispatcher = currentDispatcher.current;
  if (dispatcher === null) {
    throw new Error('react-cache: read and preload may only be called from within a ' + "component's render. They are not supported in event handlers or " + 'lifecycle methods.');
  }
  return dispatcher.readContext(Context, observedBits);
}

Just to set the record straight on this issue, I would like to implement a PR changing the library to use use-context-selector but I want approval from @clauderic or another maintainer before beginning the work. Thanks.

@srmagura my preference would be to wait for React to officially support this feature. I'm open to changes that don't require bringing in additional dependencies.
We use @dnd-kit on some fairly large and complex applications at Shopify and have not run into any significant performance issues when the components that use useDraggable, useDroppable or useSortable and their child components are properly memoized.
You'd have to make a fairly strong case that bringing in this external dependency would drastically improve performance in ways that could not otherwise be achieved to convince me that this is worth doing now rather than waiting for first-class support for context selectors in React.

Sounds good. I am pretty sure our issue can be addressed with proper memoization.
Up to you if you want to keep this issue open for when React officially supports useContextSelector or if you want to close it.

@clauderic one thing we were noticed while using with react-window was the moment we used the useSortable hook in our memoized row component, it would re-render on scroll while not horribly expensive it was causing all of the rows to re-render which upon removing the call to useSortable that behavior ended.  We are still in search of a solution.

This should mostly be resolved once https://github.com/clauderic/dnd-kit/pull/569 is shipped. It won't entirely prevent re-renders, but it will reduce un-necessary re-renders to the strict minimum.

#569

How did memoization work out for you? I've run into this issue and looking for options.

Memoization solved the problem for me. My code looks something like this:
const DraggablePresentation = React.memo(props => {
    // render the visible UI elements here
})

function Draggable(props) {
    const { setNodeRef, attributes, listeners } = useDraggable(...)

    return <div ref={setNodeRef} {...attributes} {...listeners}>
        <DraggablePresentation />
    </div>
}

Thanks @srmagura
I tried something similar with my code and memo, but it's not working as expected. Still getting hundreds of renders. Will look closer at your code.

Got it working nicely. Thanks again @srmagura

this makes you need additional care to make things works properly. As react hooks meant to be used as is, needing to separate component into two separate components just because we need to add this dnd seems ridiculous.
forking our own version seems the only solution for now.

Memoization only solve issues related to use cases that Draggable Component has complex UI to render. the main problem comes when we have many draggable components and by dragging one of them, all draggable items will rerender because useDraggable make them to rerender. i wish it was a mechanism to make useDraggable hook to returns only some part of the state. for example
  const {
    attributes,
    listeners,
    setNodeRef,
    transform,
    state: { isDragging },
  } = useDraggable({
    id,
    data,
    pick: state => ({
      isDragging: state.isDragging,
    }),
  });

for example in my use case i have no need for "over", "node" or "active" so make UI rerenders because of this changes is redundant

Thanks @srmagura @clauderic Memoization only solve issues related to use cases that Draggable Component has complex UI to render. the main problem comes when we have many draggable components and by dragging one of them, all draggable items will rerender because useDraggable make them to rerender. i wish it was a mechanism to make useDraggable hook to returns only some part of the state. for example
  const {
    attributes,
    listeners,
    setNodeRef,
    transform,
    state: { isDragging },
  } = useDraggable({
    id,
    data,
    pick: state => ({
      isDragging: state.isDragging,
    }),
  });

for example in my use case i have no need for "over", "node" or "active" so make UI rerenders because of this changes is redundant

I agree with this.

@srmagura Can you please do us a favor and send your code here? So we can get a a proper idea about your structure.

Having the same issue. When render more than 2500/3000 items, drag & drop behavior are slow (5s delay aprox.). I memoized item component, but delay still happens.
@srmagura did you try to implement use-context-selector library to reduce re-renders caused by context changes?
I think it would be the best option. If you have been working in any branch I can give you a hand!


-bash: standard: command not found

The alias to use standard on the command line doesn't work. I did an npm install standard --save-dev. Not exactly sure how npm does command aliasing or adding stuff to PATH.
What version of standard?
12.0.1
What operating system, Node.js, and npm version?

macOS 10.14.4
Node.js v8.11.3
npm 6.8.0

What did you expect to happen?
$ standard --help
standard - Use JavaScript Standard Style (https://standardjs.com)

Usage:
# etc

What actually happened?
$ standard --help
-bash: standard: command not found

What works for me
$ node_modules/standard/bin/cmd.js --help
standard - Use JavaScript Standard Style (https://standardjs.com)

Usage:
# etc

You need to run it by using npx and do npx standard --help or install it globally by doing npm install -g standard and then run standard --help.

@kevinrambaud makes complete sense. Thanks so much. Should this be added to the documentation?

Feel free to open a pull request if you think it could help people to get started more easily. The Usage section seems to be the appropriate place to do so.

I opened a pulled request, but it hasn't been reviewed and merged yet

@aaronhong Thanks for the PR. It is reviewed and merged now :)


Export elements data

Unwinds query and data results for each element, including elements within dialogs.
NB: The datasource id & element id need to match for this to happen.

@morsh as part of this PR do you want me to update the preconfigured template files to make sure they work with the export feature? Only some of the id names need to be changed.


GAPIC test breakage from HEAD

Today I ran the integration tests for GAPIC microgenerator (for Python client) and our HEAD (master branch). I found it generates several errors. We may not want to pre-release a version that breaks them. I will work on the fix ASAP. This is a release blocker, CC @jtattermusch .

}
kwargs = {}, marshal = <proto.marshal.marshal.Marshal object at 0x7fc2e4f40ac8>, key = 'success', value = {'content': 'The hail in Wales falls mainly on the snails...eventually.'}
pb_type = <class 'WaitResponse'>, pb_value = content: "The hail in Wales falls mainly on the snails...eventually."

    def __init__(self, mapping=None, **kwargs):
        # We accept several things for `mapping`:
        #   * An instance of this class.
        #   * An instance of the underlying protobuf descriptor class.
        #   * A dict
        #   * Nothing (keyword arguments only).
        #
        # Sanity check: Did we get something not on that list? Error if so.
        if mapping and not isinstance(
            mapping, (collections.abc.Mapping, type(self), self._meta.pb)
        ):
            raise TypeError(
                "Invalid constructor input for %s: %r"
                % (self.__class__.__name__, mapping,)
            )

        # Handle the first two cases: they both involve keeping
        # a copy of the underlying protobuf descriptor instance.
        if isinstance(mapping, type(self)):
            mapping = mapping._pb
        if isinstance(mapping, self._meta.pb):
            # Make a copy of the mapping.
            # This is a constructor for a new object, so users will assume
            # that it will not have side effects on the arguments being
            # passed in.
            #
            # The `__wrap_original` argument is private API to override
            # this behavior, because `MessageRule` actually does want to
            # wrap the original argument it was given.
}
kwargs = {}, marshal = <proto.marshal.marshal.Marshal object at 0x7fc2e4f40ac8>, key = 'success', value = {'content': 'The hail in Wales falls mainly on the snails...eventually.'}
pb_type = <class 'WaitResponse'>, pb_value = content: "The hail in Wales falls mainly on the snails...eventually."

    def __init__(self, mapping=None, **kwargs):
        # We accept several things for `mapping`:
        #   * An instance of this class.
        #   * An instance of the underlying protobuf descriptor class.
        #   * A dict
        #   * Nothing (keyword arguments only).
        #
        # Sanity check: Did we get something not on that list? Error if so.
        if mapping and not isinstance(
            mapping, (collections.abc.Mapping, type(self), self._meta.pb)
        ):
            raise TypeError(
                "Invalid constructor input for %s: %r"
                % (self.__class__.__name__, mapping,)
            )

        # Handle the first two cases: they both involve keeping
        # a copy of the underlying protobuf descriptor instance.
        if isinstance(mapping, type(self)):
            mapping = mapping._pb
        if isinstance(mapping, self._meta.pb):
            # Make a copy of the mapping.
            # This is a constructor for a new object, so users will assume
            # that it will not have side effects on the arguments being
            # passed in.
            #
            # The `__wrap_original` argument is private API to override
            # this behavior, because `MessageRule` actually does want to
            # wrap the original argument it was given. The `wrap` method
            # on the metaclass is the public API for this behavior.
            if not kwargs.pop("__wrap_original", False):
                mapping = copy.copy(mapping)
            self._pb = mapping
            if kwargs:
                self._pb.MergeFrom(self._meta.pb(**kwargs))
            return

        # Handle the remaining case by converging the mapping and kwargs
        # dictionaries (with kwargs winning), and saving a descriptor
        # based on that.
        if mapping is None:
            mapping = {}
        mapping.update(kwargs)

        # Update the mapping to address any values that need to be
        # coerced.
        marshal = self._meta.marshal
        for key, value in copy.copy(mapping).items():
            pb_type = self._meta.fields[key].pb_type
            pb_value = marshal.to_proto(pb_type, value)
            if pb_value is None:
                mapping.pop(key)
            else:
                mapping[key] = pb_value

        # Create the internal protocol buffer.
>       self._pb = self._meta.pb(**mapping)
E       TypeError: Parameter to MergeFrom() must be instance of same class: expected google.protobuf.Timestamp got datetime.datetime.

env/lib/python3.7/site-packages/proto/message.py:413: TypeError
__________________________________________________________________________ test_async_unary_stream_reader ___________________________________________________________________________

async_echo = <google.showcase_v1beta1.services.echo.async_client.EchoAsyncClient object at 0x7fc2e5196a20>

    @pytest.mark.asyncio
    async def test_async_unary_stream_reader(async_echo):
        content = 'The hail in Wales falls mainly on the snails.'
        call = await async_echo.expand({
            'content': content,
        }, metadata=metadata)

        # Consume the response and ensure it matches what we expect.
        # with pytest.raises(exceptions.NotFound) as exc:
        for ground_truth in content.split(' '):
            response = await call.read()
            assert response.content == ground_truth
        assert ground_truth == 'snails.'

        trailing_metadata = await call.trailing_metadata()
>       assert trailing_metadata == metadata
E       AssertionError: assert Metadata((('s...llo world'),)) == (('showcase-t...ello world'),)
E         Use -v to get the full diff

tests/system/test_grpc_streams.py:95: AssertionError
______________________________________________________________________ test_async_unary_stream_async_generator ______________________________________________________________________

async_echo = <google.showcase_v1beta1.services.echo.async_client.EchoAsyncClient object at 0x7fc2e52a07f0>

    @pytest.mark.asyncio
    async def test_async_unary_stream_async_generator(async_echo):
        content = 'The hail in Wales falls mainly on the snails.'
        call = await async_echo.expand({
            'content': content,
        }, metadata=metadata)

        # Consume the response and ensure it matches what we expect.
        # with pytest.raises(exceptions.NotFound) as exc:
        tokens = iter(content.split(' '))
        async for response in call:
            ground_truth = next(tokens)
            assert response.content == ground_truth
        assert ground_truth == 'snails.'

        trailing_metadata = await call.trailing_metadata()
>       assert trailing_metadata == metadata
E       AssertionError: assert Metadata((('s...llo world'),)) == (('showcase-t...ello world'),)
E         Use -v to get the full diff

tests/system/test_grpc_streams.py:114: AssertionError
______________________________________________________________________ test_async_stream_stream_reader_writier ______________________________________________________________________

async_echo = <google.showcase_v1beta1.services.echo.async_client.EchoAsyncClient object at 0x7fc2e526dcf8>

    @pytest.mark.asyncio
    async def test_async_stream_stream_reader_writier(async_echo):
        call = await async_echo.chat(metadata=metadata)
        await call.write(showcase.EchoRequest(content="hello"))
        await call.write(showcase.EchoRequest(content="world!"))
        await call.done_writing()

        contents = [
            (await call.read()).content,
            (await call.read()).content
        ]
        assert contents == ['hello', 'world!']

        trailing_metadata = await call.trailing_metadata()
>       assert trailing_metadata == metadata
E       AssertionError: assert Metadata((('s...llo world'),)) == (('showcase-t...ello world'),)
E         Use -v to get the full diff

tests/system/test_grpc_streams.py:173: AssertionError
_____________________________________________________________________ test_async_stream_stream_async_generator ______________________________________________________________________

async_echo = <google.showcase_v1beta1.services.echo.async_client.EchoAsyncClient object at 0x7fc2e51c5390>

    @pytest.mark.asyncio
    async def test_async_stream_stream_async_generator(async_echo):

        async def async_generator():
            yield showcase.EchoRequest(content="hello")
            yield showcase.EchoRequest(content="world!")

        call = await async_echo.chat(async_generator(), metadata=metadata)

        contents = []
        async for response in call:
            contents.append(response.content)
        assert contents == ['hello', 'world!']

        trailing_metadata = await call.trailing_metadata()
>       assert trailing_metadata == metadata
E       AssertionError: assert Metadata((('s...llo world'),)) == (('showcase-t...ello world'),)
E         Use -v to get the full diff

tests/system/test_grpc_streams.py:191: AssertionError
_______________________________________________________________________ test_async_stream_stream_passing_dict _______________________________________________________________________

async_echo = <google.showcase_v1beta1.services.echo.async_client.EchoAsyncClient object at 0x7fc2e5196d30>

    @pytest.mark.asyncio
    async def test_async_stream_stream_passing_dict(async_echo):
        requests = [{'content': 'hello'}, {'content': 'world!'}]
        call = await async_echo.chat(iter(requests), metadata=metadata)

        contents = []
        async for response in call:
            contents.append(response.content)
        assert contents == ['hello', 'world!']

        trailing_metadata = await call.trailing_metadata()
>       assert trailing_metadata == metadata
E       AssertionError: assert Metadata((('s...llo world'),)) == (('showcase-t...ello world'),)
E         Use -v to get the full diff

tests/system/test_grpc_streams.py:205: AssertionError
______________________________________________________________________________ test_nonslash_resource _______________________________________________________________________________

messaging = <google.showcase_v1beta1.services.messaging.client.MessagingClient object at 0x7fc2e527ca20>

    def test_nonslash_resource(messaging):
        expected = "users/bdfl/profile/blurbs/legacy/apocalyptic~city"
>       actual = messaging.blurb_path("bdfl", "apocalyptic", "city")
E       TypeError: blurb_path() takes 2 positional arguments but 3 were given

tests/system/test_resource_crud.py:57: TypeError
_________________________________________________________________________________ test_path_parsing _________________________________________________________________________________

messaging = <google.showcase_v1beta1.services.messaging.client.MessagingClient object at 0x7fc2e52a86d8>

    def test_path_parsing(messaging):
        expected = {"room_id": "tiki"}
        actual = messaging.parse_room_path(messaging.room_path("tiki"))

        assert expected == actual

        expected = {
            "user_id": "bdfl",
            "legacy_user_id": "apocalyptic",
            "blurb_id": "city",
        }
        actual = messaging.parse_blurb_path(
>           messaging.blurb_path("bdfl", "apocalyptic", "city")
        )
E       TypeError: blurb_path() takes 2 positional arguments but 3 were given

tests/system/test_resource_crud.py:74: TypeError

Is there an ETA for this?

A fix PR in our repo should be able to merge today. One error seems like a breakage on their end, but I still haven't found the root cause for the last error. Hopefully, we could resolve it today.


unable to load preload script. cannot read property of undefined.

I used tis minimal example but it thrown me an error what should i do

exact same problem on me
"custom-electron-titlebar": "^3.2.7",
"electron": "^15.0.0"
"vue-cli-plugin-electron-builder": "~2.1.1"
"vue": "^3.0.0"
I have no clue
let me know if you fixed

I solved using the electron version 12.0.0. This version is completely stabke

I solved using the electron version 12.0.0. This version is completely adaptable with electron-custom-builder
        "custom-electron-titlebar": "^3.2.7",
        "internet-available": "^1.0.0",
        "mysql": "^2.18.1"
    },
    "devDependencies": {
        "electron": "^12.0.0",
        "electron-builder": "^22.11.7"

    }```

I solved my problem using the older electron version i.e version 12.0.0 which works finely with the code provided by this guy.
        "custom-electron-titlebar": "^3.2.7",
        "internet-available": "^1.0.0",
        "mysql": "^2.18.1"
    },
    "devDependencies": {
        "electron": "^12.0.0",
        "electron-builder": "^22.11.7"

    }```

I solved my problem using the older electron version i.e version 12.0.0 which works finely with the code provided by this guy.
`
dependencies: {
"custom-electron-titlebar": "^3.2.7",
"internet-available": "^1.0.0",
"mysql": "^2.18.1"
},
"devDependencies": {
"electron": "^12.0.0",
"electron-builder": "^22.11.7"
}

`

I solved my problem using the older electron version i.e version 12.0.0 which works finely with the code provided by this guy.
**"dependencies": {
"custom-electron-titlebar": "^3.2.7",
"internet-available": "^1.0.0",
"mysql": "^2.18.1"
},
"devDependencies": {
"electron": "^12.0.0",
"electron-builder": "^22.11.7"
}**


Issue2284 multiply nonce

Updated Schnorr signature scheme with s = c.r + v to prevent private key extraction after signing multiple times with the same private key.

I don't think we should be changing sign for everything (namely: Transactions). Just on Agora's side for block signature.
But let's discuss on Slack / IRL first.

Closing this PR as it will be only updated for signing the blocks.


dependabot: Resolve updates

Commit Message: dependabot: Resolve updates
Additional Description:
aggregate dependabot updates and use pip-compile to resolve dependency hashes
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]
[Optional API Considerations:]

@htuch it would be good to clear this one

/lgtm deps


[SPARK-37482][PYTHON] Skip check monotonic increasing for Series.asof with 'compute.eager_check'

What changes were proposed in this pull request?
Skip check monotonic increasing for Series.asof with 'compute.eager_check'
Why are the changes needed?
monotonic increasing checking is expensive, so we should use config 'compute.eager_check' to skip this one
Does this PR introduce any user-facing change?
Yes
Before this PR
>>> s = ps.Series([1, 2, np.nan, 4], index=[10, 30, 20, 40])
>>> with ps.option_context("compute.eager_check", False):
...     s.asof(20)
...
21/11/29 17:09:39 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
21/11/29 17:09:40 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
21/11/29 17:09:40 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/u02/spark/python/pyspark/pandas/series.py", line 5220, in asof
    raise ValueError("asof requires a sorted index")
ValueError: asof requires a sorted index

After this PR, when config 'compute.eager_check' is False, pandas-on-Spark just proceeds and performs by ignoring the indeces's order
>>> s = ps.Series([1, 2, np.nan, 4], index=[10, 30, 20, 40])
>>> with ps.option_context("compute.eager_check", False):
...     s.asof(20)
...
1.0

How was this patch tested?
Unit tests

Test build #145707 has started for PR 34737 at commit 42e81e4.

Test build #145707 has finished for PR 34737 at commit 42e81e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145707/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50177/

CC @HyukjinKwon @ueshin @itholic FYI, Thanks!

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50177/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50177/

LGTM.
Would you mind file a JIRA for fixing the bug mentioned in https://github.com/apache/spark/pull/34737#discussion_r758223279 ?

Test build #145736 has started for PR 34737 at commit 7e36783.

Test build #145736 has finished for PR 34737 at commit 7e36783.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50206/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145736/

Merged to master.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50206/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50206/


Reported issue in download-assessment-items-tmp

Supposedly setting ka-lite/download-assessment-items-tmp doesn't work. Let's investigate.
echo "ka-lite-raspberry-pi ka-lite/user select root" | debconf-set-selections
echo "ka-lite-raspberry-pi ka-lite/user seen false" | debconf-set-selections
echo "ka-lite-raspberry-pi ka-lite/download-assessment-items select true" | debconf-set-selections
echo "ka-lite-raspberry-pi ka-lite/download-assessment-items seen false" | debconf-set-selections
echo "ka-lite-raspberry-pi ka-lite/init-instructions seen true" | debconf-set-selections
echo "ka-lite-raspberry-pi ka-lite/download-assessment-items-tmp select /var" | debconf-set-selections
echo "ka-lite-raspberry-pi ka-lite/download-assessment-items-tmp seen false" | debconf-set-selections
echo "ka-lite-raspberry-pi ka-lite/download-assessment-items-url select http://learningequality.org/downloads/ka-lite/0.15/content/khan_assessment.zip" | debconf-set-selections
echo "ka-lite-raspberry-pi ka-lite/download-assessment-items-url seen false" | debconf-set-selections

Works, but made a small tweak in case the directory doesn't exist.


ext_authz: OkResponse.ResponseHeadersToAdd are always appended

When returning ResponseHeadersToAdd on an OkHttpResponse from an ext_authz call using the Go library, the response headers are always appended to the target's headers, regardless of the value of HeaderValueOptions.Append.
Note, too, that this is the opposite of the default claimed on the ResponseHeadersToAdd field:
https://github.com/envoyproxy/data-plane-api/blob/main/envoy/service/auth/v3/external_auth.proto#L102

Hmm seems to be the case that they are always addCopyd. @dio correct me if I'm wrong?

I think we should have another "container" Http::HeaderVector response_headers_to_set_{};? So, when we receive if (response->ok_response().response_headers_to_add_size() > 0) we loop over the headers (options) and check for header.append().value(). We authz_response->response_headers_to_add.emplace_back() when header.append().value() and vice versa, emplace back to authz_response->response_headers_to_set. When encoding headers, we can have if (!response_headers_to_set_.empty()) { with setCopy.

@dio Yes that makes sense. I'm still not clear how the existing bug manifests itself as always appending instead of never appending, though. I think I don't yet understand how append defaults to true, given the proto comment.

I think I have a mistake in the Draft PR above (my mind was mixed up with something else).
Since we always do addCopy when we have entries inside the response_headers_to_add_ vector (set by the onComplete call, the append field value itself is ignored), hence it is always appended (not replaced).

I was previously under the assumption that addCopy (as opposed to appendCopy) only added non-existent headers and didn't append. Now that I read the HeaderMapImpl I see that's wrong, so now I need to understand the difference between these two 🤔

I think in this case using addCopy is the right one when the append value (of a HeaderValueOption) is true. I'm not sure if we are encouraged to use appendCopy anymore. cc. @asraa.


Run shared integration tests on Travis

This PR adds an npm script npm run test-shared-integration-tests. This script runs the shared integration tests from the ringpop-common repository.
It also adds the running of these tests to Travis CI.

What is this wizardry? LGTM 🎩

@severb glad you didn't ask where the tests were...

For ringpop-go we are about to land a change that lets us pick a feature branch for integration tests during development: https://github.com/uber/ringpop-go/blob/feature/reaping-faulty-nodes/test/run-integration-tests#L9
would it be possible to port that to this one as well? It has proven really useful in adding new integration tests.

LGTM. +1 for adding (e.g. an env-variable) to specify the branch of ringpop-common to checkout.

+1 too. I filed something internally for us to follow up later.

LGTM, thanks for porting the branch selection.


sftp FUSE support

From the implementation in sftp.rs , it looks like FUSE support, or close to FUSE support.
@wez , any suggestion about how easily one could create a FUSE system via the sftp implemented in this project. Thanks.

You could probably stick this crate together with https://docs.rs/fuse/ relatively easily.
To make a fuse filesystem work well requires some additional effort, so you'd need to have a good reason to put in that effort that rather than just use something like https://github.com/libfuse/sshfs.

Thanks Wez.
That makes sense. I still need to do something by myself. What I want to do
is to automatically link to a server to access a file via ftp or http
flexibly when there is a need. The client may access different servers
without a need to connect to a server when mounting. So, I think I need to
do connection while open a file, and close it when close a file.
On Sat, Nov 23, 2019 at 11:13 AM Wez Furlong<EMAIL_ADDRESS>wrote:

You could probably stick this crate together with https://docs.rs/fuse/
relatively easily.
To make a fuse filesystem work well requires some additional effort, so
you'd need to have a good reason to put in that effort that rather than
just use something like https://github.com/libfuse/sshfs.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/alexcrichton/ssh2-rs/issues/144?email_source=notifications&email_token=ACPN5S5J3IC7JMEDP57VOZTQVCNVBA5CNFSM4JQX5YF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE7MBZQ#issuecomment-557760742,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACPN5SZUDMIQRTVDBIHHEHTQVCNVBANCNFSM4JQX5YFQ
.


PyTorchEngine support latest internlm2 modeling.

May merge the latest main since https://github.com/InternLM/lmdeploy/pull/1806 has resolved pr_ete_test CI workflow error

Shall we make some log hints for users who used old internlm2?

There is no commit-id or other arguments in the config that indicate the version of code.


Add retry logic around checkExposedPort pre-flight check

… for improved robustness (fixes #513).

#513 sounds like a challenging one to reproduce - it's not something I've seen before. Based on the code, a Null Pointer Exception should not be raised if things are working. It must only happen if the exposed port check is already on the way to failing. Because it's an intermittent issue, it seems there might be a timing issue.
With this PR I've added a simple, 3 second, retry loop around the port check logic. It's hard to know if this will help, but it's probably the most we can do to try and resolve without a way to reproduce.

thanks for fix @rnorth. I see this change in CHANGELOG.mdas implemented in 1.5.0 but we are using latest 1.5.0 from maven central repository and  It's not here. Also I see this PR as open.

Damn, sorry - not sure what happened here, but undoubtedly my fault. Will release with this included ASAP.

thanks everybody for fixing. @rnorth when do we expect to release new version?


Not wait kafka message ack

I investigate whether clean up could cause promotion to fail.
For NFC clean up, it is async by thread-pool https://github.com/Commonjava/indy/blob/master/core/src/main/java/org/commonjava/indy/core/content/DefaultContentManager.java#L387 . If it fails, only prints an error.
For metadata clean, it explicitly call:
contentManager.delete( tgt, path, eventMetadata ) ->
contentGeneratorManager.handleContentDeletion( store, path, eventMetadata ) ->
AbstractMergedContentGenerator.clearMergedFile -> it only print an error log if it fails.
For pom, it calls MetadataMergePomChangeListener.onPomStorageEvent -> metaClear -> it only log warns if fails.
So, for all clean-up the indy won't fail the promotion, if I did not miss anything. Considering we'd like to keep Kafka message flexibility, I drop the 'waiting' after sending the promote complete event.

If we can confirm that is the existing case, looks good for me.


Add list of policies that match an endpoint to endpoint

This is probably a better solution than #1556 as it will provide the information consistently.

@raybejjani I'm reopening this. I do not see anything with an L3 only policy.
    "policy": {
      "allowed-ingress-identities": [
        1
      ],
      "build": 176,
      "cidr-policy": {
        "egress": [],
        "ingress": []
      },
      "id": 13279,
      "l4": {
        "egress": [],
        "ingress": []
      }
    }

Policy example:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress

Endpoints:
$ cilium endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                   IPv6                 IPv4          STATUS
           ENFORCEMENT        ENFORCEMENT
11004      Enabled            Disabled          13279      k8s:io.kubernetes.pod.namespace=kube-system   f00d::a0f:0:0:2afc   <IP_ADDRESS>   ready
                                                           k8s:k8s-app=kube-dns
29898      Disabled           Disabled          38559      reserved:health                               f00d::a0f:0:0:74ca   <IP_ADDRESS>   ready

I've been looking at this on and off. The current code definitely doesn't work for the default deny case. I think it also doesn't work correctly when using identities but those might show up as l4 and I wrote the test in the wrong place.
I'll follow up on this but I'm not sure sure I understand our policy logic well enough :(


docs: add funding information

Description

Do you introduce a breaking change?

[ ] Yes
[ ] No
[ ] I am unsure (no worries, we'll find out)

Merged as minor change, but without proper review. Hope this doesn't bother anybody.


'Unknown' appearing in IE11

Environment

Tech
Version

material-ui-pickers
v1.0.0-rc.1

material-ui
v1.0.0-beta.35

React
16.2.0

Browser
IE11

Steps to reproduce

In IE11, try to add a Date in the picker.

Expected behavior
Date should be displayed.
Actual behavior
"Unknown" appears.
The possible reason for this is in the DateFnsUtils class: the date function uses the constructor of the date class; in all browsers new Date(10-Jan-2017) returns a valid Date. However, in IE11 this returns an "Invalid Date".

Would you like to open a PR?
I have no ability to test on the IE now :(

Seems to be not reproducible with rc-3


[CP STAGING] fix: unread marker for current user action

Details

fix: unread marker for current user action
Fixed Issues

$ https://github.com/Expensify/App/issues/47375
PROPOSAL: https://github.com/Expensify/App/issues/47375#issuecomment-2287876196
Tests

Open any report
Send some messages
Mark any message as unread
Verify that the unread marker appears

[x] Verify that no errors appear in the JS console

Offline tests

Same as above
QA Steps

Open any report
Send some messages
Mark any message as unread
Verify that the unread marker appears

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native
https://github.com/user-attachments/assets/3bb1ab52-43e7-406e-bb29-e5a1827af725

Android: mWeb Chrome
https://github.com/user-attachments/assets/422f7d6d-a553-4287-b9e3-60e0740cdc6e

iOS: Native
https://github.com/user-attachments/assets/300c2a34-b367-400f-8183-30016a539e6e

iOS: mWeb Safari
https://github.com/user-attachments/assets/7537ce39-3a85-4b01-89cb-8aec0b6540a7

MacOS: Chrome / Safari
https://github.com/user-attachments/assets/05527faa-5885-4b7d-a4a6-934bb1c21009

MacOS: Desktop
https://github.com/user-attachments/assets/e0766b52-a1cf-4ea0-9d22-efe41c82052a

Reviewer Checklist

[x] I have verified the author checklist is complete (all boxes are checked off).
[x] I verified the correct issue is linked in the ### Fixed Issues section above
[x] I verified testing steps are clear and they cover the changes made in this PR

[x] I verified the steps for local testing are in the Tests section
[x] I verified the steps for Staging and/or Production testing are in the QA steps section
[x] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I checked that screenshots or videos are included for tests on all platforms
[x] I included screenshots or videos for tests on all platforms
[x] I verified tests pass on all platforms & I tested again on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[x] I verified proper code patterns were followed (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I verified that this PR follows the guidelines as stated in the Review Guidelines
[x] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG)

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: Native

https://github.com/user-attachments/assets/0851397e-be81-46ed-9f63-4591540aaf92

Android: mWeb Chrome

// build issue

iOS: Native

https://github.com/user-attachments/assets/e898f897-3a39-4c91-9b53-272598a1263c

iOS: mWeb Safari

https://github.com/user-attachments/assets/b9dcd45d-9229-447d-979d-5f5da5cb9072

MacOS: Chrome / Safari

https://github.com/user-attachments/assets/f13a8974-7e82-43c6-a201-d4d8cddb24a1

MacOS: Desktop

https://github.com/user-attachments/assets/5e3bfcef-81ee-4ccf-97fa-c53fc27013cf

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Cherry-picked to staging by https://github.com/mountiny in version: 9.0.20-1 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

@Expensify/applauseleads please QA this PR and check it off on the deploy checklist if it passes.

FYI I believe this was deployed to prod yesterday, from this checklist - #47356


remove getAmopTopicSubscriber function

Codecov Report

Merging #193 into release-2.7.0 will increase coverage by 0.03%.
The diff coverage is n/a.

@@                 Coverage Diff                 @@
##             release-2.7.0     #193      +/-   ##
===================================================
+ Coverage            17.56%   17.60%   +0.03%
  Complexity             682      682
===================================================
  Files                  418      417       -1
  Lines                12683    12658      -25
  Branches              1472     1467       -5
===================================================
  Hits                  2228     2228
+ Misses               10211    10186      -25
  Partials               244      244

Impacted Files
Coverage Δ
Complexity Δ

...rc/main/java/org/fisco/bcos/sdk/client/Client.java
0.00% <ø> (ø)
0.00 <0.00> (ø)

...ain/java/org/fisco/bcos/sdk/client/ClientImpl.java
0.00% <ø> (ø)
0.00 <0.00> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b9dd11...279966e. Read the comment docs.


nixos/jenkins: test declarative jobs

Motivation for this change
Ensure that the nixos declarative jenkins jobs work.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Added a release notes entry if the change is major or breaking
[x] Fits CONTRIBUTING.md.

nixos/jenkins has no maintainers, so as a fallback adding the maintainers of pkgs.jenkins instead as reviewers.


fix: remove dependency on lodash

isomorphic-cookie currently loads a huge chunk of lodash for just one function. By changing the dependency to lodash.pick instead of lodash, this dependency is much smaller.

Thanks @ronderksen I'll look at this ASAP.

@ronderksen This is released with v1.2.4


Add help text to sui start for ProtocolConfig overrides

If one of you wants to approve this I can merge it

@aschran I approved this as it looked low risk to me, but let me know if there's anything that I missed.


Libvirtd container support

This small series of patches will enable libvirtd image builds, and allows us to start using (experimenting) with the containerized libvirtd as the runtime for the domains.

+1

Got an ack, merging. Thanks.


Database seeding prisma2

Problem

Suggested solution

Alternatives

Additional context

Closing as a duplicate of https://github.com/prisma/prisma/issues/9


Skill bridge not reachable

Hi,
The site https://googlehome.hardill.me.uk/ is not accessible right now.
Node-RED Google Assistant Smart Home Skill Bridge
It looks like the skill is currently experiencing a problem at the moment. Give it a couple of minutes and if it's still a problem please check if there is an existing open issue on GitHub here, if not then raise one.
It this a known issue?
Many thanks
Geoffrey

Back up, no obvious reason why it was down in the logs will check.

Thanks for the quick reponse and solution. Great work!


[device_info_platform_interface] Typo in doc fixed

Description
In this PR, I have fixed the documentation error i made while writing it.
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[x] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (flutter analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x] The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
[x] I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy.
[x] I updated CHANGELOG.md to add a description of the change.
[x] I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[x] No, this is not a breaking change.

Hey @ditman kindly have a look at this one.

Merged and published! Thanks @yash1200!

Thank you @ditman :)


"One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag" error when multipart uploading a 7GB file

I'm trying to multipart upload a 7Gb file using 5MB parts (total parts count = 1434).
When calling CompleteMultipartUpload, I'm getting the error: "One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag"
I'm getting the same error when trying to multipart upload a 10Gb or 40Gb file.
However when I multipart upload a 200Mb or 1Gb file, everything works fine.

Hey @cpaul06 -
We are going to look at your issue.
Thanks.

@nicolas2bert - Here is the code with which I could reproduce the issue:
CompleteUpload.txt

@nicolas2bert - Using a part count above 1000 in the attached code, I get the error mentioned in the title: "One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag" .
But with a value smaller than 1000 (for ex: 3, 50, 999), I'm getting: "Your proposed upload is smaller than the minimum allowed object size."

@cpaul06 - I am currently looking at it and trying to reproduce your issue.
Can you give me more information about your system, memory size and storage space?
Thank you.

@cpaul06, is it possible that putting one of your parts resulted in an error so you added to your list of etags an undefined and then tried to complete the MPU with a list of etags that included undefined?  Do you have an errors in your log?  Could you add a log to your script to check for any errors?
Regarding your "upload is smaller" comment, if an object size is less than 5 MB, that error is returned.  Were your part sizes (other than the last part) greater than 5 MB?
Thanks!

@LaurenSpiegel - Please disregard the "upload is smaller" comment, it was my fault; as you said not all the parts were greater than 5MB (except the last)
When do you think scality/Arsenal#165 will be available in this repository and then in the docker container image?

Hi @cpaul06, thanks for the clarification.  We are currently testing the arsenal fix and the new docker should be out next week.

Hey @cpaul06 - this issue has been fixed. Please run the lastest / ece570f  S3 server Docker image: https://hub.docker.com/r/scality/s3server/


updating [string] field

Repo
Autoform has a bug when updating a field that has a type of [String]. Instead of array it appends the string as objects which  misses things up.

#777

It's actually a mongo bug, but it's now visible in AF 5.0. Still figuring out the best way to address it, but in the meantime, you just need to ensure that the field in the database is set to an array before submitting an update form.

Workaorund
// tags: [String]
AutoForm.hooks({
autoFormId: {
before: {
update: function(doc) {
// delete tags.0, tags.1, ... from $set
var ta = [];  // tempTagArray
var ts = {};  // temp$set
ts = this.updateDoc.$set;  // do not modify enumerator directly
for(var key in this.updateDoc.$set) {
if (key.indexOf('tags.', 0) === 0) {
ta.push(this.updateDoc.$set[key]);
delete ts[key];
}
}
// if there are tags
if (Object.keys(ta).length != 0) {
ts = _.extend(ts, { tags: ta });  // new $set
}
this.updateDoc.$set = ts;
    return doc;
  }
}

});

May be worth noting: This bug affects me too, but only happens when updating existing documents which do not yet have the to-be-modified array field. This happens a lot when the field is optional. Creating a new document with a populated array field uses the correct field type in the database.

A very clean workaround is to simply add defaultValue: [] to the schema for all optional array fields. Collection2 will then replace all unset array fields by empty arrays on the server. This cannot be bypassed on the client and can therefore not be broken by a malicious user.
AutoForm's $set and $unset queries behave correctly afterwards. The empty array causes Mongo to use array operations instead of building an object in the document.

Yes, there is already defaultValue: [] in the schema.
But if you delete some of array item, problem starts!
Yes, It should be done in Server with Collection hooks not Client AutoForm hooks.

@SenerDemiral yes that's right when you remove everything from the array field, problem starts!. problem is also there when you create a document with empty array and try updating it after.

@mologie adding defaultValue: [] to the field actually helps. when you first create the document using auto forms, you can update it after fine which you couldn't before without the defaultValue: []. but after removing all the items from that field as @SenerDemiral  mentioned, the bug is back.

Please retest with AutoForm 5.1.0 and close if your issue is resolved.

This is fixed!. Thanks @aldeed  and @SenerDemiral  for your workaround.


Crashes of  ffi/function_structs_test and ffi/function_test/0 (flaky)

We have flaky crashes in the FFI tests, from log:

/================================================================\
| ffi/function_structs_test broke (Pass -> Crash, expected Pass) |
\================================================================/

--- Command "vm" (took 10.000566s):
DART_CONFIGURATION=ReleaseIA32 out/ReleaseIA32/dart --optimization-counter-threshold=5 --random-seed=1621125267 --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/tests/ffi/function_structs_test.dart

exit code:
-6

stdout:
TransposeCoordinate(0x51baaa0 {10, 20, 0x51baac0})
returning 0x51baac0
CoordinateElemAt1(0x4534dc0)
sizeof(Coord): 20
coord[0] = {10, 10, 0x4534de8}
coord[1] = {20, 20, 0x4534dc0}
returning 0x4534dd4
SumVeryLargeStruct(0x467a000)
offsetof(a): 0
offsetof(b): 2
offsetof(c): 4
offsetof(d): 8
offsetof(e): 16
offsetof(f): 18
offsetof(g): 20
offsetof(h): 24
offsetof(i): 32
offsetof(j): 36
offsetof(k): 44
offsetof(parent): 48
offsetof(numChildren): 52
offsetof(children): 56
offsetof(smallLastField): 60
sizeof(VeryLargeStruct): 64
vls->a: 1
vls->b: 2
vls->c: 4
vls->d: 8
vls->e: 16
vls->f: 32
vls->g: 64
vls->h: 128
vls->i: 256
vls->j: 512
vls->k: 1024
vls->parent: 0x467a040
vls->numChildren: 2
vls->children: 0x467a000
vls->smallLastField: 1
2048
has parent
has 2 children
returning 2051

stderr:
===== CRASH =====
si_signo=Segmentation fault(11), si_code=2, si_addr=0xf75b84ef
version=2.5.0-edge.3c05b8199b37b5ed71ed7df2b8c86085cd7ba745 (Wed Jul 24 18:38:43 2019 +0000) on "linux_ia32"
thread=3332, isolate=main(0x454e700)
  pc 0xf75b84ef fp 0xf187edc4 Unknown symbol
  pc 0xf75b84af fp 0xf187edd8 Unknown symbol
  pc 0xf75b5775 fp 0xf187ee10 Unknown symbol
  pc 0xf75af188 fp 0xf187ee20 Unknown symbol
  pc 0xf75af098 fp 0xf187ee38 Unknown symbol
  pc 0xf75aef92 fp 0xf187ee4c Unknown symbol
  pc 0xf75ae28a fp 0xf187ee68 Unknown symbol
  pc 0xf7588d75 fp 0xf187ee80 Unknown symbol
  pc 0xf75adfd8 fp 0xf187ee98 Unknown symbol
  pc 0xf7a00c4d fp 0xf187eec4 Unknown symbol
  pc 0x018ca4a7 fp 0xf187ef38 dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&, dart::Array const&, unsigned int)
  pc 0x018ca0d4 fp 0xf187ef78 dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&)
  pc 0x018cd1da fp 0xf187efb8 dart::DartLibraryCalls::HandleMessage(dart::Object const&, dart::Instance const&)
  pc 0x01900d33 fp 0xf187f0e8 dart::IsolateMessageHandler::HandleMessage(std::__2::unique_ptr<dart::Message, std::__2::default_delete<dart::Message> >)
  pc 0x0192dce5 fp 0xf187f138 dart::MessageHandler::HandleMessages(dart::MonitorLocker*, bool, bool)
  pc 0x0192e551 fp 0xf187f198 dart::MessageHandler::TaskCallback()
  pc 0x0192eca7 fp 0xf187f1b8 out/ReleaseIA32/dart+0x152eca7
  pc 0x01a489c1 fp 0xf187f1f8 dart::ThreadPool::Worker::Loop()
  pc 0x01a48867 fp 0xf187f228 dart::ThreadPool::Worker::Main(unsigned int)
  pc 0x019bb2a4 fp 0xf187f2e8 out/ReleaseIA32/dart+0x15bb2a4
  pc 0xf7f4d295 fp 0xf187f3a8 /lib/i386-linux-gnu/libpthread.so.0+0x6295
-- End of DumpStackTrace

--- Re-run this test:
python tools/test.py -n dartk-optcounter-linux-release-ia32 ffi/function_structs_test

/============================================\
| ffi/function_test/0 succeeded again (Pass) |
\============================================/

--- Command "vm" (took 06.000020s):
DART_CONFIGURATION=ReleaseIA32 out/ReleaseIA32/dart --optimization-counter-threshold=5 --random-seed=1877274570 --ignore-unrecognized-flags --packages=/b/s/w/ir/cache/builder/sdk/.packages /b/s/w/ir/cache/builder/sdk/tests/ffi/function_test.dart

exit code:
-6

stdout:
SumPlus42(3, 4)
returning 49

stderr:
===== CRASH =====
si_signo=Segmentation fault(11), si_code=2, si_addr=0xf75af0c0
version=2.5.0-edge.3c05b8199b37b5ed71ed7df2b8c86085cd7ba745 (Wed Jul 24 18:38:43 2019 +0000) on "linux_ia32"
thread=20006, isolate=main(0x44f6700)
  pc 0xf75af0c0 fp 0xf5cbedd0 Unknown symbol
  pc 0xf75af080 fp 0xf5cbede8 Unknown symbol
  pc 0xf75acc82 fp 0xf5cbee0c Unknown symbol
  pc 0xf75abe9c fp 0xf5cbee20 Unknown symbol
  pc 0xf75abd38 fp 0xf5cbee38 Unknown symbol
  pc 0xf75abc32 fp 0xf5cbee4c Unknown symbol
  pc 0xf75aac6a fp 0xf5cbee68 Unknown symbol
  pc 0xf7588d75 fp 0xf5cbee80 Unknown symbol
  pc 0xf75aa9b8 fp 0xf5cbee98 Unknown symbol
  pc 0xf7a00c4d fp 0xf5cbeec4 Unknown symbol
  pc 0x018ca4a7 fp 0xf5cbef38 dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&, dart::Array const&, unsigned int)
  pc 0x018ca0d4 fp 0xf5cbef78 dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&)
  pc 0x018cd1da fp 0xf5cbefb8 dart::DartLibraryCalls::HandleMessage(dart::Object const&, dart::Instance const&)
  pc 0x01900d33 fp 0xf5cbf0e8 dart::IsolateMessageHandler::HandleMessage(std::__2::unique_ptr<dart::Message, std::__2::default_delete<dart::Message> >)
  pc 0x0192dce5 fp 0xf5cbf138 dart::MessageHandler::HandleMessages(dart::MonitorLocker*, bool, bool)
  pc 0x0192e551 fp 0xf5cbf198 dart::MessageHandler::TaskCallback()
  pc 0x0192eca7 fp 0xf5cbf1b8 out/ReleaseIA32/dart+0x152eca7
  pc 0x01a489c1 fp 0xf5cbf1f8 dart::ThreadPool::Worker::Loop()
  pc 0x01a48867 fp 0xf5cbf228 dart::ThreadPool::Worker::Main(unsigned int)
  pc 0x019bb2a4 fp 0xf5cbf2e8 out/ReleaseIA32/dart+0x15bb2a4
  pc 0xf7f42295 fp 0xf5cbf3a8 /lib/i386-linux-gnu/libpthread.so.0+0x6295
-- End of DumpStackTrace

--- Re-run this test:
python tools/test.py -n dartk-optcounter-linux-release-ia32 ffi/function_test/0

@sjindel-google @dcharkes

I can reproduce both these crashes locally with a handful of runs.
The first crash is only relevant after FFI 32-bit structs (the test is unsupported otherwise). The second crash does not appear to be related to 32-bit structs -- it reproduces before and after.

I am able to reproduce one variant of the crash pretty consistently in precompiled mode:
> pkg/vm/tool/gen_kernel --no-aot --platform=out/DebugX64/vm_platform_strong.dill -o /tmp/out.dill tests/ffi/function_test.dart -Ddart.developer.causal_async_stacks=true
> out/DebugX64/dart /tmp/out.dill

Fails with the following error:
returning 1337
NoSuchMethodError: The method 'FfiTrampoline' was called on null.
Receiver: null
Tried calling: FfiTrampoline(), #0      FfiTrampoline (dart:ffi)
#1      testException (file:///usr/local/google/home/asiva/workspace/dart-tot/sdk/tests/ffi/function_test.dart:440:14)
#2      main (file:///usr/local/google/home/asiva/workspace/dart-tot/sdk/tests/ffi/function_test.dart:42:5)
#3      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:305:19)
#4      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:172:12)

SumPlus42(3, 4)

@a-siva That exception is supposed to be printed. The test throws and catches it, and prints the exception even when it's been caught. The exit code should still be 0.

Problem
This issue is trigged when FLAG_write_protect_code is enabled without dual-mapping.
It surfaced as a race condition between the background compiler and the mutator thread:

Mutator
Background Compiler

Running generated code.
Compiling some function.

Requests safepoint to install code.

FFICall instruction begins.

Enters safepoint before transitioning to native.

Native code runs, in safepoint.

Installs code on the same page as the FFICall.

Switches page protection RX -> RW.

Returns into Dart code (still in safepoint).

FFICall attempts to exit safepoint.

Generated code to exit the safepoint is not itself executable. Crash.

This issue uncovers a new constraint that native code (which runs in a safepoint) cannot return or call into generated code directly (in JIT mode with write-protected code and no dual-mapping support).
This bug affects both Dart → native and native → Dart calls.
Solutions
Dart → Native
When necessary (FLAG_write_protect_code && !FLAG_dual_map_code), we can dispatch through native code through a stub.
Since stub code lives in the VM heap, it will not change protection after an isolate has initialized, since we don't compile more code into the VM isolate.
Instead of this:
__ TransitionGeneratedToNative(target_address, FPREG);
__ CallCFunction(target_address);
__ TransitionNativeToGenerated();

The generated code would look like this:
__ movq(RBX /*callee-saved*/, target_address);
__ movq(TMP, Address(THR, target::Thread::safepoint_and_call_native()));
__ movq(TMP, FieldAddress(TMP, target::Code::entry_point_offset()));
__ call(TMP)

The challenge is that we need one volatile non-argument register available to pass the target address and one callee-saved register to save the generated code's return address, since the stack holds arguments and cannot be manipulated by the stub.
However, we can save additional state in the exit frame if necessary.
Native → Dart
Native to Dart calls are tricker because we need to manufacture new target addresses during an isolate's lifetime and cannot just point to a VM stub.
Native callbacks live until the end of an isolate, so we can create pages outside the isolate heap which are mapped executable once and never re-used.
Allocate instructions outside isolate heap
We can allocate the callback's Instructions directly outside the isolate heap and point the Code object to them.
This would require that we waste at least one page on each callback, even if its code is less.
Protections are applied at the page granularity, so we can never mark the rest of the page writable without breaking the callback at the start of the page.
We've observed that the instruction sizes of native callbacks generated by our tests range from 500 - 3000 bytes, so with a page size of 4 KB, we would waste between 7X and 0.33X the trampoline size.
Allocate trampolines outside isolate heap
We can save memory by allocating several trampolines of predicable code, which record the callback ID they were generated for and call into the runtime, similarly to the multiple-mapping solution for AOT callbacks. Since callbacks IDs are assigned sequentially, we can predict the trampoline code for callbacks which have not been generated yet, avoiding any wasted memory. However, this solution is more complex and might slightly regress callback performance.
We will likely choose the first solution for now due to its simplicity.

https://dart-review.googlesource.com/c/sdk/+/111280 fixes the Dart → Native half.

Martin pointed out that the GC needs to scan and update embedded pointers in the callbacks' code on IA32, so allocating their instructions outside the heap is not possible.
We'll have to go with the trampolines solution.

https://dart-review.googlesource.com/c/sdk/+/111432 addresses the Native → Dart half.

I've confirmed in BigQuery that the flaky crashes have stopped.


Do you have any idea why it happens? Thanks a lot!

2019-12-10 14:47:13,655 ERROR worker.py:1559 -- Possible unhandled error from worker: ray_LearnerActor:update() (pid=93368, host=iMac-adminposh-6.local)
File "Deep-CFR/DeepCFR/workers/la/dist.py", line 11, in init
LocalLearnerActor.init(self, t_prof=t_prof, worker_id=worker_id, chief_handle=chief_handle)
File "Deep-CFR/DeepCFR/workers/la/local.py", line 24, in init
self.env_bldr = rl_util.get_env_builder(t_prof=t_prof)
File "python3.7/site-packages/PokerRL/rl/rl_util.py", line 84, in get_env_builder
return ENV_BUILDER(env_cls=get_env_cls_from_str(t_prof.game_cls_str), env_args=t_prof.module_args["env"])
File "python3.7/site-packages/PokerRL/game/wrappers.py", line 49, in init
super().init(env_cls=env_cls, env_args=env_args)
File "python3.7/site-packages/PokerRL/game//EnvWrapperBuilderBase.py", line 25, in init
self.lut_holder = env_cls.get_lut_holder()
File "python3.7/site-packages/PokerRL/game//rl_env/game_rules.py", line 312, in get_lut_holder
return LutHolderHoldem(cls)
File "python3.7/site-packages/PokerRL/game//look_up_table.py", line 303, in init
super().init(lut_getter=LutGetterHoldem(env_cls=env_cls))
File "python3.7/site-packages/PokerRL/game//look_up_table.py", line 100, in init
n_cards_out_lut=self.get_n_cards_out_at_LUT())
File "python3.7/site-packages/PokerRL/game//cpp_wrappers/CppLUT.py", line 18, in init
"lib_luts." + self.CPP_LIB_FILE_ENDING))
File "python3.7/site-packages/PokerRL//CppWrapper.py", line 18, in init
self._clib = ctypes.cdll.LoadLibrary(path_to_dll)
File "python3.7/ctypes/init.py", line 434, in LoadLibrary
return self._dlltype(name)
File "python3.7/ctypes/init.py", line 356, in init
self.handle = dlopen(self.name, mode)
OSError: dlopen(python3.7/site-packages/PokerRL/game//cpp_wrappers/lib_luts.so, 6): no suitable image found.  Did find:
python3.7/site-packages/PokerRL/game//cpp_wrappers/lib_luts.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00
python3.7/site-packages/PokerRL/game//cpp_wrappers/lib_luts.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00
2019-12-10 14:47:14,423 WARNING worker.py:1619 -- The actor or task with ID ffffffffffff26acd28101000000 is pending and cannot currently be scheduled. It requires {CPU: 1.000000} for execution and {CPU: 1.000000} for placement, but this node only has remaining {object_store_memory: 1.562500 GiB}, {memory: 2.832031 GiB}. In total there are 0 pending tasks and 16 pending actors on this node. This is likely due to all cluster resources being claimed by actors. To resolve the issue, consider creating fewer actors or increase the resources available to this Ray cluster.

hello, im getting a similar error, have you found a way to fix it?

hello, im getting a similar error, have you found a way to fix it?

I have the same issue with my Mac, but it works fine with Linux. I think it is a Linux compile file. And Mac is using Clang.

@EricSteinberger I guess the easiest way is the author provides us a mac version compiled file? Because of .so file and .dlibs are completely different.
self._handle = _dlopen(self._name, mode)

OSError: dlopen(/Deep-CFR/DeepCFR/PokerRL/game//cpp_wrappers/lib_luts.so, 6): no suitable image found.  Did find:
/Deep-CFR/DeepCFR/PokerRL/game//cpp_wrappers/lib_luts.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00
/Deep-CFR/DeepCFR/PokerRL/game/_/cpp_wrappers/lib_luts.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00

Hi, sorry for the delay! Yes, Mac is not supported at the moment, your guess is right. I recommend spinning up an AWS instance as described in the README - it's really cheap and then everything is reproducable. The program won't run on a mac, unfortunately.


Out of memory on writing to tantivy index

I have "ValueError: Failed to open file for read: 'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot allocate memory" }, filepath: "/index/TANTIVY/Content.en/LearningObjectives/9bab6227e72543aabb95be37a875fa77.fast" when trying to write new data to the index. Increasing container's RAM does not help. Does writer memory limit influence that, or this file reading is independent?

Thanks for the bug report. The memory budget you give to the tantivy writer is not the upper limit, we go higher than that when flushing the index to disk (Should be around 2x if I remember) . Are you using multiple threads when indexing? How much memory did you allocate? Maybe try lowering it.
Can you share your index config?

Thanks for reply! That is how it is done (FYI:That is python binding):
`
def makeSchema(self):
return SchemaBuilder()
.add_text_field("path", stored = True, index_option = "basic",  tokenizer_name = "raw")
.add_text_field("title", stored = "title" in self.storedFields, tokenizer_name = "default")
.add_text_field("content", stored = "content" in self.storedFields, tokenizer_name = "default")
.add_text_field("keywords", stored = "keywords" in self.storedFields, index_option = "basic", tokenizer_name = "whitespace")
.build()
WRITER_HEAP_MB = 256
WRITER_THREADS = 4
w = self.getIndex().writer(heap_size=TantivyIndex.WRITER_HEAP_MB*1024*1024,

num_threads=TantivyIndex.WRITER_THREADS)
....
`

Can you test it with much less memory per indexer, e.g.
WRITER_HEAP_MB = 128
WRITER_THREADS = 1

Nothing helps. @PSeitz, is this correct place for that *.fast file openning error source: https://github.com/quickwit-oss/tantivy/blob/main/src/directory/directory.rs#L124  ?

Yes, that's the place. Opening a file doesn't cost barely any memory normally, do you use some sort of ramdisk or custom Directory implementation?

Which directory are you using?
If it is MMapDirectory, keep in mind this is not necessarily a problem of lack of memory. You might be hitting a virtual memory limit.
tantivy relies on mmap for its IO, and requires a lot of virtual memory.
Make sure you do not have any limit on that.
Use your OS tools to inspect and report the resident memory usage and the virtual memory usage, to check which one is hitting your limit...

For the virtual memory size, can you post the output of
ulimit -a | grep virtual

cat /proc/meminfo | grep Vm

@fulmicoton I'm using this python binding : https://github.com/quickwit-oss/tantivy-py
And it looks like ut uses MmapDiractory: https://github.com/quickwit-oss/tantivy-py/blob/master/src/index.rs#L179

@PSeitz
root@5aff4ff9ca15:/# ulimit -a | grep virtual
virtual memory              (kbytes, -v) unlimited
root@5aff4ff9ca15:/# cat /proc/meminfo | grep Vm
VmallocTotal:<PHONE_NUMBER>7 kB
VmallocUsed:       90584 kB
VmallocChunk:          0 kB

@PSeitz , is ManagedDirectory an alternative for MmapDirectory? Or is there any way to avoid mmap-ing?

No.


Use black on more Python files.

Picked some other files that require relatively little change.

Can you resolve conflicts?

All conflicts are resolved.


⚠ Bofrost.nl has degraded performance

In 5b70b43, Bofrost.nl (https://www.bofrost.nl) experienced degraded performance:

HTTP code: 200
Response time: 947 ms

Resolved: Bofrost.nl performance has improved in 27aae9d.


Creating User struct for storing user details

Since the FirstName, LastName, Location, etc are all related to a person. We can create a struct named person and store all the information related to the person in that struct. This will help in code organization.

I would like to work on this issue. Please assign it to me under GSSoC'23.

Yah no issue, just try the code and test on your local if everything works fine then do let me know.
PS: Don't forget to restart the hardhat so that i will compile the  code .

is this task assigned or not?


Compiling pulsar-client-cpp on Ubuntu Server 20.04 did not work as intended

Describe the bug
Compiling pulsar-client-cpp on Ubuntu Server 20.04 did not work as intended
To Reproduce
Steps to reproduce the behavior:

follow the compile instruction
the first step works fine
the second and third step reports a error: cp: cannot stat '*.a': No such file or directory:

the fourth step also reports a error:

Expected behavior
Compiling pulsar-client-cpp on Ubuntu Server 20.04 should work as intended
Desktop (please complete the following information):
cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Additional context
more details in CMakeOutput.log and  CMakeError.log
CMakeError.log
CMakeOutput.log

the second and third step reports a error: cp: cannot stat '*.a': No such file or directory

Please note that the command sudo cp *.a /usr/lib only intends to copy the libs *.a (you just built) to the path /usr/lib so that you can use them in pulsar.
From your log, I kown that in your local env, the libs *.a after linking were put into path /usr/src/gtest/lib and /usr/src/gmock/lib instead of the source code root path /usr/src/gtest and /usr/src/gmock. So you need to use command sudo cp lib/*.a /usr/lib instead of sudo cp *.a /usr/lib.
Maybe the compile instructions need to be adjusted to meet such situation. I will submit a patch to fix the document.

the second and third step reports a error: cp: cannot stat '*.a': No such file or directory

Please note that the command sudo cp *.a /usr/lib only intends to copy the libs *.a (you just built) to the path /usr/lib so that you can use them in pulsar.
From your log, I kown that in your local env, the libs *.a after linking were put into path /usr/src/gtest/lib and /usr/src/gmock/lib instead of the source code root path /usr/src/gtest and /usr/src/gmock. So you need to use command sudo cp lib/*.a /usr/lib instead of sudo cp *.a /usr/lib.
Maybe the compile instructions need to be adjusted to meet such situation. I will submit a patch to fix the document.

Thanks


Modifying dictionary during iteration

Hi,
In fit.py, line 1303, you have the following:
for pname in out['posterior_samples'].keys():
                    if 'sigma_w_rv' == pname[:10]:
                        instrument = pname.split('_')[-1]
                        out['posterior_samples']['sigma_w_'+instrument] = out['posterior_samples'][pname]

That last line is effectively modifying the dictionary you're currently iterating through in the case where RV datasets are analysed, this induces the error RuntimeError: dictionary changed size during iteration
This error is showing up when reading the output of runs made with Juliet before it became a package
I fixed it by changing it like this:
out_new = dict()
                out_new['posterior_samples'] = dict()
                for pname in out['posterior_samples'].keys():
                    if 'sigma_w_rv' == pname[:10]:
                        instrument = pname.split('_')[-1]
                        out_new['posterior_samples']['sigma_w_'+instrument] = out['posterior_samples'][pname]

                for pname in out_new['posterior_samples'].keys():
                    out['posterior_samples'][pname] = out_new['posterior_samples'][pname]

It's just an ad-hoc solution though. I'll open a PR for this, just in case.
Cheers,

Hey @jvines --- thanks for this! I'll take a deeper look soon to this to include it.

Fixed in https://github.com/nespinoza/juliet/commit/2160e0d4f4bb4e818482e37e295847d599a4223d


Generator function stream does not wait for inner asynchronous loop to finish

Dart Programming Language Speciﬁcation version 1.9 reads:

Let u be the stream associated with the immediately enclosing asynchronous
for loop or generator function (9), if any. If another event e u of u occurs before
execution of s is complete, handling of e u must wait until s is complete.

The following code tests this statement for generator function:
import 'dart:async';

Stream gen1(List log) async* {
  await for (int k in new Stream.fromIterable([1, 2])) {
    log.add('s'+k.toString());
    if (log.length<=4) {
      yield k.toString();
    }
    log.add('f'+k.toString());
  }
}

Future test1() async {
  List log = [];
  await for (String s in gen1(log)) {
    log.add(s);
  }
  return log;
}

main() {
  test1().then((log) => print("$log"));
}

The test produce the following output:

[s1, 1, f1, s2, 2, f2]

The output shows that values '1' and '2' were handled in function test1() before the body of asynchronous for-in loop in function gen1() is finished.
Dart VM version: 1.12.2 (Wed Oct 21 07:10:52 2015) on "windows_x64"

The behavior is correct and matches the specification.
What that part of the specification is saying is that if another event of new Stream.fromIterable([1, 2]) occurs before the body of the await for loop is complete, it will be kept until the loop body completes. That is, event if the stream you are doing await for over spits out events as fast as possible, the body of the loop will only be for one at a time. This differs from a stream.forEach((e) async { ... }) call which will call the callback as soon as possible.
It does not say that the events output by the async* function surrounding the loop are delayed. They can be handled immediately after the yield is executed because the async* function actually does yield control at the yield statement.

I am confused with the sentence

Let u be the stream associated with the immediately enclosing asynchronous
for loop or generator function (9), if any.

Especially with the words 'enclosing' and 'if any'.
If you are correct, both of them are not right.
If there is single asynchronous for loop it should not be named as 'enclosed'
and remark 'if any' have no sense.

The text is in the section about await for loops, but it is talking about a loop surrounding the current one (if any). So, if you have an await-for loop inside another await-for loop, the outer loop must not continue until the inner one is done.
The general pattern in the spec is that any async operation inside a loop (await, await for, yield) states that a surrounding loop must pause until the body ends. If there is no async operation inside an await-for loop, then it's not required to pause between handling events.
You will find text similar to this in the section on await, and it means the same thing: that an enclosing async loop, if any, should not handle new events until the body is done.
(I can't find the same wording for yield or yield*, they just say that the function "may suspend").

Could you clarify the specification about the surrounding generator function instead of the loop?
I guess, that it should be the same:
if there is an await-for loop inside generator function, the stream related to generator function should not call event handlers till current iteration of the await-for loop finishes.

There is no restriction on the events of a generator function. The generator function's stream subscription can deliver events immediately after a yield, whether the yield is in the body of the loop or not.
The only restriction is on loops internally in a single async function - the body of the loop is not reentrant, so any events happening on the await-for stream must be saved (or the stream paused) until the body is done.

I believe that the specification should be updated with clarifications:

remove mentioning enclosing generator function
add restriction, that body of the loop is not reentrant, and all events must be handled sequentially one by one.

Should I report new issue or it is possible to reuse this one?

I don't know what Github protocol recommends, I have no problem with reusing this issue.
The restriction is already there, it just that it's implicit (async operations inside a loop states that the loop should be paused, instead of the loop stating that the body must be completed before the next event is processed).


Missing localization string for settings.dialogs.reboot.bye on future3

When rebooting the Pi from the web app, I noticed that the reboot dialog's title settings.dialogs.reboot.bye was a localization placeholder.
(My browser and OS are set to English locale.)

#2107 fixed this


[stable/external-dns] How to setup in AWS Route53 using preexisting IAM user secrets?

Is this a request for help?:
Hello guys,
I can see that PR #8587 changes extraEnv to be an array of objects. It makes external-dns be able to retrieve credentials from already existing secrets, avoiding duplication and more importantly credentials stored in helm overrides which might be checked into version control.
But, I can't see in docs how I should do this for config aws route53 using ExtraEnv to retrieve credentials from already existing secrets.  How should I do this?
Version of Helm and Kubernetes:
Helm - v2.13.1
Kubernetes - v1.11.9
Which chart:
estable/external-dns
What you expected to happen:
Setup in AWS Route53 using preexisting IAM user secrets.

Hello guys,
I figured out how to do.  It's very simple. But my doubt occurs because in docs din't have full description about this. The way like works with configs defined directly set up by  aws.secretKey and aws.accessKey in values.yaml and secrets defined by ValueFrom in ExtraEnvs are differents:
1 - When set  aws. secretKey and aws.accessKey this Chart stores the infos in an secret of the name provide by value of {{ external-dns.fullname }} and mount this secret in file of the name config in credentialsPath defined in values.yaml (pre external-dns 0.5.9 home dir should be /root/.aws), so AWS CLI is  loaded from a profile in the configuration file. But as mentioned, this approach genereted duplication and more importantly credentials stored in helm configs might be checked into version control.
2 - As documented by Amazon, environment variables provide another way to specify configuration options and credentials. Important note: By Precedence of options, It overrides any value loaded from a profile in the configuration file, that is, the values defined in aws. secretKey and aws.accessKey will be disregarded in favor AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the case defined both ways. . (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html)
So basically, when defined var envs AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY the AWS CLI is  loaded from a profile defined in the var envs. Below, an example of configuration setting var envs AWS_ACCESS_KEY_ID and  AWS_SECRET_ACCESS_KEY with an secret named existing-secret which conatins keys aws_access_key_id and aws_secret_access_key:
 - name: AWS_ACCESS_KEY_ID
   valueFrom:
     secretKeyRef:
       name: existing-secret
       key: aws_access_key_id

 - name: AWS_SECRET_ACCESS_KEY
   valueFrom:
     secretKeyRef:
       name: existing-secret
       key: aws_secret_access_key

PR #13336 submitted for improving docs.


[TACHYON-958] Create a worker-specific block master client & filesystem master client

https://tachyon.atlassian.net/browse/TACHYON-958
move worker-specific calls from BlockMasterClient to a new WorkerBlockMasterClient.
Do we need to move worker-specific calls from FileSystemMasterClient as well? I think there're only two calls.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/Tachyon-Pull-Request-Builder/4930/
Test PASSed.

How about move the other two calls ( addCheckpoint & getPinList ) out of FileSystemMasterClient into this new master client and rename it to WorkerMasterClient ?

@danielfree I think getPinList may be used in other places, but I think its a good idea to split it out.

@calvinjia for file system master client do you prefer to have a WorkerFileSystemMasterClient or merge it into single WorkerMasterClient?

@danielfree I think WorkerFileSystemMasterClient being separate would be cleaner.

@calvinjia Sure, I'll add more commits to this PR for it.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/Tachyon-Pull-Request-Builder/4946/
Test PASSed.

done

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/Tachyon-Pull-Request-Builder/4961/
Test PASSed.

please take a look @calvinjia

@danielfree Looks good overall, a few minor comments.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/Tachyon-Pull-Request-Builder/4996/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/Tachyon-Pull-Request-Builder/4997/
Test PASSed.

Thanks a lot for your review & comments @calvinjia  ! I've addressed them so PTAL.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/Tachyon-Pull-Request-Builder/5008/
Test PASSed.

LGTM