🐛  Crash

Description
In our last release (using CohesionKit 0.15.1) we have some new crashes:
crash_info_entry_0	 Swift/KeyPath.swift:1809: Fatal error: Unexpectedly found nil while unwrapping an Optional value
Crashed: com.cohesionkit.identitymap
0  libswiftCore.dylib             0x3a3c8 closure #1 in closure #1 in closure #1 in _assertionFailure(_:_:file:line:flags:) + 228
1  libswiftCore.dylib             0x3a2a0 closure #1 in closure #1 in _assertionFailure(_:_:file:line:flags:) + 332
2  libswiftCore.dylib             0x39c2c _assertionFailure(_:_:file:line:flags:) + 184
3  libswiftCore.dylib             0x34a7a4 specialized RawKeyPathComponent._projectMutableAddress<A, B>(_:from:to:isRoot:keepAlive:) + 1596
4  libswiftCore.dylib             0x1650bc WritableKeyPath._projectMutableAddress(from:) + 424
5  libswiftCore.dylib             0x168a6c swift_setAtWritableKeyPath + 420
6  Aircall                        0xa936dc closure #1 in EntityNode.observeChild<A>(_:for:) + 72 (EntityNode.swift:72)
7  Aircall                        0xa93b0c closure #1 in EntityNode.observeChild<A, B>(_:identity:update:) + 100 (EntityNode.swift:100)
8  Aircall                        0xa8bc3c partial apply for thunk for @escaping @callee_guaranteed (@in_guaranteed A) -> () +<PHONE_NUMBER> (<compiler-generated>:4341087292)
9  Aircall                        0xa8b770 closure #1 in Observable.value.didset + 32 (Observable.swift:32)
10 Aircall                        0xa8bb8c partial apply for closure #1 in Observable.value.didset +<PHONE_NUMBER> (<compiler-generated>:4341087116)
11 libswiftCore.dylib             0xdee70 Sequence.forEach(_:) + 756
12 Aircall                        0xa8b6c0 Observable.value.didset + 32 (Observable.swift:32)
13 Aircall                        0xa93b1c closure #1 in EntityNode.observeChild<A, B>(_:identity:update:) + 100 (EntityNode.swift:100)
14 Aircall                        0xa8bc3c partial apply for thunk for @escaping @callee_guaranteed (@in_guaranteed A) -> () +<PHONE_NUMBER> (<compiler-generated>:4341087292)
15 Aircall                        0xa8b770 closure #1 in Observable.value.didset + 32 (Observable.swift:32)
16 Aircall                        0xa8bb8c partial apply for closure #1 in Observable.value.didset +<PHONE_NUMBER> (<compiler-generated>:4341087116)
17 libswiftCore.dylib             0xdee70 Sequence.forEach(_:) + 756
18 Aircall                        0xa8b6c0 Observable.value.didset + 32 (Observable.swift:32)
19 Aircall                        0xa93b1c closure #1 in EntityNode.observeChild<A, B>(_:identity:update:) + 100 (EntityNode.swift:100)
20 Aircall                        0xa8bc3c partial apply for thunk for @escaping @callee_guaranteed (@in_guaranteed A) -> () +<PHONE_NUMBER> (<compiler-generated>:4341087292)
21 Aircall                        0xa8b770 closure #1 in Observable.value.didset + 32 (Observable.swift:32)
22 Aircall                        0xa8bb8c partial apply for closure #1 in Observable.value.didset +<PHONE_NUMBER> (<compiler-generated>:4341087116)
23 libswiftCore.dylib             0xdee70 Sequence.forEach(_:) + 756
24 Aircall                        0xa8b6c0 Observable.value.didset + 32 (Observable.swift:32)
25 Aircall                        0xa8bbf0 specialized Observable.value.setter + 30 (Observable.swift:30)
26 Aircall                        0xa933b0 EntityNode.updateEntity(_:modifiedAt:) + 55 (EntityNode.swift:55)
27 Aircall                        0xa871c4 EntityStore.nodeStore<A>(in:entity:modifiedAt:) + 164 (EntityStore.swift:164)
28 Aircall                        0xa94580 EntityStoreStoreVisitor.visit<A, B>(context:entity:) +<PHONE_NUMBER>
29 Aircall                        0xa94514 EntityStoreStoreVisitor.visit<A, B>(context:entity:) +<PHONE_NUMBER>
30 Aircall                        0xa8991c closure #1 in PartialIdentifiableKeyPath.init<A>(_:) +<PHONE_NUMBER>
31 Aircall                        0xa897b4 partial apply for closure #1 in PartialIdentifiableKeyPath.init<A>(_:) +<PHONE_NUMBER> (<compiler-generated>:4341077940)
32 Aircall                        0xa8757c EntityStore.nodeStore<A>(in:entity:modifiedAt:) + 188 (EntityStore.swift:188)
33 Aircall                        0xa94bac EntityStoreStoreVisitor.visit<A, B>(context:entities:) +<PHONE_NUMBER>
34 Aircall                        0xa947cc EntityStoreStoreVisitor.visit<A, B>(context:entities:) +<PHONE_NUMBER>
35 Aircall                        0xa8a174 closure #1 in PartialIdentifiableKeyPath.init<A>(_:) +<PHONE_NUMBER>
36 Aircall                        0xa89d20 partial apply for closure #1 in PartialIdentifiableKeyPath.init<A>(_:) +<PHONE_NUMBER>
37 Aircall                        0xa8757c EntityStore.nodeStore<A>(in:entity:modifiedAt:) + 188 (EntityStore.swift:188)
38 Aircall                        0xa879c4 EntityStore.storeAlias<A>(content:key:modifiedAt:) + 207 (EntityStore.swift:207)
39 Aircall                        0xa86c30 closure #1 in EntityStore.store<A>(entities:named:modifiedAt:) +<PHONE_NUMBER>
40 Aircall                        0xa891ec partial apply for closure #1 in EntityStore.store<A>(entities:named:modifiedAt:) +<PHONE_NUMBER>
41 Aircall                        0xa891b4 partial apply for closure #1 in EntityStore.store<A>(entities:named:modifiedAt:) +<PHONE_NUMBER>
42 Aircall                        0xa896a8 partial apply for closure #1 in EntityStore.transaction<A>(_:) + 216 (EntityStore.swift:216)
43 libswiftDispatch.dylib         0x1d4c partial apply for thunk for @callee_guaranteed () -> (@out A, @error @owned Error) + 28
44 libswiftDispatch.dylib         0x9ea0 thunk for @callee_guaranteed () -> (@out A, @error @owned Error)partial apply + 16
45 libswiftDispatch.dylib         0x1c80 closure #1 in closure #1 in OS_dispatch_queue._syncHelper<A>(fn:execute:rescue:) + 192
46 libswiftDispatch.dylib         0x1bb4 partial apply for thunk for @callee_guaranteed () -> () + 28
47 libswiftDispatch.dylib         0x1b8c thunk for @escaping @callee_guaranteed () -> () + 28
48 libdispatch.dylib              0x3dd4 _dispatch_client_callout + 20
49 libdispatch.dylib              0x132c4 _dispatch_lane_barrier_sync_invoke_and_complete + 56
50 libswiftDispatch.dylib         0x28a4 implicit closure #2 in implicit closure #1 in OS_dispatch_queue.sync<A>(flags:execute:) + 196
51 libswiftDispatch.dylib         0x2540 OS_dispatch_queue._syncHelper<A>(fn:execute:rescue:) + 404
52 libswiftDispatch.dylib         0x39cc OS_dispatch_queue.sync<A>(flags:execute:) + 332
53 Aircall                        0xa87b18 EntityStore.transaction<A>(_:) + 213 (EntityStore.swift:213)
54 Aircall                        0xa86bb0 EntityStore.store<A>(entities:named:modifiedAt:) +<PHONE_NUMBER>
55 Aircall                        0xa86b2c EntityStore.store<A>(entities:named:modifiedAt:) +<PHONE_NUMBER>
56 Aircall                        0x177cb4 closure #2 in LineRepository.downloadLines() + 156 (LineRepository.swift:156)
57 Aircall                        0x4ecc44 thunk for @escaping @callee_guaranteed (@guaranteed [LineInformation]) -> () +<PHONE_NUMBER> (<compiler-generated>:4335193156)
58 Combine                        0x1898c Publishers.HandleEvents.Inner.receive(_:) + 128
59 Combine                        0x18900 protocol witness for Subscriber.receive(_:) in conformance Publishers.HandleEvents<A>.Inner<A1> + 24
60 Combine                        0x1d040 Future.Conduit.fulfill(_:) + 1952
61 Combine                        0x1c870 Future.Conduit.offer(_:) + 340
62 Combine                        0x1c70c partial apply for closure #1 in Future.promise(_:) + 68
63 Combine                        0x22040 ConduitList.forEach(_:) + 276
64 Combine                        0x20cd8 Future.promise(_:) + 1312
65 Combine                        0x207ac partial apply for closure #1 in Future.init(_:) + 28
66 Aircall                        0x534d50 specialized closure #1 in closure #1 in static Future<>.async(_:) + 29 (Publisher+Async.swift:29)
67 libswift_Concurrency.dylib     0x4d764 swift::runJobInEstablishedExecutorContext(swift::Job*) + 436
68 libswift_Concurrency.dylib     0x4e9c8 swift_job_runImpl(swift::Job*, swift::ExecutorRef) + 72
69 libdispatch.dylib              0x15894 _dispatch_root_queue_drain + 392
70 libdispatch.dylib              0x1609c _dispatch_worker_thread2 + 156
71 libsystem_pthread.dylib        0x1ee4 _pthread_wqthread + 228
72 libsystem_pthread.dylib        0x1fc0 start_wqthread + 8

Could our ex but beloved staff engineer have a look to that 🙏
Reproduction steps
We don't know yet how to reproduce the problem

Screenshots/GIF
![DESCRIPTION](LINK.png)

Logs
No response
Framework version
0.15.1
Swift
No response
Xcode
No response

We have a   _ = entityStore.store(entities: [], named: .todo)
Is it possible that this empty list causes some issues?
How is working the store when the new entities has less elements than the old one? Should we use  entityStore.removeAlias(named:) before storing?

I don't see any reason for having an empty list causing the issue.
The error being "Unexpectedly found nil while unwrapping an Optional value", I'm wondering if it's not a force unwrap somewhere in the lib that's causing this issue. That being said it's unclear whether it's about the keypath itself or its value.
But I'd say it's probably about the keypath itself? So maybe the issue is a keypath that is added and removed on next update (in nestedEntitiesKeyPath), like with an enum or something like this. Maybe you could have a look to this?

The force unwrap is probably that one
private extension WritableKeyPath {
    func unwrapped<Wrapped>() -> WritableKeyPath<Root, Wrapped> where Value == Optional<Wrapped> {
        self.appending(path: \.self!)
    }
}

That mean the keyPath itself is nil but I can't explain that. We only added 2 nestedEntitiesKeyPaths and none of them is kind of "dynamic".
`extension CallMetadata: Aggregate {
public var nestedEntitiesKeyPaths: [PartialIdentifiableKeyPath] {
[
PartialIdentifiableKeyPath(.call)
]
}
extension UserAccount: Aggregate {
public var nestedEntitiesKeyPaths: [PartialIdentifiableKeyPath] {
[
PartialIdentifiableKeyPath(.user),
PartialIdentifiableKeyPath(.teams)
]
}
`

I reproduced and was able to add some break point. In the keyPath we have such things: \AliasContainer<Array<CallMetadata>>.content!.subscript(_: ) or \AliasContainer<Paginated<CallInformation, Int>>.content!.subscript(_: )
I think the issue is related to the last version where AliasContainer can be nullified.
I don't understand how we can have [PartialIdentifiableKeyPath<Self>(\.content)] while content is optional and nil now 🤔

If the content is nil it means you probably use removeAlias? If it's the case and you're inserting data back again just after I would say it might be a threading issue: some work in the lib is done asynchronously. Despite my effort it's not fully battled tested/bulletproof on this and would need to migrate to actor.
As a workaround maybe try to do each of your action asynchronously, so
_ = entityStore.store(alias: ) // or removeAlias()
DispathQueue.async { // or with Combine or Task.sleep()
  entityStore.store(alias: )
}

Let me know if it might solve or mitigate the issue 🙈

@Delphine-Garcia I made a bugfix on fix/removeAlias branch. Let me know if it fixes your issue.

No more crash reported. Thank you so much!


Version compatible with Mac OS 10.9.5?

Hi.  I would like to use your program but would rather not upgrade my operating system to Yosemite.  Do you have a version that works with 10.9.5?

It seems like the minimum OS X version supported by Electron is 10.9. I don't see any reason why Sabaki shouldn't work on 10.9.5... Can you describe what's happening when you start Sabaki?

When I try to start it, I get the following error message:  "Sabaki Cannot be opened because of a problem.  Check with the developer to make sure Sabaki works with this version of OS X.  You may need to reinstall the application.  Be sure to install any available updates for the application and OS X.  This report will be sent to Apple automatically."  What follows are the details of the problem outlined below the error message itself:
Process:         Sabaki [2247]
Path:            /Users/USER/Documents/Sabaki.app/Contents/MacOS/Sabaki
Identifier:      com.electron.sabaki
Version:         ???
Code Type:       X86-64 (Native)
Parent Process:  launchd [405]
Responsible:     Sabaki [2247]
User ID:         502
Date/Time:       2016-06-09 13:11:20.717 -0400
OS Version:      Mac OS X 10.9.5 (13F34)
Report Version:  11
Anonymous UUID:  18B3FE0F-EF4D-4A5A-F083-1AA41799A736
Sleep/Wake UUID: 2C558F23-D05D-4951-8A2E-2EBECAD46287
Crashed Thread:  0
Exception Type:  EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000002, 0x0000000000000000
Application Specific Information:
dyld: launch, loading dependent libraries
Dyld Error Message:
Library not loaded: @rpath/Electron Framework.framework/Electron Framework
Referenced from: /Users/USER/Documents/Sabaki.app/Contents/MacOS/Sabaki
Reason: no suitable image found.  Did find:
/Users/jputnam/Documents/Sabaki.app/Contents/MacOS/../Frameworks/Electron Framework.framework/Electron Framework: file too short
Binary Images:
0x7fff6f997000 -     0x7fff6f9ca817  dyld (239.4) <7AD43B9B-5CEA-3C7E-9836-A06909F9CA56> /usr/lib/dyld
Model: MacBookPro5,5, BootROM MBP55.00AC.B03, 2 processors, Intel Core 2 Duo, 2.53 GHz, 4 GB, SMC 1.47f2
Graphics: NVIDIA GeForce 9400M, NVIDIA GeForce 9400M, PCI, 256 MB
Memory Module: BANK 0/DIMM0, 2 GB, DDR3, 1067 MHz, 0x80CE, 0x4D34373142353637334548312D4346382020
Memory Module: BANK 1/DIMM0, 2 GB, DDR3, 1067 MHz, 0x80CE, 0x4D34373142353637334548312D4346382020
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x8D), Broadcom BCM43xx 1.0 (<IP_ADDRESS>.22)
Bluetooth: Version 4.2.7f4 15112, 3 services, 23 devices, 1 incoming serial ports
Network Service: AirPort, AirPort, en1
Serial ATA Device: Hitachi HTS545025B9SA02, 250.06 GB
Serial ATA Device: MATSHITADVD-R   UJ-868
USB Device: Iomega HDD
USB Device: Built-in iSight
USB Device: Internal Memory Card Reader
USB Device: Apple Internal Keyboard / Trackpad
USB Device: IR Receiver
USB Device: BRCM2046 Hub
USB Device: Bluetooth USB Host Controller
Thunderbolt Bus:

This seems like an Electron bug. Can you try downloading the latest version of Electron and see if it runs on your OS X? If it doesn't, it's definitely an Electron bug and I suggest you report the issue there. Since I don't have OS X 10.9, I'm afraid I can't do anything substantial here.

Oh, another thing: Electron only supports 64-bit OS X.

Now I'm getting the following message: "Unable to find Electron app at /Applications/Sabaki.app.  Cannot find module '/Applications/Sabaki.app'"  When I try to run Sabaki from within Electron.

Please try to run Electron on its own without Sabaki.

That works just fine.  Electron, the app, is fully functional.  I can get it to launch but trying to run Sabaki USING Electron does not work.

Yes, I understand your problem. I was just making sure that Electron itself works on your machine which it apparently does. Now we know that there is a problem with Sabaki. But it doesn't change the fact that I can't test Sabaki on Mac OS X 10.9.

                                                                                Do you have older versions of the app I could try out to see if they work?                                                                                Sent from my BlackBerry 10 smartphone.                                                                                From: Yichuan ShenSent: Friday, June 10, 2016 4:16 PMTo: yishn/SabakiReply To: yishn/SabakiCc: JGPutnam; AuthorSubject: Re: [yishn/Sabaki] Version compatible with Mac OS 10.9.5? (#101)Yes, I understand your problem. I was just making sure that Electron itself works on your machine which it apparently does. Now we know that there is a problem with Sabaki. But it doesn't change the fact that I can't test Sabaki on Mac OS X 10.9.

—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or mute the thread.

You can find all releases of Sabaki in the releases page of this repository.

FYI
I got the similar error when I tried sabaki-v0.19.0-mac-x64.7z on macOS 10.12.1.
However, I cloned the github repository and built by npm run build:osx, then the app works fine.

@y-ich I'm not sure I understood what you did to get the app to run correctly.  Can you walk me through the steps I should take to make Sabaki work on my mac?  Am I understanding you correctly that you got Sabaki to run after doing something to it when it wouldn't run previously?

I think what @y-ich did was to build Sabaki on his own, using the build instructions.

@y-ich   I tried what you did and it worked!  So pumped to use it!  Thanks a bunch!

@y-ich and @JGPutnam, can you both try downloading the prebuilt archives of v0.19.1 and see if you still got the error message? I've built them using macOS Sierra.

I confirmed that the app in sabaki-v0.19.1-mac-x64.7z starts up fine on my environment (macOS 10.12.1).
@yiishn-san, Thank you for your efforts!
And @JGPutnam-san, I am glad to hear your happy voice^^

@yishn I tried downloading the file but it seems the "structure" of the archive is damaged.  I couldn't expand it after downloading it in zipped format

That's weird... @y-ich seems to have unzipped it just fine, and there are no problems on my end either... Maybe your download was corrupted; did you try it again?

I've also confirmed that the new build (sabaki-v0.19.1-mac-x64.7z) runs on my Mac OS (10.9.5 [Mavericks])  I think I was having an issue with Stuffit expander.  In other words, Stuffit wouldn't expand the file for some reason.  I found another app that was able to do it and the program ran just fine.


LayoutList overflow hidden

Repro Steps

save a high number of layouts
open my layouts
scroll to last layout

expected behaviour
all layouts details are visible in "my layouts"
actual behaviour
last layout details is hidden
Notes

try changing height to "auto" for
see usage of List in examples/ui-controls/List.tsx ListWithinFlexLayout

Can no longer reproduce, I think it must've been fixed upstream already.

I think this has been resolved upstream already as the solution for auto on the List height is implemented, and I can no longer reproduce the issue.


Error installing Nonprofit accelerator in Environment

Hi,
I am the Project Manager of a Nonprofit Organization that works with PowerApps. When I try to install the Core Nonprofit Acceletaror "An unexpected error occurred, please refresh and try again".
I have created 2 tickets to Microsoft and they referred me here.
Please find here the logs.
logs.zip
Thanks,
Betania Calderon
<EMAIL_ADDRESS>

Any luck on this?  Curious on their support for this and vibrancy of community.


test/trace_api_test.c: fix gcc error on -Werror=strict-prototypes

Carved out of #19271.
This is a fixup of #19096.
I wonder why the
test/trace_api_test.c:70:13: error: function declaration isn't a prototype [-Werror=strict-prototypes]
   70 | static void put_trace_output()
      |             ^~~~~~~~~~~~~~~~
test/trace_api_test.c:78:12: error: function declaration isn't a prototype [-Werror=strict-prototypes]
   78 | static int test_trace_channel()
      |            ^~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors

got unnoticed, which occurs when configuring with --strict-warnings -std=gnu90.

got unnoticed, which occurs when configuring with --strict-warnings -std=gnu90.

I do not think we have CI build with such combination of options. IMO it is impossible to have all the possible combination of build time options tested via CI.

I understand that we cannot test all combinations.
Yet I'd suggest strengthening one of the existing CI configurations to include -std=gnu90 where --strict-warnings is already present, or vice versa.

This pull request is ready to merge

Merged - thanks @t8m and @tmshort for the swift approvals.

Yet the more general CI issue remains:

I understand that we cannot test all combinations. Yet I'd suggest strengthening one of the existing CI configurations to include -std=gnu90 where --strict-warnings is already present, or vice versa.

I meanwhile checked in .github/workflows/ci.yml that there are several jobs that include --strict-warnings but none that includes std=gnu90. How about including the latter to one of them? Any preferences on which one?

I meanwhile checked in .github/workflows/ci.yml that there are several jobs that include --strict-warnings but none that includes std=gnu90. How about including the latter to one of them? Any preferences on which one?

I'd add it as additional build to the compiler zoo CI runs.

Closing as this PR is merged.

I understand that we cannot test all combinations. Yet I'd suggest strengthening one of the existing CI configurations to include -std=gnu90 where --strict-warnings is already present, or vice versa.

I meanwhile checked in .github/workflows/ci.yml that there are several jobs that include --strict-warnings but none that includes std=gnu90. How about including the latter to one of them? Any preferences on which one?

I'd add it as additional build to the compiler zoo CI runs.

Hmm, but -std=gnu90 should not be specific to any compiler (and version),
and adding another build boils down to a new combination and extra run time.
Why not simply add -std=gnu90 to any of the jobs in ci.yml that already has --strict-warnings?


Json parsing issue when using openai.Completion.acreate(stream=True)

See the last line, looks like the stream iterator might be breaking up parts where it shouldn't? Unless it's an API issue. cc @ddeville
Traceback (most recent call last):
  File "/Users/nuno/Library/Caches/pypoetry/virtualenvs/platform-api-30l3Kv3P-py3.10/lib/python3.10/site-packages/openai/api_requestor.py", line 670, in _interpret_response_line
    data = json.loads(rbody)
  File<EMAIL_ADDRESS>line 346, in loads
    return _default_decoder.decode(s)
  File<EMAIL_ADDRESS>line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File<EMAIL_ADDRESS>line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 191 (char 190)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/nuno/dev/concat/server-py/concat/conversation.py", line 87, in handle_conversation
    await run
  File "/Users/nuno/dev/concat/server-py/concat/agent/openai/openai.py", line 122, in __call__
    msg, _ = await asyncio.gather(
  File "/Users/nuno/dev/concat/server-py/concat/tools/multi_prompt.py", line 87, in __call__
    return await self.call(prompt["prompt"], examples, input, ctx)
  File "/Users/nuno/dev/concat/server-py/concat/tools/sql/__init__.py", line 59, in call
    async for key, msg in ctx.stream_multi_message_async(
  File "/Users/nuno/dev/concat/server-py/concat/context/context.py", line 419, in stream_multi_message_async
    async for text in iterator:
  File "/Users/nuno/dev/concat/server-py/concat/utils/openai.py", line 37, in openai_completion_stream
    async for chunk in cast(
  File "/Users/nuno/Library/Caches/pypoetry/virtualenvs/platform-api-30l3Kv3P-py3.10/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 230, in <genexpr>
    return (
  File "/Users/nuno/Library/Caches/pypoetry/virtualenvs/platform-api-30l3Kv3P-py3.10/lib/python3.10/site-packages/openai/api_requestor.py", line 320, in wrap_resp
    async for r in resp:
  File "/Users/nuno/Library/Caches/pypoetry/virtualenvs/platform-api-30l3Kv3P-py3.10/lib/python3.10/site-packages/openai/api_requestor.py", line 635, in <genexpr>
    self._interpret_response_line(
  File "/Users/nuno/Library/Caches/pypoetry/virtualenvs/platform-api-30l3Kv3P-py3.10/lib/python3.10/site-packages/openai/api_requestor.py", line 672, in _interpret_response_line
    raise error.APIError(
openai.error.APIError: HTTP code 200 from API ({"id": "cmpl-6YxjgLI6W6XwRj8hqhDZ6aCFz19Hp", "object": "text_completion", "created":<PHONE_NUMBER>, "choices": [{"text": "sche", "index": 0, "logprobs": null, "finish_reason": null}], "model": ")

Huh, that's weird. Any chance you have a command to recreate it?

No, this was only once, out of many similar requests. The same request (or
as close to same as I could) repeated after didn’t trigger this error.
On Sun, Jan 15 2023 at 4:31 pm, hallacy @.***> wrote:

Huh, that's weird. Any chance you have a command to recreate it?
—
Reply to this email directly, view it on GitHub
https://github.com/openai/openai-python/issues/184#issuecomment-1383193806,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAN4RSOMLYQSZF5QSWDSN3WSQQ4LANCNFSM6AAAAAAT34ET6M
.
You are receiving this because you authored the thread.Message ID:
@.***>

I just got this exact error and seems like an isolated case. I noticed it was taking long to get the response, like 2min, and eventually it raised that exception. This is the python code I used with v0.26.1:
response = openai.Image.create(
  prompt='a kitten splashing on an aquarium full of candy´,
  n=10,
  size="1024x1024",
  response_format='b64_json'
)

I'm seeing this issue in PROD with bulk async requests @athyuttamre .  I validated the JSON and tried a number of other ways to avoid it but still get " Error: Unterminated string starting at: line 9 column 18 (char 225)."
my async retry function is:
def aretry_json_with_exponential_backoff(
initial_delay: float = random.random(),
exponential_base: float = 2,
jitter: bool = True,
max_retries: int = 10,
errors: tuple = (openai.error.RateLimitError, openai.error.ServiceUnavailableError, json.JSONDecodeError,openai.error.APIError),
):
"""Retry a function with exponential backoff."""
def decorator(func):
async def wrapper(*args, **kwargs):
# Initialize variables
num_retries = 0
initial_delay = random.random()
delay = initial_delay
        # Loop until a successful response or max_retries is hit or an exception is raised
        while True:
            try:
                await asyncio.sleep(initial_delay * 0.10)
                response = await func(*args, **kwargs)
                response = response["choices"][0]["message"]["content"]
                return json.loads(response)

            # Retry on specified errors
            except errors as err_msg:
                # Increment retries
                num_retries += 1
                print(f"Recalling OpenAI API #{num_retries}. Error: {err_msg}. iDelay: {initial_delay:.3f}. Delay: {delay:.3f}\n\n{str(args)}\n\n{str(kwargs)}" )
                # Check if max retries has been reached
                if num_retries > max_retries:
                    raise Exception(
                        f"Maximum number of retries ({max_retries}) exceeded."
                    )

                # Increment the delay
                delay *= exponential_base * (1 + jitter * random.random())

                # Sleep for the delay
                await asyncio.sleep(delay)

            # Raise exceptions for any errors not specified
            except Exception as err_msg:
                raise err_msg
    return wrapper
return decorator

@aretry_json_with_exponential_backoff()
async def achats_json_with_backoff(**kwargs):
return await openai.ChatCompletion.acreate(**kwargs)

any thoughts @hallacy @ellisonbg @athyuttamre  ? recurring error using openai 0.27.6

Im having the same issue as well with Azure OpenAI. It's a case by case for very specific payloads.


AxonServer Connector only uses first result when using multi-event-upcaster

When using an upcaster that upcasts a single entry into multiple entries, the AxonServer connector ignores all but the first upcast result.

Resolved in #1264, hence closing this issue.


Create first minimal Community Version

This version must include following:

[x] Cross Platform Solution (Should work both in Windows && Linux)
[x] Inetract with multiple COM devices
[x] Save configuration and restore during project loading
[x] Allow to send any binary data
[x] Add menu with some basic info and navigation to github repo for issue posting
[x] Add a status bar with serial devices activity
[x] Add a splash screen with short Info about Wissance
[x] Add an About window
[x] Set Window resize mode
[x] Add Clear Button handle
[x] Add Support Page

done


Created contracts don't have valid addresses

Downstream components prefer for contract creation output to not be an encoded CallResult, so contract creation should return something that makes these components happy instead of whatever this is.

This is going to be resolved with the next network upgrade


Using image tag with slashes very slow when parsing.

For example, this image reference: gcr.io/qingyangc-sandbox/jibtestimage:askfjl/waegawvw/aweg
Took 3 min to parse.

https://github.com/GoogleContainerTools/jib/tree/fix-imagereference2

@chanseokoh I have this PR #357 that's been sitting around for a while - it doesn't solve the issue but it might help write a better regex - feel free to use/not use it/close it in favor of a new solution


Added missing dependency specs2-junit integration

When running mvn test getting
[ERROR] /Users/.../src/test/scala/samples/specs.scala:18: error: not found: type JUnitRunner
[ERROR] @RunWith(classOf[JUnitRunner])
[ERROR]                  ^
[ERROR] one error found

Added -
    <dependency>
      <groupId>org.specs2</groupId>
      <artifactId>specs2-core_${scala.compat.version}</artifactId>
      <version>2.4.16</version>
      <scope>test</scope>
    </dependency>

thx

Sorry, but I think you need to revert this PR.
It's not in the published archetype so I thought it's missing but it's in the code.
Now you have this dependency twice

And also I didn't close the  tag. Please revert this pull request ASAP.


Fix null-pointer-exception

I think this should fix #166

Hi @entropitor,
Thanks for the PR! It turns out that I'm currently working on a few fixes and that I already came up with something along the lines of what you've done.
I'll let you know as soon as the fixes are published in order for you to try it out and let me know if the PR is still needed.

Hey, have a similar Exception in our app. See below. I think this would fix our problem (hard to reproduce), too.
null is not an object (evaluating 'this._listRef.scrollToOffset')

Stack:
- value@index.android.bundle:205:2194
- index.android.bundle:994:5820
- index.android.bundle:23:1366
- o@index.android.bundle:23:571
- callTimers@index.android.bundle:23:2778
- value@index.android.bundle:18:2975
- index.android.bundle:18:280
- value@index.android.bundle:18:2426
- callFunctionReturnFlushedQueue@index.android.bundle:18:252

Releasing this as 3.2.4 would be great. So that your bigger chances could be tested in a new minor version?? Thanks both for creating and improving this module 👍

@entropitor @jerolimov Hey guys!
The issue should be fixed in the "soon-to-be-merged" loop branch. Can you confirm? To try it:
"react-native-snap-carousel": "https://github.com/archriss/react-native-snap-carousel#18acafa"


Issue#178 file format

Pull Request Test Coverage Report for Build 700

35 of 41 (85.37%)  changed or added relevant lines in 5 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.04%) to 83.042%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

contour/plot.py
2
3
66.67%

contour/views.py
13
15
86.67%

contour/validators.py
17
20
85.0%

Files with Coverage Reduction
New Missed Lines
%

contour/views.py
1
77.58%

Totals

Change from base Build 691:
0.04%

Covered Lines:
1518

Relevant Lines:
1828

💛  - Coveralls


Cannot find module "valid-url"

Hi!
When trying to use this module I get the following error in the logs.
[01.08.2023 11:22.56.846] [ERROR] Error: Cannot find module 'valid-url'
Require stack:
- /opt/magic_mirror/modules/MMM-CalendarWeek/node_helper.js
- /opt/magic_mirror/js/app.js
- /opt/magic_mirror/serveronly/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1075:15)
    at Module._resolveFilename (/opt/magic_mirror/node_modules/module-alias/index.js:49:29)
    at Module._load (node:internal/modules/cjs/loader:920:27)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at Object.<anonymous> (/opt/magic_mirror/modules/MMM-CalendarWeek/node_helper.js:9:16)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at loadModule (/opt/magic_mirror/js/app.js:184:19)
    at loadNextModule (/opt/magic_mirror/js/app.js:220:6)
    at loadNextModule (/opt/magic_mirror/js/app.js:222:6)
    at loadNextModule (/opt/magic_mirror/js/app.js:222:6) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/opt/magic_mirror/modules/MMM-CalendarWeek/node_helper.js',
    '/opt/magic_mirror/js/app.js',
    '/opt/magic_mirror/serveronly/index.js'
  ]
}

I am using the following configuration:
                {
                        module: "MMM-CalendarWeek",
                        position: "bottom_bar", // This can be any of the regions. Best results in bottom region.
                        header: "Calendars",
                        config: {
                                calendarTitle: "false",
                                maximumEntries: "20",
                                maximumNumberOfDays: "6",
                                displaySymbol: "true",
                                showEndDate: "true",
                                allowDuplicate: "false",
                                maxTitleLength: "25",
                                hideEmptyDays: "false",
                                wrapEvents: "false",
                                tableClass: "small",
                                calendars: [
                                        {
                                                url: "http://www.calendarlabs.com/templates/ical/US-Holidays.ics",
                                                symbol: "calendar"
                                        }
                                ]
                        }
                },

Same issue. MM runs no problem with CalendarWeek commented out, but I'd really rather use it!

I am running my MagicMirror in docker (karsten13 image) with server only. When I start the MagicMirror container with the configuration provided in the first post the container just goes into a bootloop. Therefore I cannot connect to the console of the image and run npm install valid-url in the modules catalogue.
I have not had problems like this with any other modules and I am running over 6 different modules so it is probably an error within this module. Also, I am running the latest version of MagicMirror (2.24.0)
Maybe @akaep can try run npm install valid-url if he is using a "normal" installation instead of docker?

I found a way to run npm install valid-url inside the contianer. See below image.

Getting the following errors now.
[02.08.2023 12:25.24.173] [ERROR] Whoops! There was an uncaught exception...
[02.08.2023 12:25.24.180] [ERROR] Error: Cannot find module 'request'
Require stack:
- /opt/magic_mirror/modules/MMM-CalendarWeek/vendor/ical.js/node-ical.js
- /opt/magic_mirror/modules/MMM-CalendarWeek/vendor/ical.js/index.js
- /opt/magic_mirror/modules/MMM-CalendarWeek/calendarfetcher.js
- /opt/magic_mirror/modules/MMM-CalendarWeek/node_helper.js
- /opt/magic_mirror/js/app.js
- /opt/magic_mirror/serveronly/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1075:15)
    at Module._resolveFilename (/opt/magic_mirror/node_modules/module-alias/index.js:49:29)
    at Module._load (node:internal/modules/cjs/loader:920:27)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at Object.<anonymous> (/opt/magic_mirror/modules/MMM-CalendarWeek/vendor/ical.js/node-ical.js:2:15)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at Object.<anonymous> (/opt/magic_mirror/modules/MMM-CalendarWeek/vendor/ical.js/index.js:3:12)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/opt/magic_mirror/modules/MMM-CalendarWeek/vendor/ical.js/node-ical.js',
    '/opt/magic_mirror/modules/MMM-CalendarWeek/vendor/ical.js/index.js',
    '/opt/magic_mirror/modules/MMM-CalendarWeek/calendarfetcher.js',
    '/opt/magic_mirror/modules/MMM-CalendarWeek/node_helper.js',
    '/opt/magic_mirror/js/app.js',
    '/opt/magic_mirror/serveronly/index.js'
  ]
}

I found this fork by @ktoll2
https://github.com/heskja/MMM-CalendarWeek/compare/master...ktoll2:MMM-CalendarWeek:master
Apparently missing dependencies. I started to use his fork instead. Now everything is working as expected.
Need to add dependencies in package.json to the master to get this plugin to work again. I will not close this comment in case anyone else have the same problem.

I know,
I have my own fork aswell since 6 months ago  :)
https://github.com/dberntsson/MMM-CalendarWeek
And PR not accepted. So unfortunately this project isn't that well taken care of...

I switched to your fork. Cheers!
I think you should add your own fork on the 3rd party module so people can find a working version of the module.

Great
Should be visible here https://github.com/heskja/MMM-CalendarWeek/forks

Great Should be visible here https://github.com/heskja/MMM-CalendarWeek/forks

I tried using your version and I get the same error as the original one.  I deleted the mmm-calendarweek folder from within modules and then cloned yours but it didn't change anything.  Is there anything else I need to do?

Great Should be visible here https://github.com/heskja/MMM-CalendarWeek/forks

I tried using your version and I get the same error as the original one. I deleted the mmm-calendarweek folder from within modules and then cloned yours but it didn't change anything. Is there anything else I need to do?

Same here, were you able to get it to work?  I feel like there's a step I'm missing.....

@OrangeGitHubber @kelemvor33
You need to enter the module folder and install it.
So after you clone the branch you do:
cd MMM-CalendarWeek
npm install


HuggingFace checkpoint

Hi! Congrats on the clean RetNet code. I'm experimenting with the torchscale codebase and happened to find your repo with a link to a checkpoint in HF.
I noticed it is now gone, do you have plans to reupload it? or further details on the training dataset/hyperparameters?
Thanks in advance :)

Hi there! Sorry for new year delay. Last time we uploaded the model, it was quite under-trained and the performance was not as good as expected. We are still training the model to make sure it's safe to deploy. Will also ping here when the model goes live again.

Thanks for the reply! I'm working with the RetNet architecture in a downstream task, so a base pretrained model would definitely help out.


Error During Installation

I want to thank you for the great work and I believe this tool is very useful, however I got some errors during installation. See below.
inspect.go:27: x.Buffer undefined (type *bufio.Scanner has no field or method Buffer)
inspect.go:28: x.Buffer undefined (type *bufio.Scanner has no field or method Buffer)

Which version of golang you are using?

This is a problem that happens on golang < 1.6 because the method Buffer don't exists.
If you need to build it, you can just remove those two lines and it should work in the vast majority of cases, since this Buffer call is there to manage big files.


external_subnet_id should not have a default empty value as it is required

with TF 0.14, the template fails with:

Error: Error in function call

  on compute.tf line 147, in data "ibm_is_subnet" "default_route_subnet":
 147:   identifier = element(local.secondary_subnets, length(local.secondary_subnets) - 1)
    |----------------
    | local.secondary_subnets is empty list of string

Call to function "element" failed: cannot use element function with a negative
index.

in
locals {
  secondary_subnets = compact(list(var.cluster_subnet_id, var.internal_subnet_id, var.external_subnet_id))
  external_floating_ip = var.external_subnet_id == "" ? false : var.bigip_external_floating_ip
}

cluster_subnet_id, interval_subnet_id are described as Optional. It seems that external_subnet_id is required, but is assigned a default value of "" thus will fail if you don't specify it explicitely

Please read the README.md file completely.
It clearly states that 'management_subnet_id' AND 'external_subnet_id' are 'required'. The list was empty because all three 'cluster_subnet_id', 'internal_subnet_id', and 'external_subnet_id' (required) were left blank. There are numerous error paths that could be reproduced if you do not properly read the README.md and apply appropriately.
Note that 1NIC deployments are NOT supported with this template. That is not an optional for this module. If you want a 1NIC deployment, please define your own template. The tmos-cloudinit images support 1NIC deployment, so you can do it, we just don't recommend or support it in this module.

I also got the issue while provisioning from the IBM Cloud catalog (https://cloud.ibm.com/catalog/content/ibmcloud_schematics_bigip_multinic_declared-1.0-d33f1544-e938-478a-b0dd-d883370f08d0-global) as the field is not listed as required because of the default value.
Here is the UI:

I hear you about reading the README but notice that only two parameters have default values so I would expect all the default are good defaults and save me some reading -- and if more values are required, they should not have a default so I would be forced to fill them here
Once I configured a value of course it worked.

Indeed...
I'm not that concerned with the catalog tile as we are seeing customization trumping a single tile workflow from our customers. Right now the scarcity of tile resources imposed by IBM is the real limitation to having 'just works' deployments. There are simply too many desired use cases by our customers to model in IaC HCL. Even terraform would tell you create multiple modules. One tile to rule them all is naive. We can remove the 'helper text' for those who don't read it. I'm still looking for the public facing documentation on the tile UX features.
There is not default possible without some discovery of VPC subnets.  There are no default subnets with a VPC. We can not reference VPC subnets by name as this is not unique in IBM VPC cloud and the terraform provider from IBM decided not to resolve the resource by name.
On the Schematics side, we can certainly add ibm_is_subnet data provider lookups on the management_subnet_id and external_subnet_id so the error would be much more obvious.

We've added the external_subnet_id data check in the module such that the error message will reflect that there is not subnet defined.


How test the soapBluster library?

I managed to build a soapBluster project, but I don’t know how to test and use it. I created a java gradle project in IntelliJ IDEA, and added dependences according to the your recommendation to the build.gradle file. Also some changes were added to function main of Main.java class:
import com.google.iot.m2m.LocalEndpointManager;
import com.google.iot.m2m.Client;
import com.google.iot.m2m.Transaction;
import com.google.iot.m2m.Message;

public class Main {

    public static void main(String[] args)
    {
        System.out.println("Hello coap!");

        LocalEndpointManager manager = new LocalEndpointManager();

        Client client = new Client(manager, "coap://coap.me/test");

        Transaction transaction = client.newRequestBuilder().send();

        Message response = transaction.getResponse();

        System.out.println("Got response: " + response);
    }

}

However, apart from compilation errors, I received nothing:
Information:java: Errors occurred while compiling module 'testCoapBluster.testCoap.main'
Information:javac 1.8.0_201 was used to compile java sources
Information:2/28/19 9:42 AM - Compilation completed with 10 errors and 0 warnings in 1 s 635 ms
/home/michael_k/IdeaProjects/testCoap/src/main/java/Main.java
Error:(1, 26) java: package com.google.iot.m2m does not exist
Error:(2, 26) java: package com.google.iot.m2m does not exist
Error:(3, 26) java: package com.google.iot.m2m does not exist
Error:(4, 26) java: package com.google.iot.m2m does not exist
Error:(12, 9) java: cannot find symbol
  symbol:   class LocalEndpointManager
  location: class Main
Error:(12, 44) java: cannot find symbol
  symbol:   class LocalEndpointManager
  location: class Main
Error:(14, 9) java: cannot find symbol
  symbol:   class Client
  location: class Main
Error:(14, 29) java: cannot find symbol
  symbol:   class Client
  location: class Main
Error:(16, 9) java: cannot find symbol
  symbol:   class Transaction
  location: class Main
Error:(18, 9) java: cannot find symbol
  symbol:   class Message
  location: class Main

What is the matter and how to fix errors?
File build.gradle:
plugins {
    id 'java'
}

group 'testCoapBluster'
version '1.0-SNAPSHOT'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.12'
    compile 'com.google.iot.coap:coap:0.01.00'
}

Bug was fixed and the app began to breathe. Here are the results:
Hello coap!
Got response: <CONTENT IN ACK MID:53455 TOK:3888 2.05 RADDR:"/<IP_ADDRESS>:5683" OPTS:[ETag:2c6c6e79aaadc775, Content-Format:text/plain;charset=utf-8] TEXT:"welcome to the ETSI plugtest! last change: 2019-0...">

But you need to make changes to the texts of the samples, as shown below:
import com.google.iot.coap.LocalEndpointManager;
import com.google.iot.coap.Client;
import com.google.iot.coap.Transaction;
import com.google.iot.coap.Message;

public class Main
{
    public static void main(String[] args)
    {
        System.out.println("Hello coap!");
        LocalEndpointManager manager = new LocalEndpointManager();
        Client client = new Client(manager, "coap://coap.me/test");
        Transaction transaction = client.newRequestBuilder().send();
        try
        {
            Message response = transaction.getResponse();
            System.out.println("Got response: " + response);
        }
        catch(Exception exception)
        {
            System.out.println("Exception: " + exception);
        }
    }
}

Yes, I'll go ahead and update the README.


Support Gnome 47

Tested and things are working as expected on Gnome 47

@Krypion17 can you merge and get this released please ? This is the in my list of used extensions not yet updated for 47 and I am sure for a lot of others too :)


US VPS / Cloud providers not blocked

Does anyone know of any working VPS or Cloud providers that aren't blocked on Netflix?

I think Kamatera may work.

I set mine up yesterday it worked for a while then it stopped working

My bad it works on my computer but not my iPhone app

This is well documented. The iOS app may bypass DNS all together.

Oh okay 😊


fix: examples moved repo

some of the urls were stale

Thanks for helping to update this!

Thanks for helping to update this!

Thanks for creating such an awesome repo :)


Add room logic for day-only and night-only

Added CanChangeTime in LogicFunctions.cs and put that as a requirement into day-only and night-only exits.

Forgot to mention: In the room "Lake Hylia Lakebed Temple Entrance", the time changes as well, but I didn't put it into logic, because then you would have to wait for day/night while staying behind the boulder at the LBT entrance.


联邦学习图像分类-加载cluster数据集卡住

Issue Type
Bug
Source
binary
Secretflow Version
0.7.18b0
OS Platform and Distribution
ubuntu
Python version
3.8.0
Bazel version
No response
GCC/Compiler version
No response
What happend and What you expected to happen.
加载集群上的数据集的时候卡住

Reproduction code to reproduce the issue.
cluster_config ={
    'parties': {
        'alice': {
            # replace with alice's real address.
            'address': '<IP_ADDRESS>:8275',
            'listen_addr': '<IP_ADDRESS>:8275'
        },
        'bob': {
            # replace with bob's real address.
            'address': '<IP_ADDRESS>:8275',
            'listen_addr': '<IP_ADDRESS>:8275'
        },
    },
    'self_party': 'alice'
}
sf.init(address='<IP_ADDRESS>:6379', cluster_config=cluster_config)
alice,bob=sf.PYU("alice"),sf.PYU("bob")
train_data = load({bob: '/home/mi/PycharmProjects/SecretflowDemo/mnist/x_train2.npy',
alice: '/Users/xiaomi/PycharmProjects/secretflow-demo/src/data/mnist/node/x_train1.npy'})

2023-01-17 16:45:54,404	INFO worker.py:1352 -- Connecting to existing Ray cluster at address: <IP_ADDRESS>:6379...
[2023-01-17 16:45:54,416 I 61418 2404405] global_state_accessor.cc:357: This node has an IP address of <IP_ADDRESS>, while we can not find the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.
2023-01-17 16:45:54,416	INFO worker.py:1538 -- Connected to Ray cluster.
2023-01-17 16:45:54 INFO fed.barriers [alice] --  RecverProxy was successfully created.
(RecverProxyActor pid=102377, ip=<IP_ADDRESS>) INFO:fed.barriers:Successfully start Grpc service without credentials.
2023-01-17 16:45:55 INFO fed.barriers [alice] --  SendProxy was successfully created.

这两个数据集本身大吗？ 如果本地加载大概要多久呀

mnist数据集，我给拆开了，放在两台机器上。本地加载很快就能完成

可以贴一下源代码吗？我们可能需要本地重现一下了

mnist_classify_pytorch.txt

我在本地配置好的集群没有复现出问题，怀疑是启动时候sf.init的问题。
可以参考例子改一下试试，如果用cluster_config模式启动，每一方都需要有这个配置来进行sf.init
@zhouaihui

@zhangxingmeng 我按照您提供的方式init集群，仍然没有解决数据集加载卡住的问题
2023-01-28 10:19:30,605	INFO worker.py:1352 -- Connecting to existing Ray cluster at address: <IP_ADDRESS>:6379...
[2023-01-28 10:19:30,636 I 1228 10875848] global_state_accessor.cc:357: This node has an IP address of <IP_ADDRESS>, while we can not find the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.
2023-01-28 10:19:30,637	INFO worker.py:1538 -- Connected to Ray cluster.
2023-01-28 10:19:31 INFO fed.barriers [alice] --  RecverProxy was successfully created.
(RecverProxyActor pid=185753, ip=<IP_ADDRESS>) INFO:fed.barriers:Successfully start Grpc service without credentials.
2023-01-28 10:19:31 INFO fed.barriers [alice] --  SendProxy was successfully created.

hi, @cs1317 , 每一方启动的时候，address是使用各自的ray地址，这里可以确认一下吗？
比如alice是
sf.init(address='<IP_ADDRESS>:6379', cluster_config=cluster_config)

bob则是
sf.init(address='<IP_ADDRESS>:6379', cluster_config=cluster_config)

@zhouaihui  不是主节点的ip+port吗？

@cs1317 , 你使用的是多个ray集群模式，每一个机构都是单独的ray集群，主节点指的是每个机构的ray的主节点。

@zhouaihui
设置为多集群模式，仍然会卡住
alice的init:
cluster_config ={
'parties': {
'alice': {
# replace with alice's real address.
'address': '<IP_ADDRESS>:8275',
'listen_addr': '<IP_ADDRESS>:8275'
},
'bob': {
# replace with bob's real address.
'address': '<IP_ADDRESS>:8275',
'listen_addr': '<IP_ADDRESS>:8275'
}
},
'self_party': 'alice'
}
sf.init(address='<IP_ADDRESS>:6379', cluster_config=cluster_config)
bob的init:
cluster_config ={
'parties': {
'alice': {
# replace with alice's real address.
'address': '<IP_ADDRESS>:8275',
'listen_addr': '<IP_ADDRESS>:8275'
},
'bob': {
# replace with bob's real address.
'address': '<IP_ADDRESS>:8275',
'listen_addr': '<IP_ADDRESS>:8275'
}
},
'self_party': 'bob'
}
sf.init(address='<IP_ADDRESS>:6379', cluster_config=cluster_config)
日志信息为：
/Users/xiaomi/opt/anaconda3/envs/secretflow/bin/python /Users/xiaomi/PycharmProjects/secretflow-demo/mnist/mnist_classify_pytorch.py
2023-01-28 17:34:53,786	INFO worker.py:1352 -- Connecting to existing Ray cluster at address: <IP_ADDRESS>:6379...
[2023-01-28 17:34:53,799 I 26575 11500908] global_state_accessor.cc:357: This node has an IP address of <IP_ADDRESS>, while we can not find the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.
2023-01-28 17:34:53,801	INFO worker.py:1538 -- Connected to Ray cluster.
2023-01-28 17:34:54 INFO fed.barriers [alice] --  RecverProxy was successfully created.
(RecverProxyActor pid=26641) INFO:fed.barriers:Successfully start Grpc service without credentials.
2023-01-28 17:34:55 INFO fed.barriers [alice] --  SendProxy was successfully created.
2023-01-28 17:34:55 INFO fed.cleanup [alice] --  Start check sending thread.
2023-01-28 17:34:55 INFO fed.cleanup [alice] --  Start check sending monitor thread.
2023-01-28 17:35:57,987	WARNING worker.py:1851 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffffd9e2e3e2630c5a4d5856fafe01000000 Worker ID: 2e583d8c58d9eda93baba3b0382510e36372e7dc6975db48b3336411 Node ID: 77c6c344750838facda45a06f53c5c20bf67495e8092051f029966da Worker IP address: <IP_ADDRESS> Worker port: 10004 Worker PID: 26648 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
2023-01-28 17:35:57 WARNING fed.cleanup [alice] --  Failed to send ObjectRef(f91b78d7db9a6593d9e2e3e2630c5a4d5856fafe0100000001000000) with error: The actor died unexpectedly before finishing this task.
class_name: SendProxyActor
actor_id: d9e2e3e2630c5a4d5856fafe01000000
pid: 26648
name: SendProxyActor
namespace: 4311bac8-f3d1-49f0-823c-dde5e2cacad6
ip: <IP_ADDRESS>
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
(SendProxyActor pid=26648) *** SIGSEGV received at time=1674898557 ***
(SendProxyActor pid=26648) PC: @      0x372ffffffff  (unknown)  0x0
(SendProxyActor pid=26648) [2023-01-28 17:35:57,964 E 26648 11501434] logging.cc:361: *** SIGSEGV received at time=1674898557 ***
(SendProxyActor pid=26648) [2023-01-28 17:35:57,964 E 26648 11501434] logging.cc:361: PC: @      0x372ffffffff  (unknown)  0x0
(SendProxyActor pid=26648) Fatal Python error: Segmentation fault
(SendProxyActor pid=26648)
(SendProxyActor pid=26648) Stack (most recent call first):
(SendProxyActor pid=26648)   File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/collections/init.py", line 891 in getitem
(SendProxyActor pid=26648)   File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 627 in dump
(SendProxyActor pid=26648)   File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 73 in dumps
如果改为单集群的主从节点，则在执行load()时报错
train_data = load({alice: '/Users/xiaomi/PycharmProjects/secretflow-demo/mnist/x_train1.npy',bob: '/home/mi/PycharmProjects/SecretflowDemo/mnist/x_train2.npy'},allow_pickle=True)
错误日志:
/Users/xiaomi/opt/anaconda3/envs/secretflow/bin/python /Users/xiaomi/PycharmProjects/secretflow-demo/mnist/mnist_classify_pytorch.py
2023-01-28 16:57:30,632        INFO worker.py:1352 -- Connecting to existing Ray cluster at address: <IP_ADDRESS>:6379...
[2023-01-28 16:57:30,659 I 9219 11423135] global_state_accessor.cc:357: This node has an IP address of <IP_ADDRESS>, while we can not find the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.
2023-01-28 16:57:30,690        INFO worker.py:1538 -- Connected to Ray cluster.
init device
ConvNet create
(SPURuntime pid=191303, ip=<IP_ADDRESS>) 2023-01-28 16:57:31.957 [error] [server.cpp:BRPC:969] Fail to listen <IP_ADDRESS>:8275
(SPURuntime pid=191303, ip=<IP_ADDRESS>) 2023-01-28 16:57:31,963        ERROR worker.py:763 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::SPURuntime.init() (pid=191303, ip=<IP_ADDRESS>, repr=<secretflow.device.device.spu.SPURuntime object at 0x7f2146d702e0>)
(SPURuntime pid=191303, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/secretflow/device/device/spu.py", line 278, in init
(SPURuntime pid=191303, ip=<IP_ADDRESS>)     self.link = spu_link.create_brpc(desc, rank)
(SPURuntime pid=191303, ip=<IP_ADDRESS>) RuntimeError: what:
(SPURuntime pid=191303, ip=<IP_ADDRESS>)         [external/yacl/yacl/link/transport/channel_brpc.cc:126] brpc server failed start
(SPURuntime pid=191303, ip=<IP_ADDRESS>) stacktrace:
(SPURuntime pid=191303, ip=<IP_ADDRESS>) #0 yacl::link::FactoryBrpc::CreateContext()+0x7f2157f360c1
(SPURuntime pid=191303, ip=<IP_ADDRESS>) #1 pybind11::cpp_function::initialize<>()::{lambda()#3}::_FUN()+0x7f2156d7f510
(SPURuntime pid=191303, ip=<IP_ADDRESS>) #2 pybind11::cpp_function::dispatcher()+0x7f2156d69c5b
(SPURuntime pid=191303, ip=<IP_ADDRESS>) #3 PyCFunction_Call+0x5587927b8c96
(SPURuntime pid=9359) 2023-01-28 16:57:38.527 [error] [server.cpp:BRPC:969] Fail to listen <IP_ADDRESS>:8275
(SPURuntime pid=9359) 2023-01-28 16:57:38,668        ERROR worker.py:763 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::SPURuntime.init() (pid=9359, ip=<IP_ADDRESS>, repr=<secretflow.device.device.spu.SPURuntime object at 0x7fbaf1d20c40>)
(SPURuntime pid=9359)   File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/device/device/spu.py", line 278, in init
(SPURuntime pid=9359)     self.link = spu_link.create_brpc(desc, rank)
(SPURuntime pid=9359) RuntimeError: what:
(SPURuntime pid=9359)         [external/yacl/yacl/link/transport/channel_brpc.cc:126] brpc server failed start
(SPURuntime pid=9359) stacktrace:
(SPURuntime pid=9359) #0 yacl::link::ReceiverLoopBrpc::Start()+0x1c43cf199
(SPURuntime pid=9359) #1 yacl::link::FactoryBrpc::CreateContext()+0x1c43cad59
(SPURuntime pid=9359) #2 pybind11::cpp_function::initialize<>()::{lambda()#1}::__invoke()+0x1c279fe7d
(SPURuntime pid=9359) #3 pybind11::cpp_function::dispatcher()+0x1c2783561
(SPURuntime pid=9359) #4 cfunction_call_varargs+0x10eaa7758
(SPURuntime pid=9359) #5 _PyObject_MakeTpCall+0x10eaa6b91
(SPURuntime pid=9359) #6 call_function+0x10ebf77e1
(SPURuntime pid=9359) #7 _PyEval_EvalFrameDefault+0x10ebf53b0
(SPURuntime pid=9359) #8 _PyEval_EvalCodeWithName+0x10ebe78d7
(SPURuntime pid=9359) #9 _PyFunction_Vectorcall+0x10eaa811e
(SPURuntime pid=9359) #10 PyVectorcall_Call+0x10eaa73ab
(SPURuntime pid=9359) #11 _PyEval_EvalFrameDefault+0x10ebf5734
(SPURuntime pid=9359) #12 _PyEval_EvalCodeWithName+0x10ebe78d7
(SPURuntime pid=9359) #13 _PyFunction_Vectorcall+0x10eaa811e
(SPURuntime pid=9359) #14 PyVectorcall_Call+0x10eaa73ab
(SPURuntime pid=9359) #15 _PyEval_EvalFrameDefault+0x10ebf5734
2023-01-28 17:07:39,211        WARNING worker.py:2307 -- Local object store memory usage:
(global lru) capacity:<PHONE_NUMBER>
(global lru) used: 0%
(global lru) num objects: 0
(global lru) num evictions: 0
(global lru) bytes evicted: 0
Traceback (most recent call last):
File "/Users/xiaomi/PycharmProjects/secretflow-demo/mnist/mnist_classify_pytorch.py", line 131, in
train_data = load({alice: '/Users/xiaomi/PycharmProjects/secretflow-demo/mnist/x_train1.npy',bob: '/home/mi/PycharmProjects/SecretflowDemo/mnist/x_train2.npy'},allow_pickle=True)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/data/ndarray.py", line 274, in load
parts[device] = device(_get_item)(idx, data)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/device/device/pyu.py", line 127, in wrapper
sfd.remote(self._run)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/ray/remote_function.py", line 226, in remote
return func_cls._remote(args=args, kwargs=kwargs, **updated_options)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/distributed/primitive.py", line 82, in _remote
args, kwargs = _resolve_args(*args, **kwargs)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/distributed/primitive.py", line 72, in _resolve_args
actual_vals = ray.get(list(refs.values()))
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/ray/_private/worker.py", line 2311, in get
raise value
ray.exceptions.ObjectFetchTimedOutError: Failed to retrieve object 725328266daa6b20ffffffffffffffffffffffff1400000002000000. The ObjectRef was created at: (task call)
/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/device/device/pyu.py:wrapper:127
/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/data/ndarray.py:load:253
/Users/xiaomi/PycharmProjects/secretflow-demo/mnist/mnist_classify_pytorch.py::131
Fetch for object 725328266daa6b20ffffffffffffffffffffffff1400000002000000 timed out because no locations were found for the object. This may indicate a system-level bug.
Process finished with exit code 1

@cs1317

如果使用多ray集群模式，麻烦同时发一下alice和bob的日志。从alice的日志看，无法访通bob的地址<IP_ADDRESS>:8275。你能确认两边启动的时间是差不多吗？
如果使用仿真模式，则你的配置可能需要按照单个ray集群修改为正确的ip。从报错看，你在<IP_ADDRESS>上启动了SPU，但是监听地址写的是<IP_ADDRESS>

如果只是测试，建议先使用仿真模式进行实验，该模式更方便快速上手和验证。参见 https://www.secretflow.org.cn/docs/secretflow/zh_CN/getting_started/deployment.html#simulation
注意，仿真和生产两种模式不能混用。

@zhouaihui 这是我全部的实现过程，麻烦帮忙看看有啥问题没
secretflow+pytorch联邦学习.txt

@zhouaihui 这是我全部的实现过程，麻烦帮忙看看有啥问题没 secretflow+pytorch联邦学习.txt

hi， @cs1317 , 请问报的什么错误？麻烦把报错日志发一下

2023-01-30 14:52:42,500	INFO worker.py:1352 -- Connecting to existing Ray cluster at address: <IP_ADDRESS>:6379...
[2023-01-30 14:52:42,652 I 23823 14919238] global_state_accessor.cc:357: This node has an IP address of <IP_ADDRESS>, while we can not find the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.
2023-01-30 14:52:42,663	INFO worker.py:1538 -- Connected to Ray cluster.
init device
ConvNet create
(SPURuntime pid=210909, ip=<IP_ADDRESS>) 2023-01-30 14:52:43.786 [error] [input_messenger.cpp:BRPC:97] Fail to parse response from <IP_ADDRESS>:6379 by baidu_std at client-side
(SPURuntime pid=210909, ip=<IP_ADDRESS>) 2023-01-30 14:52:44.793 [error] [input_messenger.cpp:BRPC:97] Fail to parse response from <IP_ADDRESS>:6379 by baidu_std at client-side
(SPURuntime pid=210909, ip=<IP_ADDRESS>) 2023-01-30 14:52:45.795 [error] [input_messenger.cpp:BRPC:97] Fail to parse response from <IP_ADDRESS>:6379 by baidu_std at client-side
(SPURuntime pid=210909, ip=<IP_ADDRESS>) 2023-01-30 14:52:46.797 [error] [input_messenger.cpp:BRPC:97] Fail to parse response from <IP_ADDRESS>:6379 by baidu_std at client-side
(SPURuntime pid=210909, ip=<IP_ADDRESS>) 2023-01-30 14:52:47.799 [error] [input_messenger.cpp:BRPC:97] Fail to parse response from <IP_ADDRESS>:6379 by baidu_std at client-side
(SPURuntime pid=210909, ip=<IP_ADDRESS>) 2023-01-30 14:52:49.802 [error] [input_messenger.cpp:BRPC:97] Fail to parse response from <IP_ADDRESS>:6379 by baidu_std at client-side
(SPURuntime pid=210909, ip=<IP_ADDRESS>) 2023-01-30 14:52:53.805 [error] [input_messenger.cpp:BRPC:97] Fail to parse response from <IP_ADDRESS>:6379 by baidu_std at client-side
(SPURuntime pid=210909, ip=<IP_ADDRESS>) 2023-01-30 14:52:53,810	ERROR worker.py:763 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::SPURuntime.init() (pid=210909, ip=<IP_ADDRESS>, repr=<secretflow.device.device.spu.SPURuntime object at 0x7fafea092340>)
(SPURuntime pid=210909, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/secretflow/device/device/spu.py", line 278, in init
(SPURuntime pid=210909, ip=<IP_ADDRESS>)     self.link = spu_link.create_brpc(desc, rank)
(SPURuntime pid=210909, ip=<IP_ADDRESS>) RuntimeError: what:
(SPURuntime pid=210909, ip=<IP_ADDRESS>) 	[external/yacl/yacl/link/context.cc:140] connect to mesh failed, failed to setup connection to rank=0
(SPURuntime pid=210909, ip=<IP_ADDRESS>) stacktrace:
(SPURuntime pid=210909, ip=<IP_ADDRESS>) #0 pybind11::cpp_function::initialize<>()::{lambda()#3}::_FUN()+0x7faffa0a8519
(SPURuntime pid=210909, ip=<IP_ADDRESS>) #1 pybind11::cpp_function::dispatcher()+0x7faffa092c5b
(SPURuntime pid=210909, ip=<IP_ADDRESS>) #2 PyCFunction_Call+0x55966b458c96
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:52:55,619	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'", grpc_status:12, created_time:"2023-01-30T14:52:55.618817379+08:00"}"
(raylet, ip=<IP_ADDRESS>) >
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:52:55,619	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {created_time:"2023-01-30T14:52:55.618782299+08:00", grpc_status:12, grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'"}"
(raylet, ip=<IP_ADDRESS>) >
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:52:57,626	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {created_time:"2023-01-30T14:52:57.625901099+08:00", grpc_status:12, grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'"}"
(raylet, ip=<IP_ADDRESS>) >
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:52:57,626	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {created_time:"2023-01-30T14:52:57.625940541+08:00", grpc_status:12, grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'"}"
(raylet, ip=<IP_ADDRESS>) >
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:52:59,631	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'", grpc_status:12, created_time:"2023-01-30T14:52:59.630513193+08:00"}"
(raylet, ip=<IP_ADDRESS>) >
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:52:59,631	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {created_time:"2023-01-30T14:52:59.630550038+08:00", grpc_status:12, grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'"}"
(raylet, ip=<IP_ADDRESS>) >
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:53:01,635	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {created_time:"2023-01-30T14:53:01.635162374+08:00", grpc_status:12, grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'"}"
(raylet, ip=<IP_ADDRESS>) >
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:53:01,635	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {created_time:"2023-01-30T14:53:01.635205153+08:00", grpc_status:12, grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'"}"
(raylet, ip=<IP_ADDRESS>) >
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:53:03,638	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'", grpc_status:12, created_time:"2023-01-30T14:53:03.638130819+08:00"}"
(raylet, ip=<IP_ADDRESS>) >
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:53:03,640	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {created_time:"2023-01-30T14:53:03.639708351+08:00", grpc_status:12, grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'"}"
(raylet, ip=<IP_ADDRESS>) >
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:53:05,642	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {created_time:"2023-01-30T14:53:05.641977422+08:00", grpc_status:12, grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'"}"
(raylet, ip=<IP_ADDRESS>) >
(raylet, ip=<IP_ADDRESS>) 2023-01-30 14:53:05,644	ERROR utils.py:1421 -- Internal KV Get failed
(raylet, ip=<IP_ADDRESS>) Traceback (most recent call last):
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/utils.py", line 1408, in internal_kv_get_with_retry
(raylet, ip=<IP_ADDRESS>)     result = gcs_client.internal_kv_get(key, namespace)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 179, in wrapper
(raylet, ip=<IP_ADDRESS>)     return f(self, *args, **kwargs)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/ray/_private/gcs_utils.py", line 271, in internal_kv_get
(raylet, ip=<IP_ADDRESS>)     reply = self._kv_stub.InternalKVGet(req, timeout=timeout)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
(raylet, ip=<IP_ADDRESS>)     return _end_unary_response_blocking(state, call, False, None)
(raylet, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
(raylet, ip=<IP_ADDRESS>)     raise _InactiveRpcError(state)
(raylet, ip=<IP_ADDRESS>) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
(raylet, ip=<IP_ADDRESS>) 	status = StatusCode.UNIMPLEMENTED
(raylet, ip=<IP_ADDRESS>) 	details = "[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'" (raylet, ip=<IP_ADDRESS>) 	debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:6379 {created_time:"2023-01-30T14:53:05.644096737+08:00", grpc_status:12, grpc_message:"[<IP_ADDRESS>:6379][E1002]Fail to find method on /ray.rpc.InternalKVGcsService/InternalKVGet'"}"
(raylet, ip=<IP_ADDRESS>) >
(SPURuntime pid=23991) 2023-01-30 14:53:05,871	ERROR worker.py:763 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::SPURuntime.init() (pid=23991, ip=<IP_ADDRESS>, repr=<secretflow.device.device.spu.SPURuntime object at 0x7fb00a850b20>)
(SPURuntime pid=23991)   File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/device/device/spu.py", line 278, in init
(SPURuntime pid=23991)     self.link = spu_link.create_brpc(desc, rank)
(SPURuntime pid=23991) RuntimeError: what:
(SPURuntime pid=23991) 	[external/yacl/yacl/link/context.cc:141] connect to mesh failed, failed to setup connection to rank=1
(SPURuntime pid=23991) stacktrace:
(SPURuntime pid=23991) #0 yacl::link::Context::ConnectToMesh()+0x1c43a270c
(SPURuntime pid=23991) #1 pybind11::cpp_function::initialize<>()::{lambda()#1}::__invoke()+0x1c2761e86
(SPURuntime pid=23991) #2 pybind11::cpp_function::dispatcher()+0x1c2745561
(SPURuntime pid=23991) #3 cfunction_call_varargs+0x10e860758
(SPURuntime pid=23991) #4 _PyObject_MakeTpCall+0x10e85fb91
(SPURuntime pid=23991) #5 call_function+0x10e9b07e1
(SPURuntime pid=23991) #6 _PyEval_EvalFrameDefault+0x10e9ae3b0
(SPURuntime pid=23991) #7 _PyEval_EvalCodeWithName+0x10e9a08d7
(SPURuntime pid=23991) #8 _PyFunction_Vectorcall+0x10e86111e
(SPURuntime pid=23991) #9 PyVectorcall_Call+0x10e8603ab
(SPURuntime pid=23991) #10 _PyEval_EvalFrameDefault+0x10e9ae734
(SPURuntime pid=23991) #11 _PyEval_EvalCodeWithName+0x10e9a08d7
(SPURuntime pid=23991) #12 _PyFunction_Vectorcall+0x10e86111e
(SPURuntime pid=23991) #13 PyVectorcall_Call+0x10e8603ab
(SPURuntime pid=23991) #14 _PyEval_EvalFrameDefault+0x10e9ae734
(SPURuntime pid=23991) #15 _PyEval_EvalCodeWithName+0x10e9a08d7
2023-01-30 15:03:09,370	WARNING worker.py:2307 -- Local object store memory usage:
(global lru) capacity:<PHONE_NUMBER>
(global lru) used: 0%
(global lru) num objects: 0
(global lru) num evictions: 0
(global lru) bytes evicted: 0
Traceback (most recent call last):
File "/Users/xiaomi/PycharmProjects/secretflow-demo/mnist/mnist_classify_pytorch.py", line 119, in
train_data = load({alice: '/Users/xiaomi/PycharmProjects/secretflow-demo/mnist/x_train1.npy',bob: '/home/mi/PycharmProjects/SecretflowDemo/mnist/x_train2.npy'},allow_pickle=True)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/data/ndarray.py", line 274, in load
parts[device] = device(_get_item)(idx, data)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/device/device/pyu.py", line 127, in wrapper
sfd.remote(self._run)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/ray/remote_function.py", line 226, in remote
return func_cls._remote(args=args, kwargs=kwargs, **updated_options)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/distributed/primitive.py", line 82, in _remote
args, kwargs = _resolve_args(*args, **kwargs)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/distributed/primitive.py", line 72, in _resolve_args
actual_vals = ray.get(list(refs.values()))
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/ray/_private/worker.py", line 2311, in get
raise value
ray.exceptions.ObjectFetchTimedOutError: Failed to retrieve object d496f28b52ef01d5ffffffffffffffffffffffff1300000002000000. The ObjectRef was created at: (task call)
/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/device/device/pyu.py:wrapper:127
/Users/xiaomi/opt/anaconda3/envs/secretflow/lib/python3.8/site-packages/secretflow/data/ndarray.py:load:253
/Users/xiaomi/PycharmProjects/secretflow-demo/mnist/mnist_classify_pytorch.py::119
Fetch for object d496f28b52ef01d5ffffffffffffffffffffffff1300000002000000 timed out because no locations were found for the object. This may indicate a system-level bug.

2023-01-31 14:12:38,998	INFO worker.py:1352 -- Connecting to existing Ray cluster at address: <IP_ADDRESS>:6379...
[2023-01-31 14:12:39,011 I 70765 15816459] global_state_accessor.cc:357: This node has an IP address of <IP_ADDRESS>, while we can not find the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.
2023-01-31 14:12:39,039	INFO worker.py:1538 -- Connected to Ray cluster.
init device
ConvNet create
(SPURuntime pid=219152, ip=<IP_ADDRESS>) 2023-01-31 14:12:40,130	ERROR worker.py:763 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::SPURuntime.init() (pid=219152, ip=<IP_ADDRESS>, repr=<secretflow.device.device.spu.SPURuntime object at 0x7f10213e92e0>)
(SPURuntime pid=219152, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/secretflow/device/device/spu.py", line 278, in init
(SPURuntime pid=219152, ip=<IP_ADDRESS>)     self.link = spu_link.create_brpc(desc, rank)
(SPURuntime pid=219152, ip=<IP_ADDRESS>) RuntimeError: what:
(SPURuntime pid=219152, ip=<IP_ADDRESS>) 	[external/yacl/yacl/link/transport/channel_brpc.cc:126] brpc server failed start
(SPURuntime pid=219152, ip=<IP_ADDRESS>) stacktrace:
(SPURuntime pid=219152, ip=<IP_ADDRESS>) #0 yacl::link::FactoryBrpc::CreateContext()+0x7f10325b90c1
(SPURuntime pid=219152, ip=<IP_ADDRESS>) #1 pybind11::cpp_function::initialize<>()::{lambda()#3}::_FUN()+0x7f1031402510
(SPURuntime pid=219152, ip=<IP_ADDRESS>) #2 pybind11::cpp_function::dispatcher()+0x7f10313ecc5b
(SPURuntime pid=219152, ip=<IP_ADDRESS>) #3 PyCFunction_Call+0x561814e43c96
(SPURuntime pid=219152, ip=<IP_ADDRESS>) 2023-01-31 14:12:40.126 [error] [server.cpp:BRPC:969] Fail to listen <IP_ADDRESS>:8275
(SPURuntime pid=219153, ip=<IP_ADDRESS>) 2023-01-31 14:12:50,129	ERROR worker.py:763 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::SPURuntime.init() (pid=219153, ip=<IP_ADDRESS>, repr=<secretflow.device.device.spu.SPURuntime object at 0x7fa2a3eea370>)
(SPURuntime pid=219153, ip=<IP_ADDRESS>)   File "/home/mi/anaconda3/envs/sf/lib/python3.8/site-packages/secretflow/device/device/spu.py", line 278, in init
(SPURuntime pid=219153, ip=<IP_ADDRESS>)     self.link = spu_link.create_brpc(desc, rank)
(SPURuntime pid=219153, ip=<IP_ADDRESS>) RuntimeError: what:
(SPURuntime pid=219153, ip=<IP_ADDRESS>) 	[external/yacl/yacl/link/context.cc:140] connect to mesh failed, failed to setup connection to rank=0
(SPURuntime pid=219153, ip=<IP_ADDRESS>) stacktrace:
(SPURuntime pid=219153, ip=<IP_ADDRESS>) #0 pybind11::cpp_function::initialize<>()::{lambda()#3}::_FUN()+0x7fa2b5640519
(SPURuntime pid=219153, ip=<IP_ADDRESS>) #1 pybind11::cpp_function::dispatcher()+0x7fa2b562ac5b
(SPURuntime pid=219153, ip=<IP_ADDRESS>) #2 PyCFunction_Call+0x560858cc8c9

train_data = load({alice: '/Users/xiaomi/PycharmProjects/secretflow-demo/mnist/x_train1.npy',bob: '/home/mi/PycharmProjects/SecretflowDemo/mnist/x_train2.npy'},allow_pickle=True)
print('train_data load')

logs.zip

已成功解决数据集加载问题，问题出现的原因是：
我主节点用了mac系统，从节点用了ubuntu系统，在各位大佬的帮助下，查出原因，把主从节点都换成ubuntu就好了

resolved
Close


Point of query origin

For security reasons, I must limit what data is being served to the ANB desktop client. Is there a way to see if a query originated from the ANB desktop client or the web client? I thought it might be in the request header somewhere but I couldn't find anything in the header or documentation about the header.
(Sorry to create so many tickets at the same time.)

Hello, I'm afraid that information isn't currently available on the SPI.
Though technically you could implement a request modifier extension point on the i2 Analyze server to extract the User-Agent header and forward it, I would first examine the requirements here. Any request header information relating to the User-Agent is not verifiable, e.g. I can make a request from a browser and set the User-Agent to toaster.
So the User-Agent shouldn't be depended upon for a security concern, but it's okay for optimizations and tailoring an experience. Can you clarify the requirement and the intention of it?
For security concerns, it's more appropriate to make decisions based on the permissions of the User themselves, rather than the client they are using. This information is available on the SPI: https://i2group.github.io/analyze-connect/content/example-connectors/user-configuration.html (when the server is configured to send these headers to the connector).

Thank you! Yes, we are already using the user name to filter documents. Unfortunately, I can't go into details about the security concerns but your guidance is already very helpful. Thank you!

Hi @chriiisiscoding  are you happy for this to be closed now?

Yes. Thank you for your help!


windows: error: Recipe _ could not be run because just could not find the shell: program not found

Hey, I wonder why this behaves like that? Related https://github.com/casey/just/discussions/2187
C:\Users\buk\WebstormProjects\temp>cat justfile
# default target
_:
    @just --list

C:\Users\buk\WebstormProjects\temp>just --list
Available recipes:

C:\Users\buk\WebstormProjects\temp>just
error: Recipe `_` could not be run because just could not find the shell: program not found

C:\Users\buk\WebstormProjects\temp>just --version
just 1.29.0

Solution is here: https://github.com/casey/just/issues/1517#issuecomment-1489102234


Make Extract::suffixExists protected

Currently the suffixExists method on the Extract class is private, it would be useful if it was protected, so you could override default behavior and provide an alternate data source

This would be useful; we've got a use case where we need to validate the TLD, but also have a limited list of additional extensions we want to allow for some local domains.

Release in 1.2.7 :v:


fix(Header): Update icons

Make icons larger, rotate the selector dropdown icon, and make sure they're centered.

Preview: https://virtual-assistant-pr-virtual-assistant-169.surge.sh

Admittedly, my window is small, but we probably want to avoid letting the UI get in this state:

I also noticed that the logo in the header is loading after the rest of the chatbot everytime I switch between different display modes. Do you happen to see this behavior? Is it avoidable?

I'm going to move this to draft for right now - I wonder if the new layout will fix the one spacing issue (if not, I'll look at it after that merges). I'll look into the brand issue too - I suspect it's because we have classes based on dark mode/display mode now. There's probably a better way to do that.

I'm pushing a change to the title component - I'm making it so it can conditionally render based on displayMode. It should fix the logo issue you were seeing, but I'm totally open to feedback on the API or whether we want to do something else.