[Bug]: [homebridge-eufy-security] This plugin slows down Homebridge. The write handler for the characteristic 'Selected RTP Stream Configuration' on the accessory 'Office E464' didn't respond at all!. Please check that you properly call the callback! See https://homebridge.io/w/JtMGR for more info.

What happened?
Describe The Bug:
I am receiving an error when trying to open camera streams for the Eufy cameras.
To Reproduce:
Attempt to open a camera stream in Apple HomeKit.
Expected behavior:
Steam would work consistently every time I open a stream.
Device Type
Indoor Camera (Supported)
Plugin Version
v2.0.x (Supported)
HomeBridge Version
v1.4.x (Supported)
NodeJS Version
v16 (Supported)
Operating System
Ubuntu 18.04+ (Supported)
Relevant log output
[7/2/2022, 3:47:11 PM] [homebridge-eufy-security] This plugin slows down Homebridge. The write handler for the characteristic 'Selected RTP Stream Configuration' on the accessory 'Office E464' didn't respond at all!. Please check that you properly call the callback! See https://homebridge.io/w/JtMGR for more info.

Hi @metalcated
Can you share a few more things?

your config json (without the credentials)
enable debug messages (instructions here) and some more log messages around the stream start event (~1-2 seconds before and up to 20 seconds after)

{
    "platforms": [
        {
            "name": "Config",
            "port": 8581,
            "platform": "config",
            "auth": "form",
            "tempUnits": "c",
            "lang": "auto",
            "theme": "dark-mode"
        },
        {
            "enableCamera": true,
            "pollingIntervalMinutes": 30,
            "ignoreDevices": [
                ""
            ],
            "hkAway": 0,
            "username"<EMAIL_ADDRESS>            "platform": "EufySecurity",
            "hkHome": 1,
            "hkNight": 3,
            "ignoreStations": [
                ""
            ],
            "hkOff": 63,
            "enableDetailedLogging": 0,
            "password": "password",
            "country": "US",
            "_bridge": {
                "username": "MAC:ADDRESS",
                "port": 59101
            },
            "cameras": [
                {
                    "serialNumber": "SN",
                    "enableCamera": true,
                    "enableButton": true,
                    "motionButton": true,
                    "rtsp": true,
                    "unbridge": true,
                    "forcerefreshsnap": false,
                    "useCachedLocalLivestream": false,
                    "videoConfig": {
                        "audio": true
                    }
                },
                {
                    "serialNumber": "SN",
                    "enableCamera": true,
                    "enableButton": true,
                    "motionButton": true,
                    "rtsp": true,
                    "unbridge": true,
                    "forcerefreshsnap": false,
                    "useCachedLocalLivestream": false,
                    "videoConfig": {
                        "audio": true
                    }
                },
                {
                    "serialNumber": "SN",
                    "enableCamera": true,
                    "enableButton": true,
                    "motionButton": true,
                    "rtsp": true,
                    "unbridge": true,
                    "forcerefreshsnap": false,
                    "useCachedLocalLivestream": false,
                    "videoConfig": {
                        "audio": true
                    }
                },
                {
                    "serialNumber": "SN",
                    "enableCamera": true,
                    "enableButton": true,
                    "motionButton": true,
                    "rtsp": false,
                    "unbridge": true,
                    "forcerefreshsnap": false,
                    "useCachedLocalLivestream": false,
                    "videoConfig": {
                        "audio": true
                    }
                },
                {
                    "serialNumber": "SN",
                    "enableCamera": true,
                    "enableButton": true,
                    "motionButton": true,
                    "rtsp": true,
                    "unbridge": true,
                    "forcerefreshsnap": false,
                    "useCachedLocalLivestream": false,
                    "videoConfig": {
                        "audio": true
                    }
                },
                {
                    "serialNumber": "SN",
                    "enableCamera": true,
                    "enableButton": true,
                    "motionButton": true,
                    "rtsp": true,
                    "unbridge": true,
                    "forcerefreshsnap": false,
                    "useCachedLocalLivestream": false,
                    "videoConfig": {
                        "audio": true
                    }
                }
            ]
        },
        {
            "name": "Wemo",
            "mode": "auto",
            "platform": "Wemo"
        },
        {
            "email"<EMAIL_ADDRESS>            "password": "password",
            "name": "myQ",
            "platform": "myQ"
        },
        {
            "name": "TplinkSmarthome",
            "addCustomCharacteristics": true,
            "broadcast": "IP",
            "deviceTypes": [
                "plug",
                "bulb"
            ],
            "platform": "TplinkSmarthome"
        },
        {
            "name": "Nest",
            "refreshToken": "token",
            "platform": "Nest"
        },
        {
            "refreshToken": "token",
            "options": {
                "ffmpegCodec": "libx264",
                "streamQuality": 2,
                "alertCheckRate": 10,
                "alertCooldownRate": 180,
                "importantOnly": true,
                "motionDetection": true,
                "streamingSwitch": true,
                "announcementsSwitch": true,
                "audioSwitch": true,
                "fieldTest": false
            },
            "platform": "Nest-cam",
            "_bridge": {
                "username": "MAC:ADDRESS",
                "port": 34408
            }
        },
        {
            "name": "Govee",
            "username"<EMAIL_ADDRESS>            "password": "password",
            "apiKey": "api-key",
            "apiWhitelist": true,
            "debug": false,
            "_bridge": {
                "username": "MAC:ADDRESS",
                "port": 49718
            },
            "platform": "Govee"
        }
    ],
    "accessories": [],
    "bridge": {
        "pin": "home-pin",
        "name": "Homebridge 693E",
        "bind": [
            "ens160",
            "ens192"
        ],
        "port": 52238,
        "username": "MAC:ADDRESS",
        "advertiser": "bonjour-hap"
    },
    "disabledPlugins": [
        "homebridge-wemo",
        "homebridge-google-nest-sdm",
        "homebridge-gsh"
    ]
}

This seems to be a different error message when attempting to open a feed on the eufyCam Pro 2 cameras (same error on both cameras).
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] DEBUG: Backyard RTSP rtspStream:true
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] DEBUG: Backyard RTSP cameraConfig: true
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] DEBUG: Backyard RTSP true
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] DEBUG: Backyard RTSP "rtsp://<user_token>:<user_password>@CAM_IP/live1"
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] DEBUG: Backyard RTSP URL: rtsp://<user_token>:<user_password>@CAM_IP/live1
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] DEBUG: Backyard VIDEOCONFIG: {"audio":true,"stillImageSource":"-i https://cdn-us.eufylife.com/etc","maxWidth":640,"maxHeight":480,"maxFPS":10,"maxStreams":2,"maxBitrate":99,"vcodec":"libx264","packetSize":188,"source":"-i rtsp://<user_token>:<user_password>@CAM_IP/live1","acodec":"libfdk_aac","encoderOptions":"-preset ultrafast -tune zerolatency","forceMax":false}
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] INFO:  Backyard Starting video stream: 640x480, 30 fps, 299 kbps  (AAC-eld)
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] DEBUG: Backyard Stream command: ffmpeg -hide_banner -loglevel level -use_wallclock_as_timestamps 1 -i rtsp://<user_token>:<user_password>@CAM_IP/live1 -an -sn -dn -r 30 -vcodec libx264 -pix_fmt yuv420p -color_range mpeg -f rawvideo -preset ultrafast -tune zerolatency -filter:v scale='min(640,iw)':'min(480,ih)':force_original_aspect_ratio=decrease,scale='trunc(iw/2)*2:trunc(ih/2)*2' -b:v 299k -bufsize 598k -maxrate 299k -payload_type 99 -ssrc 4150782 -f rtp -srtp_out_suite AES_CM_128_HMAC_SHA1_80 -srtp_out_params <session_hash> srtp://MY_IP:55138?rtcpport=55138&pkt_size=188 -progress pipe:1
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] DEBUG: Backyard Stream command: ffmpeg -i rtsp://<user_token>:<user_password>@CAMERA_IP/live1 -vn -sn -dn -codec:a libfdk_aac -profile:a aac_eld -flags +global_header -ar 16k -b:a 24k -ac 1 -payload_type 110 -ssrc 15302105 -f rtp -srtp_out_suite AES_CM_128_HMAC_SHA1_80 -srtp_out_params fxZiLig8J3xkTORB1wdYFJPhgjpkqWQCv4EKIlB+ srtp://MY_IP:62136?rtcpport=62136&pkt_size=188 -progress pipe:1
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] ERROR: Backyard FFmpeg exited with code: 1 and signal: null (Error)
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] DEBUG: Stopping session with id: c5d49b26-739d-432a-b4b8-145549822862
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] INFO:  Backyard Stopped video stream.
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] ERROR: Backyard FFmpeg exited with code: 1 and signal: null (Error)
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] DEBUG: Stopping session with id: c5d49b26-739d-432a-b4b8-145549822862
[7/3/2022, 8:39:54 PM] [EufySecurity-2.0.1] DEBUG: No session to stop.

Seems to be hit or miss with getting the stream to work from one camera to another. I even tried the same with the Solo Cam P24's I have and got the same error on one while another worked fine.

There seems to be a problem with the ffmpeg process. To narrow down the cause I'd like you to try something:

disable audio on the stream (remove audio: true from every videoConfig for the cameras you'd like to test)
enable debug output for ffmpeg (add debug: true to the videoConfig of the cameras) --> so in essence you could just replace audio with debug
try again once with rtsp enabled. If that doesn't work please disable rtsp also and try once more.

Closed, since this is a stale issue.
Please report back if the issue persists. Also please try latest beta beforehand.

Yeah sorry I have not had time to dig back into this but it is still an issue. Video simply does not work now after having been re-connected since I opened this issue. Not a single camera works in HomeKit. 100% works in the Eufy app. I will try and get back to this later on tonight or tomorrow if/when I have time. Thanks.

Ok, as I mentioned please try the latest beta and the steps I mentioned above.
If this doesn't help please send me the log-lib.log file. The storage path should be written to the log at startup.

Again is stale... please provide the logs if the issue persists.

Sorry for the huge delay in my response, work has been absolutely insane. Here is a test while trying to pull up a stream in HomeKit.
[12/1/2022, 5:33:10 PM] [EufySecurity-2.1.3] INFO:  [Backyard] [Video Process] Starting video stream: 1280x720, 30 fps, 299 kbps
[12/1/2022, 5:33:10 PM] [EufySecurity-2.1.3] INFO:  [Backyard] [Video Process] Starting audio stream: 16 kHz, 24 kbps, codec: libfdk_aac
[12/1/2022, 5:33:10 PM] [EufySecurity-2.1.3] ERROR: [Backyard] [Video Process] FFmpeg exited with code: 1 and signal: null (Error)
[12/1/2022, 5:33:19 PM] [homebridge-eufy-security] This plugin slows down Homebridge. The write handler for the characteristic 'Selected RTP Stream Configuration' on the accessory 'Backyard 99BE' didn't respond at all!. Please check that you properly call the callback!

 The write handler for the characteristic 'Selected RTP Stream Configuration' on the accessory 'Backyard 99BE' didn't respond at all!. Please check that you properly call the callback! See https://homebridge.io/w/JtMGR for more info.
[12/1/2022, 5:57:05 PM] [EufySecurity-2.1.3] DEBUG: Backyard Receive Apple HK Stop request{"sessionID":"f938e5cb-848d-445d-a228-cb55d324f274","type":"stop"}
[12/1/2022, 5:57:05 PM] [EufySecurity-2.1.3] DEBUG: Stopping session with id: f938e5cb-848d-445d-a228-cb55d324f274
[12/1/2022, 5:57:05 PM] [EufySecurity-2.1.3] DEBUG: Backyard RTSP rtspStream:true
[12/1/2022, 5:57:05 PM] [EufySecurity-2.1.3] DEBUG: Backyard RTSP cameraConfig: true
[12/1/2022, 5:57:05 PM] [EufySecurity-2.1.3] DEBUG: Backyard RTSP true
[12/1/2022, 5:57:05 PM] [EufySecurity-2.1.3] DEBUG: Backyard RTSP "rtsp://user:pass@<camera_ip>/live1"
[12/1/2022, 5:57:05 PM] [EufySecurity-2.1.3] INFO:  Backyard Stopped video stream.

Hi @metalcated
There are multiple ways to go about this:

Please install the beta version. At least this should lead to a few more debug messages that could help.

There seems to be an error with the ffmpeg process. In the beta the according error should be printed out.

Did you try to disable the rtsp option and stream with p2p-streams?

Yeah I was just checking that before seeing your comment above. I just disabled rstp and after a while checked the camera, seemed to work still.

Well I'd suggest to test a few days and report back if anything changes.

@metalcated and? How's it going?

Disabling rstp seems to be working. I left one camera with it enabled and that still is not working. I just disabled it and tried again. Working. The only other thing I would like to see is possibly improvement in the video quality. The native app looks 10x better than through HomeKit. I know if you enabled HomeKit compatibility via the Eufy app, it mentions reduced quality, is that something out of the control of this plugin?

Well the video quality is influenced by a factor of things.
You have some settings in the eufy app itself to set video quality etc.
Then there are plugin settings. You'd have to share your config json, so that I can check if there's anything going on there. But essentially you could try the copy preset in the advanced setting for the cameras. Maybe that works and fulfils your needs.
Then there's the possibility that your homebridge system just struggles with the video encoding. If that's the case you'd have to try with another system.
Regarding the RTSP: did you check your rtsp settings in the eufy app for every camera? Maybe there is something wrong there? But if the normal mode is working, I'd probably just stick with this.

Copy does seem considerably better yes. I can share the config if you still feel some additional tweaking could be done?

In general you won't do any better than to use copy quality wise. As long as this setting works for you, leave it at that.
The only thing you could try otherwise, would be to tweak the settings in the eufy app, since these will define the quality of the source material for the plugin.
Is the issue resolved by this? Then we could close it for good measure. :-)

Yep, this does seem resolved at this point. Thanks for all of your help!


Parser problems

With this html code:
{{ menuItem.label | translate }}
The result is (with namespacing, but it doesn't matter)
{
    "{{ menuItem": {
        "label | translate }}": ""
    },
}

I believe this never should fall into parser regex (use only 'KEY' | translate), but above everything else, I've also used delimiter [[ ]] and it still generated this one (delimiter works for other standard extractions).

Echo... echo...

I fixed this problem in my fork, see here:
https://github.com/SKoschnicke/grunt-angular-translate/compare/master...SKoschnicke:dots_in_angular_exps?expand=1
Expressions are still extracted, but the namespacing doesn't break anymore. I think also extracting angular expressions helps to not forget these cases when translating.

Done by merge #61.
Thanks a lot @SKoschnicke


prow test framework

Description
Prow test framework.
Prereq for release PR
Converted multiple test scripts to become ready for prow/openshift-ci
Fixes #76
Type of change

[ ] Test Framework

How Has This Been Tested?

[ ] make test-prow
[ ] various other ventures

Test Configuration:

Toolchain: build-devel

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

/lgtm


sidebar nav-item dropdown-menu in dashboard example

can u add nav-item dropdown/accordion functionality for submenu in bootstrap4 admin example

We don't have an admin example. Are you referring to the Dashboard example?

We don't have an admin example. Are you referring to the Dashboard example?

yes , imean dashboard sorry about that..

I fix this issue, using data-toggle="collapse"  and css
Preview:

Snippet

Closing since problem was solved with collapse and custom code. No plans to add more to the Dashboard example at this time.


ci: specify opentel receiver endpoint

When running on the CI we an opentel receiver to push the traces to an external collector.
With recent versions of the receiver, it now listens only on localhost which was prevent it from receiving the traces to begin with. Now we specify the domain name of the receiver.

This fixes the flaky rest client test :)

bors merge

bors cancel

bors merge

Operation not permitted: '/tmp/disk_0'

:open_mouth:
Probably previous cancel killed the test without cleanup? I guess we need to ensure final cleanup does clean everything up, maybe using a specific folder within tmp to make it easier to cleanup, example: one single rm -rf /tmp/ci-run-99
bors merge


adding codecov badge

Description

Checklist
I confirm that I have completed the following checks:

[ ] I have justified any large differences in the regression tests caused by this pull request in the comments.
[ ] I have added new tests where appropriate for the changes I have made.
[ ] If I have had to change any existing unit or integration tests, I have justified this change in the pull request comments.
[ ] If I have made documentation changes, I have checked they render correctly.
[ ] I have added documentation for my change, if appropriate.

Welcome to Codecov :tada:
Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.
Thanks for integrating Codecov - We've got you covered :open_umbrella:


Multiworld logs break the website.

Uploading a MW log doesn't display the locations correctly. I can't provide an example log because github doesn't let me upload JSONs.

Closed by #16


Add chat-based messaging to ClosingMin interactions

Add chat-based messaging to ClosingMin interactions

Moved to new repository https://gitlab.com/mukn/solidity-example


enable libz

https://github.com/curl/curl-docker/issues/52

latest > podman run -it curlimages/curl:8.1.2 -V
has zlib support enabled.


image sensor specification

Hi, I could not find information about image sensor used to record the rgb images. could you give me a link or sensor type?
Thank you so much

The videos were taken by iPhone 5. You can find some camera details here: https://www.gsmarena.com/apple_iphone_5-4910.php

@fyu
Thank you for you quick reply. It helps a lot.
Since image resolution is 1280x720, which is more close to the front camera(1280 x 960 (1.2 MP) ). Is it correct t that front camera is used ?  Since the Rear camera is 3264 x 2448 (8 MP)..

It is the rear camera. The image resolution in video mode of the rear camera is 1280x720. 3264 x 2448 is the max static image resolution.

@fyu
Got it, Thank you !


fix: wording to redirect to macOS_Catalina.md

Checklist

[x] documentation is changed or added
[x] commit message follows commit guidelines

Description of change
Clarifies that macOS_Catalina.md is meant for any user that's upgraded beyond 10.15 and not just users that upgraded to 10.15 Catalina

First PR to open source ever 😅  It's pretty negligible but thought it might be helpful for others looking for node version/macOS node-gyp support

👌 thanks for making this project your first @nickwang14!
I'm fine with this change, I just can't validate it. Did you use these instructions to help an upgrade to Big Sur? Were the instructions accurate?
@cclauss: any experience or thoughts on this? Should we be renaming the doc too?

Hey @rvagg! I still have a couple points pending before I can fully confirm the upgrade is working well.
To provide context, I'm trying to fix a node-gyp dependency gap when upgrading from React 16.12 to 16.14.
I am now on Monterey (Mac OS 12.1) coming from 10.12, the docs have stipulated that it's not for less than 10.15, but this is the latest update documentation that I can find 🙏
I can document my findings and help contribute to the macOS_Updates.md when I manage to clear things up if that helps 🙏

That would be wonderful @nickwang14. This doc comes from a compilation of user-supplied feedback when the big Catalina bork hit us, @cclauss did an excellent job of making sense of it all into a doc. But we're beyond that now and if we're still having similar problems then we should reflect the latest advice and consider renaming it.

The chatter on our issues only lasted for ~3 months when Catalina first came out.  My sense is that there is no longer much need for this content but perhaps it is still relevant for upgrades from pre-Catalina to post-Catalina.  Unclear to me but the drop in issue traffic on this topic is clear.


document how to run F-API locally

everything is already running, now you want to see what the rest of world knows

[ ] first follow regular deploy
[ ] add a f-API locally manually via docker run
[ ] query tool needs to point to f-API now: update .env for query tool -> docker compose up again
[ ] manually make a new .env file (call it something else?), put it somewhere else
[ ] embed this new page on the overview with a little bit of context: https://neurobagel.org/overview/
[ ] mention that if the number of local n-APIs increases, show how to add more

for reference: https://miro.com/app/board/uXjVNOZNJ9A=/?share_link_id=768891461417

I think deploying a local query tool with n-API + Graph is not a good idea anymore, now that we have the f-API scenario locally. Consider these deployment scenarios:

one local node only:

deploy 1: 1x n-API, 1x graph, 1x query tool talking to n-API (what we have so far)

one local node + remote federation

deploy 1: 1x n-API, 1x graph (default would be 1x query tool but that makes little sense)
deploy 2: 1x f-API, 1x query tool talking to f-API

two local nodes:

deploy 1: 1x n-API, 1x graph
deploy 2: 1x n-API, 1x graph
deploy 3: 1x f-API, 1x query tool

Any time we ship a new version of the n-API images, it doesn't make sense to then

shutdown the n-API and update
redeploy the n-API stack including the query tool AND THEN
shut down the query tool on every n-API stack except one again

Long story short: I propose that we change our deployment configuration recommendation and files like so:

deployment recipe "local node": n-API + graph backend
deployment recipe "f-API": f-API + query tool
deployment recipe "query tool only": query tool (would only be used if you are running a single n-API locally, i.e. the old default)


FIFA 19 demo grass render issues..

Screenshoot:

logs show no issues..
System information

GPU: Titan V
Driver: 399.24 (Windows)
Wine version: none
DXVK version: 0.72

Apitrace file(s)
Log files

d3d11.log:
FIFA19_demo_d3d11.log
dxgi.log:
FIFA19_demo_dxgi.log

Apitrace?

Game asks for ID3D12Device, which is interesting. But yeah, needs an apitrace, and should be tested on Linux.

Fifa 19 demo doesn't work on Wine. https://bugs.winehq.org/show_bug.cgi?id=45830

Ok, will add apitrace soon!

After this patch https://source.winehq.org/git/wine.git/commit/7e39544bfa6cf3e68a8567c202624f7f2d3fc7c8 FIFA 19 works on wine.
Look what a strange thing. In the minigame before the match we can see the grass:

In the gameplay, we can't see the grass:

The game crashes when I try to get a apitrace:

@stumts can you post the full wine console output when running the game?

Log files

wine.log (without any WINEDEBUG)
wine.log
d3d11.log
fifa19_demo_d3d11.log
dxgi.log
fifa19_demo_dxgi.log

Nothing interesting. Guess I'll take a look when the wine patch lands.

I get black grass on Windows 2 using dxvk

Not even the grass of all stadiums is black. Some work perfectly fine, some are completely black and some are only partially black.

Black screen also confirmed with following settings:

Ubuntu 18.04.01
Lutris with DXVK support 0.90
esync-staging-pba-3.18
AMD Radeon R9 280x

I also have a weird issue, where when my controller is plugged in first it all of the sudden always moves to the right. I need to remove the controller from "Start > Control Center > Gamecontroller", then start the game. Confirm the keyboard as input device (so that the Enter/Space/etc. buttons are shown within the game rather than the XBOX A/B/X/Y buttons) and then move around and play with my controller. This way it does not move arbitrarily.

The black grass/stadium problem is here with DXVK 0.93:

Arch Linux
DXVK 0.93
Wine esync-staging-pba-3.21
NVIDIA GTX 1080Ti

Can anyone record an apitrace? Screenshots aren't exactly helpful, and Origin doesn't work for me.

The game crashes for me when I try to get an apitrace. I'll try again.
According to wine bug the mentioned wine patch is fixed with 7e39544bfa6cf3e68a8567c202624f7f2d3fc7c8 in Wine 3.17:
https://bugs.winehq.org/show_bug.cgi?id=45523

Nope the game crashed when I try to get an apitrace. Here you are the loh if it can be of some help:
log.zip

Here is the apitrace to the crash:
https://my.pcloud.com/publink/show?code=XZCTKg7ZOwNnjqAYGQF1NUtCVB75jXAGfBQy

It's not useful unless it actually shows the grass rendering issue.

I know but this is I can get :(

May be try RenderDoc then?
It can be depth bias issue similar to SOTTR on nvidia and can be driver related.

A user from discord called Aymen has gotten an apitrace from windows.
https://drive.google.com/open?id=1ISC5VdAFD31MUdPOyCQid0ViqzGNsikm
Is It usefull for you? Anyway, I'll try to get the fifa 19 apitrace from wine in diferent way.

@stumts yep it's me :)
But your link points weirdly to https://github.com/doitsujin/dxvk/issues/url
here is the apitrace link https://drive.google.com/file/d/1ISC5VdAFD31MUdPOyCQid0ViqzGNsikm/view
Hope it helps

System Information:

SO: Ubuntu 18.04.1
GPU: Nvidia GTX 850m
Video driver: 415.18.02
Wine version: 3.21 (vanilla)
DXVK version: 0.93

Apitrace File:

https://drive.google.com/open?id=1nx6bMpBpaPY2DVr0kQmD_1JwfjrVh_qv

Log Files:

d3d11.log: FIFA19_d3d11.log
dxgi.log: FIFA19_dxgi.log

I'm able to reproduce the problem, but I don't think I'll be able to fix this any time soon. It doesn't look like DXVK is doing anything wrong.

May be it's Wine problem?

No, it's not. It's quite obviously a DXVK bug but I don't know what it's doing wrong.

Ok, we'll wait...u are the best!! thnxs for taking the time!

Thanks for the reply!
My observation about VRAM was coming from nvidia-smi, as you suggest, and an htop in the side to see CPU + RAM. After a clean reinstall I managed to make different texture resolution consume different amount of VRAM. However, although the game runs at 120 FPS and has about 300MB VRAM to spare, in some cutscenes the FPS drop to 2 or 3. Anyway, this has gotten too specific to be useful or worth looking into.
Thanks for the great work! Cheers,

First, thanks for dxvk, it's great. :) It works well with a lot of steam games I have. I tried Fifa19 last night, but it runs a lot slower than on Windows. I don't have a good computer, but I can play Ffifa19 smoothly on Windows with 1080p. But on Linux, the fps is 8-10 on 1080p, and 15-20 on 1600x900. My question is why? I thought dxvk doesn't add any overhead, does it? Can I finetuune it somehow? Maybe esync?
My config:
Distro:Ubuntu Disco Dingo (development branch)
Kernel:5.0.0-7-generic
RAM:10 GB
GPU Driver:NVIDIA 410.104
GPU:NVIDIA GeForce GTX 750
CPU:AMD FX-6300 Six-Core
Wine: 4.4 staging repo
DXVK. 1.0.1

First of all, check whether you are actually GPU bound (in the nvidia settings). DXVK doesn't run well on old GPUs, especially since yours only has 1GB of VRAM (or is it 2GB?).

Yes, as I see the GPU is the bottleneck, it only has 1 GB of VRAM. I can get a GTX 670 2 GB, which is a stronger GPU, but as you mentioned DXK doesn't run well on old cards. Should I buy a newer one? What is your recommendation for my setup?

doitsujin, please help on how to fix the black grass on fifa 19

dxvk-fifa.tar.gz
you uploaded this in the past, how do i get to use it so it can work for me as well?

Just use the latest release. It should just work.

i downloaded the fifa 19 repack, it still gives me the same problem

im using hd graphics 5500

We don't support cracked games. That's almost certainly the reason why it doesn't work.

The game does not support Intel graphics and the exact same issue is known to happen on Windows as well on those GPUs.


Ignore conversion default save dirs and pycache

sigh, as much as i would love to help improve, the owners don't seem to care.


Resolved issue #138,#137,#136,#135,#134

@4darsh-Dev Please merge it and assign the labels as required.I tried to make 2 pr's but it comitted in single one.

@4darsh-Dev I did these css changes.please check

@Garima-149 , mark the issues as completed,
PR is merged.

@4darsh-Dev I marked them already.
Can I ask what levels you have assigned to issues I raised? Since, I made a single pr for all 5 issues.

there was only single pr for all issues so , I gave max level3
only points are counted from PRs labels, i thnk.

@4darsh-Dev okay I didn't know it before, otherwise I would have made multiple ones.

not possible @Garima-149  now,
it will look spammy, I have told you multiple times to not make single PR already,
Will take care for the next times.

GSSoC 24 has been completed Finally'
It was a great experience working with you all
Thanks💗 for your valuable contributions!
PA nomination has been started, Do fill out the forms soon. Share your experiences and let's connect on socials


Get package version from central store in projects

Hi,
cant find how to do this :)
i am doing some custom deployments for nunit, and would like to take 1.0.3 from the central Packages.props to avoid duplicating the 1.0.3 i both central and in project.
<None Include="$(NuGetPackageRoot)NUnit.TeamCityEventListener\1.0.3\tools\**">
      <Link>Nunit\Addins\%(RecursiveDir)%(FileName)%(Extension)</Link>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>

is there some function like
$([MSBuild]::GetPathOfFileAbove('Directory.Build.props', '$(MSBuildThisFileDirectory)../'))
as
$([MSBuild]::GetPackageVersion('NUnit.TeamCityEventListener')) that should resolve to the current version either central or overridden?
thanks

Use the GeneratePathProperty feature I added:
https://blog.dangl.me/archive/accessing-nuget-package-paths-in-your-net-sdk-based-csproj-files/

Alright, this is exactly what I was looking for. Thanks for implementing.
On Mon, 30 Sep 2019, 18:05 Jeff Kluge<EMAIL_ADDRESS>wrote:

Use the GeneratePathProperty
https://github.com/NuGet/NuGet.Client/pull/2271 feature I added:
https://blog.dangl.me/archive/accessing-nuget-package-paths-in-your-net-sdk-based-csproj-files/
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/MSBuildSdks/issues/132?email_source=notifications&email_token=AAH6HSC5YAHTT644KU4OM7DQMIIU7A5CNFSM4I3RLPM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD757C2Q#issuecomment-536605034,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAH6HSFA7CDMX4QPP6BP6XDQMIIU7ANCNFSM4I3RLPMQ
.

@jeffkl sorry, after all this is not work yet! can u confirm vs 2019.3 supports this for non web sdks?

The name of the property is $(PkgPackage_Name_With_Periods_Replaced_With_Underscores).
So the property for NUnit.ConsoleRunner is $(PkgNUnit_ConsoleRunner).
You can see them all in obj\projectname.nuget.g.props

Oh sorry. Need some glasses of course...
On Wed, 2 Oct 2019, 18:26 Jeff Kluge<EMAIL_ADDRESS>wrote:

The name of the property is
$(PkgPackage_Name_With_Periods_Replaced_With_Underscores).
So the property for NUnit.ConsoleRunner is $(PkgNUnit_ConsoleRunner).
You can see them all in obj\projectname.nuget.g.props
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/MSBuildSdks/issues/132?email_source=notifications&email_token=AAH6HSBMHGEOA6WAI4U664TQMS4TDA5CNFSM4I3RLPM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAFE4TQ#issuecomment-537546318,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAH6HSHQPA53SDIJ4JIFTWDQMS4TDANCNFSM4I3RLPMQ
.


BMC state is still checked using Get BMC State Deprecated when xyz interface are available

During code update, BMC state is still checked using Get BMC State Deprecated when xyz interface are available. Need a better logic.

The problem is set sate variable was done using org interface. As older org interfaces are still present this logic does not work. Need to make decision based on new xyz interface.


[BUG] Console version hardcodes TargetFramework to 16299 for UWP

Describe the bug
Generating events for UWP platform using console hardcodes the TargetPlatform to 16299.
Console app should take another parameter to specify the TargetPlatform
Steps To Reproduce
pharmacist generate-platform -p uwp -o c:\logs\ -output-prefix _Event
Additional context
Should this be applicable for other platforms too?

I can confirm this issue. I've tried to build ReactiveUI.Events and modified the targetframework to uap10.0.18362. The generated uap.cs file from Pharmacist contains a lot uses of deprecated or obsolete classes. I would love to know how to fix this or maybe anyone else can. ReactiveUI.Events is due to this reason not usable in more modern UWP applications.

Yes it's a known issue. It's on my to-do list for sometime soon just have to have a free weekend.

After a little more research. The generated .cs file for UWP did not so many issues after all.  I had  to add the SDKReferences WindowsDesktop and WindowsMobile Extensions for UWP manually. This also solves errors when using any of these api's with ReactiveUI.Events in UWP applications.

Fixed in 2.0.3


Add Nutanix Cloud Provider project to bundle

Description of changes:
Add Nutanix Cloud Provider project to bundle
Testing (if applicable):
$ make dev-release
...
gitTag: v0.0.0-dev+build.0
number: 1
version: v0.0.0-dev+build.0

✅ Successfully completed dry-run of release process

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/ok-to-test

/lgtm
/approve

/retest

/retest


Compat bounds for stdlibs

Here's what I get when I try to install SCT v0.4 in a Julia 1.6 environment with DI:
ERROR: LoadError: Unsatisfiable requirements detected for package SparseConnectivityTracer [9f842d2f]:
 SparseConnectivityTracer [9f842d2f] log:
 ├─possible versions are: 0.1.0-0.4.1 or uninstalled
 ├─restricted to versions 0.4 by an explicit requirement, leaving only versions 0.4.0-0.4.1
 └─restricted by compatibility requirements with SparseArrays [2f01184e] to versions: 0.1.0 or uninstalled — no versions left
   └─SparseArrays [2f01184e] log:
     ├─possible versions are: 0.0.0 or uninstalled
     ├─restricted to versions [0.0.0, 1] by DifferentiationInterface [a0c0ee7d], leaving only versions 0.0.0
     │ └─DifferentiationInterface [a0c0ee7d] log:
     │   ├─possible versions are: 0.4.1 or uninstalled
     │   └─DifferentiationInterface [a0c0ee7d] is fixed to version 0.4.1
     └─SparseArrays [2f01184e] is fixed to version 0.0.0

I think it is connected to https://discourse.julialang.org/t/psa-compat-requirements-in-the-general-registry-are-changing/104958 and https://discourse.julialang.org/t/why-random-jl-is-fixed-to-version-0-0-0/105957/2:

in DI, I put "<0.0.1,1" in all the stdlib compats because I perform Pkg operations inside the test suite
in SCT you just put "1", but maybe I would need the "<0.0.1" too

I realized this in the following CI run: https://github.com/gdalle/DifferentiationInterface.jl/actions/runs/9203495645/job/25315149741?pr=269
When I perform a Pkg operation during tests, SCT gets downgraded from v0.4 to v0.1. I presume it's because SCT v0.1 didn't have stdlib compat bounds yet


Upload test results

Fixes #1298

[x] Merge #1288 first

Codecov Report

Merging #1299 (626822b) into master (f923592) will increase coverage by 10.23%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #1299       +/-   ##
=============================================
+ Coverage     76.03%   86.27%   +10.23%
+ Complexity     1017       85      -932
=============================================
  Files           187       20      -167
  Lines          2938      255     -2683
  Branches        508       16      -492
=============================================
- Hits           2234      220     -2014
+ Misses          435       25      -410
+ Partials        269       10      -259

Impacted Files
Coverage Δ

...zally/configuration/RulesValidatorConfiguration.kt

...ndo/zally/configuration/NoSecurityConfiguration.kt

cli/zally/commands/supported_rules.go

...in/org/zalando/zally/core/JsonPointerExtensions.kt

web-ui/src/server/handlers/index.js

...do/zally/ruleset/zalando/ApiMetaInformationRule.kt

...alando/zally/ruleset/zally/TagAllOperationsRule.kt

...o/zally/configuration/RestTemplateConfiguration.kt

...zally/ruleset/zalando/PascalCaseHttpHeadersRule.kt

...alando/zally/ruleset/zalando/NoVersionInUriRule.kt

... and 157 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f923592...626822b. Read the comment docs.

The tooling doesn't work for open source repositories due to security reasons.


How to save and load the constant tensor of custom layer

System information.

Have I written custom code (as opposed to using a stock example script provided in Keras):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Python version:
Bazel version (if compiling from source):
GPU model and memory:
Exact command to reproduce:

Describe the problem.
I want to save a model which contains some custom layers, especially there are some constant input tensors inside, like this
(where the GCNconv is something like https://github.com/danielegrattarola/spektral/blob/master/spektral/layers/convolutional/gcn_conv.py)
import GCNconv from spektral.layer
img_input = Input(shape=input_shape)
x = Conv2D(3, (3, 3), activation='relu', padding='same', name='conv1_1')(img_input)
x = Conv2D(3, (3, 3), activation='relu', padding='same', name='conv1_2')(x)
x = MaxPooling2D((2, 2), strides=(2, 2), name='pool1')(x)    # output x: (B*H*W*3)
x = Reshape(H*W, 3)     # output x: (B*(H*W)*3)

gcn = GCNConv(1024)      # The custom layer

A = tf.Variable(np.eye(3), trainable=True, shape=(3, 3), name='adj')      #The constant input tensor
y = gcn([x, A])      # Use the constant tensor

Then save the model use model.save() or the model.to_json(), the serialize method in https://github.com/tensorflow/tensorflow/blob/5dcfc51118817f27fad5246812d83e5dccdc5f72/tensorflow/python/keras/engine/node.py#L199
will treat the input tensor A of the custom layer as a _CONSTANT_VALUE. This leads to the JSON saved in the format as:
{"class_name": "GCNConv", "config": {"name": "gcn_conv", "trainable": true, "dtype": "float32", "activation": "linear", "use_bias": false, "kernel_initializer": {"class_name": "LecunNormal", "config": {"seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": {"class_name": "L2", "config": {"l2": 0.0005000000237487257}}, "bias_regularizer": null, "kernel_constraint": null, "bias_constraint": null, "channels": 1024}, "name": "gcn_conv", "inbound_nodes": [[["reshape", 0, 0, {}], ["_CONSTANT_VALUE", -1, [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]], {}]]]}

When load the model  with model_from_json
model = tf.keras.models.model_from_json(loaded_model_json, custom_objects={'GCNConv': GCNConv})

According to https://github.com/tensorflow/tensorflow/blob/5dcfc51118817f27fad5246812d83e5dccdc5f72/tensorflow/python/keras/engine/node.py#L181, The _CONSTANT_VALUE  A cannot be converted back to a tf.tensor, but will be a List of tensor :
[[<tf.Tensor: shape=(), dtype=float32, numpy=1.0>, <tf.Tensor: shape=(), dtype=float32, numpy=0.0>, <tf.Tensor: shape=(), dtype=float32, numpy=0.0>], [...], [<tf.Tensor: shape=(), dtype=float32, numpy=0.0>, <tf.Tensor: shape=(), dtype=float32, numpy=0.0>, <tf.Tensor: shape=(), dtype=float32, numpy=1.0>]]

Is there any way to solve the problem or bypass it?
Describe the current behavior.
_CONSTANT_VALUE in JSON will be deserialized into a List.
Describe the expected behavior.
The constant tensor can be loaded from JSON as a tf.tensor.

Same problem happened when save the model into '.h5' and '.pb', maybe I need to carefully define the custom layer? Is there any guidance for that?

@ambarion ,
Can you please take a look at this SO link which delivers the similar information.It helps.Thanks!


License file mispelled as Liscence

https://github.com/climbalima/let-s-Split-v2/blob/master/Liscence

Fixed by PR #2


Listen on port 53 on FreeBSD

Hi,
just looked around the closed issues but I can't find a good answer.
I want to run dnscrypt-proxy on port 53 but I always get:
listen udp <IP_ADDRESS>:53: bind: permission denied
I already tried user_name = 'root' or listen_address = [] and also checked via sockstat if port 53 is already in use (but isn't).
Is there another way I'm not aware of?

Hi!
How are you starting it? Are you using the FreeBSD port?
You should start is as root. You don't need to change user_name or listen_address.

Unfortunately, I don't know much about HardenedBSD (assuming that this is what HBSD stands for).
But maybe there is something to explicitly allow command to be able to listen to specific ports?

Setting dnscrypt_proxy_suexec:="YES" as rc variable helps. Now I can finalize the plugin :) Thanks


[BUG] Proces killed after ~ 3 hours of running

Describe the bug
Hi, I have clean install of Ubuntu 22.04, and latest .jar from repo. When I run
java -Xmx120g -jar planetiler.jar --download --area=planet in screen, it is killed after approx. 3 hours of runtime. I have tried it twice but with the same result.
To Reproduce
Steps to reproduce the behavior:
java -Xmx120g -jar planetiler.jar --download --area=planet
Screenshots
2:45:08 INF [mbtiles] -  features: [ 325M  11% 1.7M/s ] 212G  tiles: [  10M 134k/s ] 9.1G     cpus: 29.5 gc:  9% heap: 78G/128G direct: 18M postGC: 36G     read(60% 54%) -> merge(53%) -> (4.6k/6.4k) -> encode(69% 84% 71% 85% 86% 83% 75% 73% 90% 67% 76% 72% 67% 84% 73% 85% 82% 73% 81% 67% 75% 87% 86% 84% 73% 65% 89% 72% 75% 89%) -> (6.4k/6.4k) -> write(69%)     last tile: 12/1924/1875 (z12 46%) https://www.openstreetmap.org/#map=12/15.02969/-10.89844 2:45:18 INF [mbtiles] -  features: [ 341M  11% 1.5M/s ] 212G  tiles: [  11M  45k/s ] 9.6G     cpus: 29.9 gc:  8% heap: 27G/128G direct: 18M postGC: 16G     read(51% 62%) -> merge(72%) -> (4.9k/6.4k) -> encode(74% 86% 85% 78% 79% 77% 81% 82% 83% 85% 77% 85% 76% 88% 75% 85% 79% 84% 81% 79% 82% 78% 81% 81% 80% 82% 83% 75% 75% 81%) -> (6k/6.4k) -> write(79%)     last tile: 12/2069/1728 (z12 50%) https://www.openstreetmap.org/#map=12/27.05913/1.84570 2:45:28 INF [mbtiles] -  features: [ 356M  12% 1.5M/s ] 212G  tiles: [  11M  25k/s ] 10G     cpus: 29.6 gc: 11% heap: 27G/128G direct: 18M postGC: 19G     read(71% 63%) -> merge(60%) -> (4.9k/6.4k) -> encode(86% 83% 86% 84% 78% 86% 84% 79% 82% 83% 87% 84% 85% 80% 86% 78% 80% 76% 81% 82% 83% 80% 88% 78% 71% 78% 79% 82% 79% 79%) -> (5.3k/6.4k) -> write(66%)     last tile: 12/2150/1306 (z12 52%) https://www.openstreetmap.org/#map=12/54.47004/8.96484 2:45:38 INF [mbtiles] -  features: [ 369M  12% 1.3M/s ] 212G  tiles: [  11M  17k/s ] 10G     cpus: 29.6 gc: 13% heap: 38G/128G direct: 18M postGC: 21G     read(52% 65%) -> merge(72%) -> (5.6k/6.4k) -> encode(83% 80% 80% 86% 84% 75% 83% 84% 84% 82% 82% 81% 84% 77% 77% 75% 82% 81% 84% 78% 79% 80% 82% 82% 76% 76% 79% 77% 84% 78%) -> (5.8k/6.4k) -> write(65%)     last tile: 12/2206/1254 (z12 53%) https://www.openstreetmap.org/#map=12/57.04073/13.88672 2:45:48 INF [mbtiles] -  features: [ 381M  12%   1M/s ] 212G  tiles: [  12M  19k/s ] 11G     cpus: 29.8 gc: 12% heap: 26G/128G direct: 18M postGC: 24G     read(60% 57%) -> merge(51%) -> (6.3k/6.4k) -> encode(86% 83% 83% 86% 80% 80% 86% 80% 81% 85% 83% 79% 86% 81% 79% 78% 83% 86% 84% 84% 81% 76% 84% 87% 82% 82% 81% 78% 83% 83%) -> (6.4k/6.4k) -> write(65%)     last tile: 12/2274/2 (z12 55%) https://www.openstreetmap.org/#map=12/85.03594/19.86328 2:45:59 INF [mbtiles] -  features: [ 392M  13% 1.1M/s ] 212G  tiles: [  12M  16k/s ] 11G     cpus: 29.7 gc: 13% heap: 31G/128G direct: 18M postGC: 26G     read(53% 61%) -> merge(21%) -> (5.9k/6.4k) -> encode(84% 75% 85% 86% 84% 84% 85% 85% 85% 85% 80% 83% 83% 83% 82% 83% 79% 77% 84% 81% 85% 85% 84% 84% 85% 86% 85% 85% 85% 83%) -> (6.4k/6.4k) -> write(55%)     last tile: 12/2331/3150 (z12 56%) https://www.openstreetmap.org/#map=12/-69.09994/24.87305 Killed
Environment:

Hardware: AMD Ryzen™ 9 5950X, 16 Core Vermeer (Zen3)  Simultaneous Multithreading Virtualization (AMD-V), 128 GB DDR4 ECC, 2 x 3.84 TB NVMe SSD
OS: Ubuntu 22.04
Java version and distribution: openjdk version "18.0.2-ea" 2022-07-19
OpenJDK Runtime Environment (build 18.0.2-ea+9-Ubuntu-222.04)
OpenJDK 64-Bit Server VM (build 18.0.2-ea+9-Ubuntu-222.04, mixed mode, sharing)
Maven version: [e.g. 3.8.1]

Additional context
With --area=planet it works normally.

Can you try this once:
java -Xmx20g \
  -jar planetiler.jar \
  `# Download the latest planet.osm.pbf from s3://osm-pds bucket` \
  --area=planet --bounds=planet --download \
  `# Accelerate the download by fetching the 10 1GB chunks at a time in parallel` \
  --download-threads=10 --download-chunk-size-mb=1000 \
  `# Also download name translations from wikidata` \
  --fetch-wikidata \
  --mbtiles=output.mbtiles \
  `# Store temporary node locations at fixed positions in a memory-mapped file` \
  --nodemap-type=array --storage=mmap

Planet.md says this is the right thing to do if you do not have 1.5x the memory of the input file

@wipfli is correct, you definitely need some different options when running on the planet.  quickstart.sh sets up the recommended settings for you as well.
My guess is it used too much memory and the OS killed it... it's not obvious from the logs because the logs say it's only using 31G, but what can happen is that the JVM is allowed to use up to 120G because you set -Xmx120g and even though the heap shrunk to 31G, java hasn't released that extra back to the OS yet and during the mbtiles writing phase, sqlite requests quite a bit of memory outside the JVM heap.
Adding --storage=mmap and a smaller heap between -Xmx20g and -Xmx30g like the command @wipfli is the best way to fix it.  That will let planetiler scale up to much larger input files without needing a bigger machine.
Another option is to tell the JVM to more aggressively release unused memory by using an option like -XX:MaxHeapFreeRatio=40 but I think the better route would be to just run the command Oliver suggested from planet.md.

Hi @wipfli and @msbarry. Thank you for your super fast response. I have tried your command @wipfli. This is new error from it:
0:51:37 INF [wikidata] -  blocks: [  39k    0/s ] nodes: [ 8.1B    0/s ] ways: [ 915M    0/s ] rels: [  10M    0/s ] wiki: [ 1.6M 3.4k/s ] 351M
cpus: 0.1 gc:  0% heap: 10G/21G direct: 81k postGC: 2.6G
parse() -> pbf( -%) ->    (0/94) -> filter( -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%) ->    (0/1M) -> fetch( -%)
0:51:37 INF [wikidata] - DONE fetched:1605286
0:51:37 INF [wikidata] - Finished in 9m47s cpu:2h28m37s gc:26s avg:15.2
0:51:37 INF [wikidata] -   pbf     1x(40% 3m52s sys:1m40s wait:54s done:4m23s)
0:51:37 INF [wikidata] -   filter 31x(44% 4m16s sys:1s wait:43s done:4m19s)
0:51:37 INF [wikidata] -   fetch   1x(14% 1m25s sys:4s wait:8m9s)
0:51:40 INF - loaded from 1603876 mappings from /root/planetiler/data/sources/wikidata_names.json in 4s cpu:13s avg:3.7
0:51:40 DEB - ✓ 334G storage on / (/dev/md2) requested for read phase disk, 2T available
0:51:40 DEB -  - 82G used for temporary node location cache
0:51:40 DEB -  - 11G used for temporary multipolygon geometry cache
0:51:40 DEB -  - 240G used for temporary feature storage
0:51:40 DEB - ✓ 361G storage on / (/dev/md2) requested for write phase disk, 2T available
0:51:40 DEB -  - 240G used for temporary feature storage
0:51:40 DEB -  - 120G used for mbtiles output
0:51:40 WAR - ❌ 21G JVM heap requested for read phase, 21G available (increase JVM -Xmx setting)
0:51:40 WAR -    - 21G used for temporary profile storage
0:51:40 DEB - ✓ 93G storage on / (/dev/md2) requested for read phase, 2T available
0:51:40 DEB -  - 82G used for array node location cache (switch to sparsearray to reduce size)
0:51:40 DEB -  - 11G used for multipolygon way geometries
Exception in thread "main" java.lang.IllegalStateException: Insufficient resources for read phase, use the --force argument to continue anyway:
21G JVM heap requested for read phase, 21G available
at com.onthegomap.planetiler.util.ResourceUsage.checkAgainstLimits(ResourceUsage.java:162)
at com.onthegomap.planetiler.Planetiler.checkMemory(Planetiler.java:601)
at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:520)
at com.onthegomap.planetiler.basemap.BasemapMain.run(BasemapMain.java:51)
at com.onthegomap.planetiler.basemap.BasemapMain.main(BasemapMain.java:14)
at com.onthegomap.planetiler.Main.main(Main.java:50)

Try -Xmx25g

Tried -Xmx30g. Still running for 1 hour now. Will let you know:)

Finished! Thank you for your help:)
2:07:33 INF [mbtiles] - Finished in 55m36s cpu:26h4m54s gc:5m29s avg:28.1
2:07:33 INF [mbtiles] -   read    2x(13% 7m27s sys:2m39s wait:43m28s done:6s)
2:07:33 INF [mbtiles] -   merge   1x(16% 8m52s sys:5s wait:42m17s done:1s)
2:07:33 INF [mbtiles] -   encode 30x(82% 45m20s sys:3s wait:4m2s done:1s)
2:07:33 INF [mbtiles] -   write   1x(28% 15m27s sys:3m38s wait:35m45s)
2:07:33 INF - Finished in 2h7m34s cpu:59h38m16s gc:12m33s avg:28.1
2:07:33 INF - FINISHED!
2:07:33 INF -
2:07:33 INF - ----------------------------------------
2:07:33 INF -   overall          2h7m34s cpu:59h38m16s gc:12m33s avg:28.1
2:07:33 INF -   wikidata         4m59s cpu:2h25m2s gc:19s avg:29.2
2:07:33 INF -     pbf     1x(83% 4m8s sys:1m56s wait:26s done:8s)
2:07:33 INF -     filter 31x(86% 4m17s wait:15s done:7s)
2:07:33 INF -     fetch   1x(1% 4s wait:4m49s done:7s)
2:07:33 INF -   lake_centerlines 0.9s cpu:6s avg:7.1
2:07:33 INF -   water_polygons   46s cpu:18m46s gc:4s avg:24.7
2:07:33 INF -     read     1x(47% 21s wait:18s done:2s)
2:07:33 INF -     process 31x(69% 32s wait:10s)
2:07:33 INF -     write    1x(53% 24s sys:9s wait:19s)
2:07:33 INF -   natural_earth    4s cpu:15s avg:3.6
2:07:33 INF -     read     1x(84% 4s)
2:07:33 INF -     process 31x(5% 0.2s wait:4s)
2:07:33 INF -     write    1x(0% 0s wait:4s)
2:07:33 INF -   osm_pass1        5m47s cpu:2h29m10s gc:24s avg:25.8
2:07:33 INF -     read     1x(66% 3m51s sys:2m1s wait:1m17s done:2s)
2:07:33 INF -     process 31x(73% 4m12s sys:5s block:7s wait:1m3s)
2:07:33 INF -   osm_pass2        54m2s cpu:26h23m25s gc:5m3s avg:29.3
2:07:33 INF -     read     1x(14% 7m40s sys:5m24s wait:44m27s done:32s)
2:07:33 INF -     process 31x(82% 44m27s sys:31s block:8s wait:1m28s)
2:07:33 INF -     write    1x(62% 33m44s sys:8m51s wait:14m13s)
2:07:33 INF -   boundaries       8s cpu:10s avg:1.3
2:07:33 INF -   sort             5m56s cpu:1h55m56s gc:1m14s avg:19.5
2:07:33 INF -     worker  8x(31% 1m49s sys:26s wait:1m30s done:3s)
2:07:33 INF -   mbtiles          55m36s cpu:26h4m54s gc:5m29s avg:28.1
2:07:33 INF -     read    2x(13% 7m27s sys:2m39s wait:43m28s done:6s)
2:07:33 INF -     merge   1x(16% 8m52s sys:5s wait:42m17s done:1s)
2:07:33 INF -     encode 30x(82% 45m20s sys:3s wait:4m2s done:1s)
2:07:33 INF -     write   1x(28% 15m27s sys:3m38s wait:35m45s)
2:07:33 INF - ----------------------------------------
2:07:33 INF -   features        217GB
2:07:33 INF -   mbtiles 84GB


update couchdb role for service script

adds https://github.com/emord/ansible-couchdb-cluster/commit/826df6010e191f17c35be598e9e4ba79d538c47f
@dimagi/scale-team
buddy @orangejenny

nice. do you know if it will retry after a period? or does this pretty much mean we'll have to bring up couch on restart?

@benrudolph that's something we'll need to solve elshwere in our stack. like after ecryptfs is run.
Right now we also have to start ES psql and riak manually

root@pillow1:~# service couchdb2 start
couchdb2 could not start because data directory does not exist

I've never been so excited for a database to fail to start

haha, nice!


namespaces, teams & repos pages broken on the latest :head docker image

Description
On current :head docker image, listings for namespaces, teams & repos are totally broken even if the related XHR request seem to work correctly

Any error on the console tab?

No one

irc transcript
13:23 <kiorky> hello
13:23 <kiorky> im the guy for https://github.com/SUSE/Portus/issues/1902#issuecomment-408063802
13:23 <vitoravelino> hey :)
13:23 <kiorky> if we can get this fixed more interactivly than through github ^^
13:23 <vitoravelino> we can try...
13:25 <kiorky> So nothing stays in the console log.
13:25 <vitoravelino> that's weird...
13:25 <vitoravelino> you also mentioned that nothing is working
13:26 <kiorky> Uhm, if i am not clear:
13:26 <kiorky> i can login, normally but listings on all pages except the users & registry sections are empty
13:26 <vitoravelino> I got that
13:27 <vitoravelino> any error on the teams page?
13:27 <kiorky> on the teams page, if i try to click on "create new team", it seems also broken as it doesnt trigger anything anymore.
13:27 <kiorky> and yes, i see none of the teams.
13:27 <vitoravelino> in the console tab?
13:28 <kiorky> but what's bizarrious is that i see a XHR that returns the teams data correctly, thats what i tried to explain with the screenshot
13:29 <vitoravelino> indeed it is... because the teams page, for example, there's a vue component that just renders the data on the screen
13:29 <vitoravelino> without needing to fetch
13:29 <vitoravelino> in the namespaces page we do fetch namespaces
13:30 <kiorky> yep, you are correct, the xhr i saw is on namespace page
13:30 <vitoravelino> and to fetch that, the js part should be working
13:30 <kiorky> nothing on the team page is XHR fetched.
13:30 <vitoravelino> it's weird to not having the render part working but the fetch part working
13:31 <vitoravelino> did you try in another browser?
13:32 <kiorky> https://x.x.net/  / x : x
13:32 <kiorky> yep, i tried with the same results in another browser.
13:34 <vitoravelino> hm... weird
13:34 <vitoravelino> it's not being recognized as a vue app by vuedevtools
13:35 <kiorky> to work on : https://github.com/SUSE/Portus/pull/1903
13:35 <vitoravelino> probably because it's prod env
13:36 <kiorky> i mounted /app & /lib as volumes, would it be that the problem ?
13:40 <vitoravelino> I think vue is having a problem parsing the data
13:40 <vitoravelino> since it's in prod mode, no error will be thrown by vue
13:46 <kiorky> how do i enable it ?
13:48 <vitoravelino> the problem is that for the official docker images we use a rpm that already has everything compiled there
13:48 <vitoravelino> so there's no easy way of debugging this
13:53 <kiorky> i can go inside the running container and edit somethingf
13:54 <kiorky> what's strange is that it was working a day ago
13:54 <vitoravelino> hm... we had a new docker image published yesterday
13:56 <vitoravelino> and a lot of changes happened to the UI between the old image (+- a month and a half ago) and the current one

and final transcript (skipped a large uninsterressant part)
15:24 <vitoravelino> the image doesn't have the most recent changes to generate the proper bundle                                                                                |
15:24 <vitoravelino> so the bundle before my optimization might be broken by default                                                                                |
15:25 <vitoravelino> I'll ask my friend to trigger another rebuild making sure that the latest code is used                                                                                |
15:25 <kiorky> Can i reformulate as this: "The uploaded image on the docker hub has a not uptodate assets bundle, and this one might by broken"                                                                                |
15:25 <kiorky> s/by/be/                                                                                |
15:26 <vitoravelino> I need to confirm but I'm almost sure that's the case                                                                                |
15:26 <kiorky> Just to be sure i have understood everything                                                                                |
15:26 <vitoravelino> try https://gist.github.com/vitoravelino/4f6e5e731227894e22ef3b9163b2b6cb                                                                                |
15:26 <vitoravelino> just to fix the problem in the meantime for you while we fix this on our side                                                                                |
15:27 <vitoravelino> lemme know if it worked                                                                                |
15:29 <kiorky> which file has to be replaced by that gist                                                                                |
15:29 <kiorky> ?                                                                                |
15:29 <kiorky> application-d5844e91426a4322e887.js                                                                                |
15:30 <kiorky> ?                                                                                |
15:30 <vitoravelino> assets/webpack/application-80066879b423b63a7e7e.js                                                                                |
15:30 <vitoravelino> inside of the container                                                                                |
15:30 <kiorky> ok                                                                                |
15:30 <vitoravelino> I guess the source map line can be ignored                                                                                |
15:40 <vitoravelino> oh, it's the file that is being served is probably the js.gz one                                                                                |
15:40 <kiorky> https://registry.makina-corpus.net/assets/webpack/application-80066879b423b63a7e7e.js seems fine                                                                                |
15:41 <kiorky> but my chrome is somehow using the old version                                                                                |
15:41 <kiorky> even if i clear cache, wrf.                                                                                |
15:41 <kiorky> *wtf                                                                                |
15:41 <vitoravelino> probably because nginx is serving the gzipped one                                                                                |
15:42 <vitoravelino> could you remove application-80066879b423b63a7e7e.js.gz                                                                                |
15:42 <vitoravelino> and run gzip application-80066879b423b63a7e7e.js ?                                                                                |
15:45 <kiorky> yes it did it                                                                                |
15:45 <vitoravelino> great                                                                                |
15:45 <vitoravelino> that's good news                                                                                |
15:45 <vitoravelino> apparently just rebuilding the rpm with the proper code and docker image will fix the issue                                                                                |
15:46 <kiorky> Can you ping me when you guys trigger it ?                                                                                |
15:46 <kiorky> for me to test after                                                                                |
15:46 <vitoravelino> definitely                                                                                |
15:47 <vitoravelino> not sure if he will be able to do it today                                                                                |
15:47 <vitoravelino> but I'll ping on the issue you've opened, ok?                                                                                |
15:47 <kiorky> ok                                                                                |
15:47 <vitoravelino> thank you for your time and patience :)

16:04 <vitoravelino> my friend checked the latest docker image and it already contains the proper js file                                                                                |
16:04 <vitoravelino> could you try to pull it again?                                                                                |
16:05 <vitoravelino> cat .gitcommit                                                                                |
16:05 <vitoravelino> should return                                                                                |
16:05 <vitoravelino> 8c9b4f87f84e4f65a321f66adc5b015de67d4b93                                                                                |
16:09 <kiorky> ;2~8c9b4f87f84e4f65a321f66adc5b015de67d4b93                                                                                |
16:09 <kiorky> i think there may be a bug with the anonymous volume usage                                                                                |
16:09 <kiorky> that is not up to date                                                                                |
16:10 <vitoravelino> bash-4.3# du -h public/assets/webpack/application-83eae1419ff9e5af5a8c.js                                                                                |
16:10 <vitoravelino> 476K    public/assets/webpack/application-83eae1419ff9e5af5a8c.js                                                                                |
16:10 <vitoravelino> from a fresh image                                                                                |
16:10 <vitoravelino> and yours were ~733kb                                                                                |
16:11 <kiorky> the file isnt here                                                                                |
16:11 <kiorky> bash-4.3# ls public/assets/webpack/                                                                                |
16:11 <kiorky> application-80066879b423b63a7e7e.js  application-80066879b423b63a7e7e.js.gz  application-80066879b423b63a7e7e.js.map  application-80066879b423b63a7e7e.js.map.gz  manifest.json                                        |
16:11 <kiorky> ill try to down/up from fresh.                                                                                |
16:12 <vitoravelino> yeah, try that                                                                                |
16:38 <vitoravelino> did it work?                                                                                |
16:46 <kiorky> sorry, doing multiple things at once...                                                                                |
16:46 <vitoravelino> no worries                                                                                |
16:46 <kiorky> so docker compose down everything did not refreshed the anon volume                                                                                |
16:46 <kiorky> that's the problem in the first place i think.                                                                                |
16:46 <vitoravelino> can we close the issue?                                                                                |
16:47 <kiorky> nope without a solution :)                                                                                |
16:47 <vitoravelino> since it's not related to portus but your setup                                                                                |
16:47 <kiorky> nope                                                                                |
16:47 <kiorky> its a problem with the docker compose file of the repo                                                                                |
16:47 <kiorky> (my compose file is based on it)                                                                                |
16:49 <kiorky> see https://github.com/corpusops/Portus/blob/master/examples/compose/docker-compose.yml#L34                                                                                |
16:49 <kiorky> ups, wrong url, even if the file isnt touched: https://github.com/SUSE/Portus/blob/master/examples/compose/docker-compose.yml#L34                                                                                |
16:50 <kiorky> the first time you do dockercompose, the static volume is created from the content of /srv/Portus/public                                                                                |
16:50 <kiorky> but when you restart the containers with a newer image, i think that the static volume isnt updated with the content of the newer image                                                                                |
17:07 <vitoravelino> you can nuke that volume                                                                                |
17:07 <vitoravelino> and I guess this volume isn't necessary in the end                                                                                |
17:10 <kiorky> it is                                                                                |
17:10 <kiorky> it is because it is shared between the nginx container and portus one                                                                                |
17:11 <kiorky> nginx serve the volume which is populated by portus container.                                                                                |
17:17 <vitoravelino> I suggest you to open a specific issue related to this, I don't have much experience with this one specifically                                                                                |
17:35 <kiorky> Uhm, its more complicated.                                                                                |
17:36 <kiorky> my ticket exhibited in the first place a problem with statics, but in fact, it's because the static volume managment workflow has a flaw, and we need to refresh it. So the bug is still valid, let me rename it       |
               accordingly

docker-compose down -v fixed it for me but now there are other parts of the UI that don't work. Creating a registry returns a successful status code but I just get sent back to the "Create Registry" page. All the other views just redirect me back to "Create Registry"

With latest head, i reproduce again the bug.

Sorry for the noise, bad volume was still in place.


AsDataSet method crashes with OutOfMemoryException

Hello,
I got seven similar files, which are all between 1 and 2 MB in size. However, one of them always crashes with an OutOfMemoryException as soon as I call the AsDataSet method.
The strange thing is, that this only happens in my big project. If I add the current NuGet to a simple cmd app, it runs fine. It consumes a whole lot of RAM, though, between 700 and 800 MB for that 1,222 KB file, to be exact.
Is there a way to solve this, or at least to debug the error further?

Is the Excel file publicly available so I can debug it?

@appel1 Unfortunately it is not. It's a file for internal use only. Any chance to help me anyways?

You could build the source code and debug yourself to figure out why it allocates a lot of memory.

I might try that, as I find some time.

I have the same problem, any suggestion?? @AndreR

@amirjuve Can you send me the Excel file that triggers this so I can debug this?

@appel1  your mail please

<EMAIL_ADDRESS>

@amirjuve Thank you very much. Sounds great. Hopefully it's the exact same error and your file can help in fixing this problem. I haven't found a workaround or solution yet.

How did the testing work out? Does it work with latest develop branch? Was it .xls or .xlsx?

I never received any files that could reproduce the described issue.

This is pretty hard to diagnose without anything to go on. Closing this for now. @amirjuve @AndreR I'd be happy to hear if latest version works for you or not

https://docs.google.com/spreadsheets/d/1R4o0k2JqPzi8jwOOhQjextNg-yieWiMu72kaxjoKsUU/edit?usp=sharing

@amirjuve Thanks! I saved the linked Google Spreadsheet as an .xlsx and opened with the TestApp from ExcelDataReader's 2.1 and 3.1 repositories, but it seems to work fine for me, and no excess memory usage. Not sure what to look for here?

@andersnm  Thanks for the quick reply, can you check a .xls file to be sure.

@amirjuve  Can you try and email me these files instead? You should have my email address from when I requested permission to the documents.
If I take the file from Google Spreadsheets, it has already been imported and "fixed" by Google Docs, so I cannot diagonose these as-is.

Can you give me your ersonal mail ?
2017-08-15 15:46 GMT+02:00 andersnm<EMAIL_ADDRESS>
@amirjuve https://github.com/amirjuve Can you try and email me these
files instead? You should have my email address from when I requested
permission to the documents.
If I take the file from Google Spreadsheets, it has already been imported
and "fixed" by Google Docs, so I cannot diagonose these as-is.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ExcelDataReader/ExcelDataReader/issues/193#issuecomment-322470008,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFRlgI-avZ65dg0iSMsxxZac6gD5s6fnks5sYaE_gaJpZM4MxwCJ
.

@amirjuve I have sent a mail to you with the subject "ExcelDataReader testing" - can you please reply to that?

Thank you! It now works without all the workarounds and it doesn't crash anymore!

@AndreR, how do you solve it? Thanks in advance. I'm also facing same issue when i try to read the excel file which is around 300 MB.

@kevinkwtun When working with large workbooks it is generally prefered to use ExcelDataReaders reader methods (instead of AsDataSet) to avoid loading everything into memory. These methods work on a per-row basis, giving more control how to store the data in your app.
Alternatively you could try to pass UseColumnDataType = false as an option to AsDataSet() to halve the amount of memory used. It does not set the DataColumn.ColumnType properties, but avoids creating an internal copy of the the datatables.


Switch to non alpine node docker image to support prisma on amd64

Running
dart run electricsql_cli generate

failed with the following error on my M1 mac machine.
generate command failed: Exception: Database introspection failed
Exit code: 1
Stderr: [introspection-engine] error: Error: spawn<EMAIL_ADDRESS>ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:286:19)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn<EMAIL_ADDRESS>  path<EMAIL_ADDRESS>  spawnargs: []
}
Error: spawn<EMAIL_ADDRESS>ENOENT

Stdout: Prisma schema loaded from schema.prisma
Datasource "db": PostgreSQL database "todos_electrified", schema "public" at "localhost:65432"

- Introspecting based on datasource defined in schema.prisma
✖ Introspecting based on datasource defined in schema.prisma

Switching to the non-alpine image was enough to get it working.

@opsb Nice catch, thank you! I believe this could be the culprit https://github.com/prisma/prisma/issues/8478#issuecomment-1415834970 , but we are using version 4.8.x of Prisma, which is the same one Electric Javascript client is using for the Prisma introspection.
I guess we could use the non Alpine version, although it's much heavier.
Before merging, could you try with prisma version 4.10 and Node Alpine and see if it works? Maybe it's safe to use 4.10 for Prisma, we can double check with the Electric team if there would be any problem running the introspection with that version.

I gave 4.10.0 a try with alpine and I don't see the original error but there's a different one
electric-1  | 16:21:36.839 pid=<0.3659.0> proxy_session_id=115 [error] Received unknown prisma introspection query: "SELECT tbl.relname AS table_name, namespace.nspname as namespace, tbl.relhassubclass\nFROM pg_class AS tbl\nINNER JOIN pg_namespace AS namespace ON namespace.oid = tbl.relnamespace\nWHERE\n  ( -- grab tables when\n    -- it's an oRdinary table ('r') and is not a partition;\n    -- NOTE: CockroachDB puts NULLs in 'relispartition'\n    (tbl.relkind = 'r' AND ((tbl.relispartition is NULL) OR tbl.relispartition = 'f'))\n      OR -- when it's a partition\n    tbl.relkind = 'p'\n  )\n  AND namespace.nspname = ANY ( $1 )\nORDER BY namespace, table_name;\n" with columns ["table_name", "namespace", "tbl.relhassubclass"]

Looks like prisma has made a change which the sync engine doesn't accommodate yet (at least in the latest version of the docker build anyway).

Ok, that was my worry. The Electric Migrations Proxy is designed around a particular behavior of the Prisma introspection, which doesn't seem to be compatible with version 4.10.
Let's merge the non Alpine version then. Thank you!


add back checkout step

The v6 action was successful in my pipeline: https://github.com/leehuwuj/ragapp/actions/runs/9737834007/job/26870481351


Improved Markup Doc

Need a improvised doc with the next-stable release

yup & it is done !
documented all the functions and features and more !
check v0.9.2


Qualification tool: Add output stats file for unsupported operators

This PR contributes to https://github.com/NVIDIA/spark-rapids-tools/issues/1157 .
This PR has the following changes:

It creates a new csv file (qualification_statistics.csv) with the qualification statistics run per-sql level for unsupported operators.

Changes made:

Changes made in python user-tools to read qualification output files.
Made changes in qualification.py to create SparkQualificationStats object passing the context. The logic of creating statistics and generating a csv out file is in SparkQualificationStats.
Statistics generation can be run independent of qualification run. This is for the case where the Qual runs were completed and if we want to get stats based of that. To support this, added SparkQualStats class to read the stats argument and then create SparkQualificationStats object to generate the report.

Output:

Generate output file from user-tools cmd.

spark_rapids qualification --platform=onprem  --eventlogs=<EVENTLOGPATH>

Sample Output File: `qualification_statistics.csv`

AppId,SQLID,Operator_Name,Count,Stage Task Exec Duration(Seconds),Impacted Stage duration,% of Stage Duration,Supported(Boolean)
App ID,SQL ID,Operator Name,Count,Stage Task Exec duration(seconds),Impacted Stage duration(seconds),% of Stage Duration,Supported
app-20240220043231-0000,1,ColumnarToRow,5,1334460.475,1200.855,23.665,False
app-20240220043231-0000,5,Scan ExistingRDD,4,35.14,35.296,0.696,False
app-20240220043231-0000,14,Scan json,1,7.507,0.747,0.015,False
app-20240220043231-0000,15,ObjectHashAggregate,5,15.236,4.502,0.089,False
app-20240220043231-0000,15,Scan ExistingRDD,4,14.852,4.072,0.08,False
app-20240220043231-0000,15,filesizehistogramagg,5,15.236,4.502,0.089,False

Generate output file using python cmd(Assumption that qual run was completed earlier).

1. With `qual_output` argument
    spark_rapids stats --qual_output=/home/test/<QUAL_OUTPUT_DIRECTORY>/rapids_4_spark_qualification_output

2. `output_folder` arg where the results to be stored.
spark_rapids stats --qual_output=/home/test/<QUAL_OUTPUT_DIRECTORY>/rapids_4_spark_qualification_output --output_folder=/home/test/output_directory

Follow on

Add supported Execs to the statistics output file. This would require reading execs.csv output file. Will add it in the follow on PR.

It will be nice if we can get an initial quick feedback from @viadea to incorporate his input in the tuning of this PR.

Discussed with @viadea offline. His feedback is that having it as a part of Qualification tool should be sufficient so that the stats are produced along with the qual tool results. Asking users to run CLI  just for generating the statistics file (given the qual tool is run) wouldn't be a good experience.
@amahussein - Should we keep it as the part of Qual tool run?  Or would it help to create a framework to run this as a separate tool ? Wanted to check if you have any preference.

It will be nice if we can get an initial quick feedback from @viadea to incorporate his input in the tuning of this PR.

Discussed with @viadea offline. His feedback is that having it as a part of Qualification tool should be sufficient so that the stats are produced along with the qual tool results. Asking users to run CLI just for generating the statistics file (given the qual tool is run) wouldn't be a good experience. @amahussein - Should we keep it as the part of Qual tool run? Or would it help to create a framework to run this as a separate tool ? Wanted to check if you have any preference.

Yes, I meant that it runs as part of the Qualification tool (enabled by default).
But there is a command to allow running the stats report on existing directory path. This is what we did for "prediction". The latter is part of the processing of the qualification, but it is also possible to run prediction on existing tools folders. This is to allow getting reports on older runs and to reiterate on the stats without having to go through the entire cycle.
One last thing. How to keep compatibility? for example, it is possible that the stats is scanning a folder that was generated by older tools version. in that case, it might crash.
Any thoughts about that?

Thanks @parthosa  for the review. I have addressed all the review comments. PTAL.

thoughts on renaming Operator_Name to just be Operator.. seems shorter and matches "Unsupported Operator" in the fact if you remove Unsupported.

Stage Task Exec Duration(Seconds).

What is this based off of?   I assume our qualification tool scala logic?  Not qualx for instance.

Impacted Stage duration

Same as above - How is this calculated?

% of Stage Duration -

I assume this is just based on our guess of stage duration divided by enter stage duration?

Supported(Boolean) -

what if you have 5 projects and we support 2 or the 5?  Does this show up as another row

Statistics generation can be run independent of qualification run.

Was it a requirement from someone?   What benefits does it provide?  Anything new is going to add testing load and maintenance, as well as just another option for documentation and user to see.  If we don't have clear benefit to user or dev, I would prefer not to have it.

Was it a requirement from someone? What benefits does it provide? Anything new is going to add testing load and maintenance, as well as just another option for documentation and user to see. If we don't have clear benefit to user or dev, I would prefer not to have it.

Yes, it is PRD requirement.
We can discuss offline @nartal1 's discussion with Hao and Felix.

Stage Task Exec Duration(Seconds).
What is this based off of?   I assume our qualification tool scala logic?  Not qualx for instance.

Yes, this is based off qualification tool logic. We get these durations from rapids_4_spark_qualification_output_stages.csv .

Impacted Stage duration
Same as above - How is this calculated?

Impacted Stage duration is sum of all stage durations for the SQLID.

% of Stage Duration -
I assume this is just based on our guess of stage duration divided by enter stage duration?

This is the % of Impacted Stage Duration to the total App Duration. Now, I am not certain if this info will be useful. Or should we change the column name if it's confusing?

Supported(Boolean) -
what if you have 5 projects and we support 2 or the 5?  Does this show up as another row

Currently, we are looking into unsupported operators only so this case won't occur but once we add supported operators, we should see these in 2 different rows.

Impacted Stage duration is sum of all stage durations for the SQLID.

Stage durations here seem like they would be very hard to get accurate. I think we should think about just doing task time and potentially % total task time per sql

Stage durations here seem like they would be very hard to get accurate. I think we should think about just doing task time and potentially % total task time per sql

Looking into this if we can get % total task time per sql

Stage durations here seem like they would be very hard to get accurate. I think we should think about just doing task time and potentially % total task time per SQL

@tgravescs - Updated the PR to report % total task time.
@parthosa, @amahussein @cindyyuanjiang -  This includes all the operators where we can map stageID to Execs. PTAL.

can you update sample output in description?

can you update sample output in description?
Thanks @tgravescs ! I had updated the sample output in dropdown section earlier. Now it's in the description itself. PTAL.

Can you please describe each of these columns:

Stage Task Exec Duration(s),Total SQL Duration(s),% of Total SQL Duration,Supported
577418.50,5278814.00,10.94,True

Stage task exec duration - is the task duration for the stage in seconds
Total SQL Duration - ??? - is this the total of all task durations for all stages in that SQL id?
% of Total SQL Duration - I assume just percent based on above 2 numbers

I want to make sure the columns are clear to the user. from just reading it I would have expected the total SQL duration to just be the wall clock time but from previous discussions that isn't what we were talking about so want to make sure and then maybe come up with a name that is more obvious to the user.

It will be nice if we can get an initial quick feedback from @viadea to incorporate his input in the tuning of this PR.

Discussed with @viadea offline. His feedback is that having it as a part of Qualification tool should be sufficient so that the stats are produced along with the qual tool results. Asking users to run CLI just for generating the statistics file (given the qual tool is run) wouldn't be a good experience. @amahussein - Should we keep it as the part of Qual tool run? Or would it help to create a framework to run this as a separate tool ? Wanted to check if you have any preference.

Yes, I meant that it runs as part of the Qualification tool (enabled by default). But there is a command to allow running the stats report on existing directory path. This is what we did for "prediction". The latter is part of the processing of the qualification, but it is also possible to run prediction on existing tools folders. This is to allow getting reports on older runs and to reiterate on the stats without having to go through the entire cycle.
One last thing. How to keep compatibility? for example, it is possible that the stats is scanning a folder that was generated by older tools version. in that case, it might crash. Any thoughts about that?

ok if we want to have separate cli for users to run after the fact I guess I'm ok with it. it adds more things - documentation and testing mainly, but more commands always potential for user confusion.   please make sure we have followups to add docs/tests.
I thought this requirement was mostly from Felix  and would just get this with the normal qualification runs.  Let me talk to Felix and Hao

ok if we want to have separate cli for users to run after the fact I guess I'm ok with it. it adds more things - documentation and testing mainly, but more commands always potential for user confusion. please make sure we have followups to add docs/tests.
I thought this requirement was mostly from Felix and would just get this with the normal qualification runs. Let me talk to Felix and Hao

That's a good point @tgravescs
In fact, I agree that this is better to be an internal CLI rapids_tools_dev (should be added to dev_cli.py). The purpose of that new CLI is for the internal team to be able to run the report on existing output folders.
It is unlikely that users will need to run that CLI.

yeah I'm good with a dev command for it.  Talked to Hao and Felix and they agreed not needed the public/external command


Suggestion: Remove "no folder opened" info message

Or delay it until commands are invoked.
Right now whenever I create a new vscode window/instance this message pops up.

This should be fixed now, try updating to<EMAIL_ADDRESS>


ListView?

Hey!
Do you see any issue with using ListView over ScrollView?   I was thinking about doing something similar with a much larger list and a only one or two flips instead of the three.
I ask because I am trying the example using ListView and it is doing some funky stuff and very slow.
Not sure if I have messed something up or there is a problem else.
Thanks

Hmmm, seems to be running much better.  Maybe there was some issue going on with simulator.  ListView seems to be going ok for me now.  Thanks!


#0000 refactor variable names

Refactor: variable names, change 'vars' for 'let' and 'const' where possible

Don't remember, is it relevant or not, addFragment method is working now, the rest is not urgent, closing this PR


No ability to use FirebaseUI to link existing accounts

It appears there is no way to actually initiate the FirebaseUI screens with the explicit purpose of linking to other accounts, particularly Phone Number.
Am I missing something?

@bagintz Ha! One of these days I need to either blackmail some iOS developer into implementing this feature or learn Swift and do it myself. 😆
Ok, let me back up a little and give you some context. This feature is fully implemented on the Android side of things and there's a ton of discussion here, but no one has done much on the iOS side yet and Firebase wants to keep compatablity between the two features so the Android PR isn't going to get merged anytime soon. You can find some install instructions here and links to the documentation here.
@samtstern This issue can be closed as a duplicate of #123.

Yep :-/ thanks @SUPERCILEX

Dear @SUPERCILEX  Can I ask if the issue is resolved ? I have problem linking accounts.

Hi @samtstern  seems this issue has been resolved perfectly great by @SUPERCILEX  and still not avialble officially. Wouldn't it make sense to made the improvement avialble ? Even if IOS version is not alligned with this feature it doesn't make sense to hold the feature any longer for android users, specially since this has been avialble since 2017 or even before. Having the feature avialable will also encourage IOS team to deliver the feature sooner.... otherwise we will be waiting and waiting for an already avialble feature that is super helpful and mostly needed. Thanks in advance @samtstern  , and also thanks a lot @SUPERCILEX  for developing this super useful feature!


NICECouplingBlock.jacobian()

Set a unified form to call jacobian.
Fix the bug as below
"jacobian() got an unexpected keyword argument 'c'"
when using NICECouplingBlock along with conditions.

Thank you very much, Sir!

I should say thanks because this project helps me a lot!


helm repo add openyurt https://openyurtio.github.io/openyurt-helm  error

再使用helm添加镜像仓库时出现错误，但是添加其他的仓库是正常的 所以应该是openyurt的镜像仓库可能哪里的配置出现了错误
当前的helm版本为：
[root@node1 ~]# helm version
version.BuildInfo{Version:"v3.14.4", GitCommit:"81c902a123462fd4052bc5e9aa9c513c4c8fc142", GitTreeState:"clean", GoVersion:"go1.21.9"}
添加仓库时的具体错误为：
[root@node1 ~]# helm repo add openyurt https://openyurtio.github.io/openyurt-helm
Error: looks like "https://openyurtio.github.io/openyurt-helm" is not a valid chart repository or cannot be reached: Get "https://openyurtio.github.io/openyurt-helm/index.yaml": read tcp <IP_ADDRESS>:47358-><IP_ADDRESS>:443: read: connection reset by peer
[root@node1 ~]# helm repo add stable http://mirror.azure.cn/kubernetes/charts
"stable" has been added to your repositories
[root@node1 ~]#

@hanlovecode Thank you for raising the issue.
From the logs, It seems that there is a problem in your network that causing the openyurt-helm repo can not be added.
In my network environment, openyurt-helm repo has been added successfully as following:
MacBook-Pro:~ rambohe$ helm repo add openyurt-helm  https://openyurtio.github.io/openyurt-helm
"openyurt-helm" has been added to your repositories

by the way, repo: https://openyurtio.github.io/charts is recommended to use from OpenYurt v1.5.0, please enjoy it.

@hanlovecode I think that the problem of this issue has been solved, so i will close this issue.


Reading records fails

Describe the bug
When i try to see records of a table, whatever the way, i can't, and here is the error :
[1717968897729] ERROR (TypeError):
TypeError: Cannot read properties of undefined (reading 'total')
at li. (/home/vscode/.vscode-server/extensions/mtxr.sqltools-driver-pg-0.5.4/out/ls/plugin.js:118:17015)
at Generator.next ()
at a (/home/vscode/.vscode-server/extensions/mtxr.sqltools-driver-pg-0.5.4/out/ls/plugin.js:118:15544)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[1717968897730] ERROR (ext): ERROR: Execute query error Cannot read properties of undefined (reading 'total'), 'Cannot read properties of undefined (reading 'total')'
ns: "error-handler"
[1717968897730] ERROR (ext): ERROR: Error while showing table records Cannot read properties of undefined (reading 'total'), {"code":-1,"data":{"driver":"PostgreSQL","driverOptions":{}}}
ns: "error-handler"
And a window open with infinite loading screen
To Reproduce
Steps to reproduce the behavior:

Go to SQLTools extension tab, on any table you want
Click on show table records
See error

Expected behavior
See the actuals records of the table
Screenshots

Desktop (please complete the following information):

SQLTools Version [e.g. v0.17.x] v0.28.3
VS Code Version: [e.g 1.x.y] v1.90.0
OS: [e.g. Windows, Linux, Mac] Windows (in devcontainer debian)
Driver:

[X] PostgreSQL/Redshift v0.5.4
[ ] MySQL/MariaDB
[ ] MSSQL/Azure
[ ] SQLite
[ ] Other? Which...

Database version: [e.g. MySQL v5.6, PostgreSQL vX...] Postgresql 16

Additional context
Add any other context about the problem here.

I'm also having this issue. Using Neon database if that matters/helps with context


Implement registration flow

may i ask what are extraData, manager, clientId? how can i get those and pass them to register func? thanks
my steps:

get captcha token
did.services.getRecommendationVerifier({ chainId: 'AELF' }) -> verifier
did.services.getVerificationCode()
did.services.verifyVerificationCode()
did.services.register

may i ask what are extraData, manager, clientId? how can i get those and pass them to register func? thanks my steps:

get captcha token
did.services.getRecommendationVerifier({ chainId: 'AELF' }) -> verifier
did.services.getVerificationCode()
did.services.verifyVerificationCode()
did.services.register

hi, @thiendangvan
extraData:  including login environment、device、login time, this is an optional parameter.
clientId: the unique device ID, used as a parameter for socket queries.
manager: the manager address. you can get managerAddress like below code:
const manager = did.didWallet.create();
const managerAddress = wallet.managementAccount!.address;

may i ask what are extraData, manager, clientId? how can i get those and pass them to register func? thanks my steps:

get captcha token
did.services.getRecommendationVerifier({ chainId: 'AELF' }) -> verifier
did.services.getVerificationCode()
did.services.verifyVerificationCode()
did.services.register

hi, @thiendangvan
extraData: including login environment、device、login time, this is an optional parameter. clientId: the unique device ID, used as a parameter for socket queries. manager: the manager address. you can get managerAddress like below code:
const managerAddress = wallet.managementAccount!.address;

hi @wade-portkey, i've tried but it seems i can only get the managementAccount?.address after i call did.didWallet.create method? if not i got undefined
const account = did.didWallet.create() --> required?
console.log({ account })
console.log(111, did.didWallet.managementAccount?.address)

may i ask what are extraData, manager, clientId? how can i get those and pass them to register func? thanks my steps:

get captcha token
did.services.getRecommendationVerifier({ chainId: 'AELF' }) -> verifier
did.services.getVerificationCode()
did.services.verifyVerificationCode()
did.services.register

hi, @thiendangvan
extraData: including login environment、device、login time, this is an optional parameter. clientId: the unique device ID, used as a parameter for socket queries. manager: the manager address. you can get managerAddress like below code:
const managerAddress = wallet.managementAccount!.address;

hi @wade-portkey, i've tried but it seems i can only get the managementAccount?.address after i call did.didWallet.create method? if not i get undefined
const account = did.didWallet.create() --> required?
console.log({ account })
console.log(111, did.didWallet.managementAccount?.address)

hi, @thiendangvan , you are right. you should call did.didWallet.create() first, then you can get wallet.managementAccount!.address.

@wade-portkey, i got below error when register with extraData = ""
register {"error":{"code":null,"message":"Your request is not valid!","details":"The following errors were detected during validation.\n - The ExtraData field is required.\n","data":{},"validationErrors":[{"message":"The ExtraData field is required.","members":["extraData"]}]},"status":400}

const registerResult = await did.services
        .register({
          type: 'Email',
          loginGuardianIdentifier: email,
          extraData: '',
          chainId: 'AELF',
          verifierId: verifier.id,
          verificationDoc,
          signature,
          context: {
            clientId: '123456',
            requestId: nanoid(),
          },
          manager: did.didWallet.managementAccount!.address,
        })
        .catch(e => console.log('register', JSON.stringify(e)))

@wade-portkey, i got below error when register with extraData = ""
register {"error":{"code":null,"message":"Your request is not valid!","details":"The following errors were detected during validation.\n - The ExtraData field is required.\n","data":{},"validationErrors":[{"message":"The ExtraData field is required.","members":["extraData"]}]},"status":400}

const registerResult = await did.services
        .register({
          type: 'Email',
          loginGuardianIdentifier: email,
          extraData: '',
          chainId: 'AELF',
          verifierId: verifier.id,
          verificationDoc,
          signature,
          context: {
            clientId: '123456',
            requestId: nanoid(),
          },
          manager: did.didWallet.managementAccount!.address,
        })
        .catch(e => console.log('register', JSON.stringify(e)))

@thiendangvan sorry for misled you yesterday. you can try code like below:
const extraData = JSON.stringify({
    deviceInfo: {
      deviceName: 'your device name';
      deviceType?: 'OTHER or MAC or IOS or WINDOWS or ANDROID';
    },
    transactionTime: Date.now(),
  });

and pass the extraData to register function


Event: Teresa Dietrich quit

What type of event change are you requesting

[x] Add new event
[ ] Update event details
[ ] Remove event

{
  "date_str": "2023-05-01",
  "title": "Teresa Dietrich left Stack Overflow",
  "slug": "teresa-dietrich-left-stack-overflow",
  "type": "staff resignation",
  "summary": "Chief Product Officer, formerly also Chief Technology Officer",
  "body": "
> Teresa’s last day with the company was Friday, April 21st. In the 3+ years that she was here, Teresa had a huge impact on the company and the website. In addition to the Product team, Teresa had the Community team under her purview, and for more than half of her tenure she also acted as Chief Technology Officer.
",
  "classes": [
    "tag-staff",
    "tag-resignation"
  ],
  "links": [
    {
      "text": "MSE post",
      "url": "https://meta.stackexchange.com/questions/388800/announcement-teresa-dietrich-leaves-stack"
    }
  ]
},

The body HTML looks incorrect, since to make a blockquote you need to use the
 tag instead of starting the line with > , and the span contained pasted styles from an external editor (MS Word or Google Docs).
If the event body HTML is complex, please try using the Event Editor: https://se-timeline.glitch.me/event-editor
I'll get the body fixed this time. Thanks for contributing!


Is heredoc correct?

I wonder if this heredoc works in line 61 https://github.com/hypriot/image-builder-rpi/blob/master/build.sh#L61
chroot ${BUILD_PATH} /bin/bash <<"EOCHROOT"

as it should expand variables for FSTAB in line 84 https://github.com/hypriot/image-builder-rpi/blob/master/build.sh#L84
  echo ${FSTAB} > /etc/fstab

I thought expansion needs a heredoc without the quotes.
I'll test inside a vagrant box.

The heredoc has to be this way with chroot! So, that's absolute correct in  https://github.com/hypriot/image-builder-rpi/blob/master/build.sh#L61. Without the quotes in <<"EOCHROOT" you'll get an error.

So we can pin all the kernel packages by using the KERNEL_* variables, that part is missing ;-)

I have added a test to proof that /etc/fstab is correct in #3

@DieterReuter Are you sure? The test in build https://travis-ci.org/hypriot/image-builder-rpi/builds/101444491 says "empty" /etc/fstab.

resolved in the meantime