Windows: sync stops at 99%+ > node is disconnected

Preflight Checklist

[rene-sch ] I have searched DeFiChain’s Github Issue Page and none matches my issue
[rene-sch] I have read through How To Submit Issues for DeFi App

Issue Details

Current DeFi App Version:

 2.1.10

Previous DeFi App Version:

 2.1.3

Operating System (OS):

 Windows 10

Screenshot(s) / screen recording(s) of issue

Actual Behavior

I wanted to update the app from version 2.1.3 to 2.1.10.
After the synchronization was at 98.89% the error came: "Connection to the node disconnected"
Then I increased the swap file on my Windows 10 (12 GB RAM). After that I tried several times but always at 98.89% the message Connection to the node was disconnected
Expected Behavior

I expected the app to work properly after installation
App log / binary log / app config

LogData.zip

attached a screenshot

attached a screenshot

Note: I had the same problem with the update from 2.1.3 to 2.1.9

Note: I had the same problem with the update from 2.1.3 to 2.1.9

Here are the right logs, sorry for that
App Config.zip

Here are the right logs, sorry for that
App Config.zip

For me it works with every systems when I do it the following:

Start the defid.exe without the app

Open cmd and type
.\defid.exe -conf="%USERPROFILE%\.defi\defid.conf"

Observe the RAM consumption of the app in the taskmanager. When it exceeds 1 GB (with your amount of RAM, you can wait until it exceeds 2GB - 4GB) select the cmd window and press Ctrl + C to shutdown the node. Do not close the cmd window with "X", the blockchain can be corrupted
Wait until you can enter the next command and start from step 1. Check whether it syncs from your last synced block number.

Remember you need to start it with the parameter -reindex the first time ( .\defid.exe -conf="%USERPROFILE%\.defi\defid.conf" -reindex ) like described in the release notes.
Why it helps?
defid.exe will need about 300 MB of RAM when you restart the process, but can already use the last blockchain state. Then it will increase again (I could see increasing it to almost 14GB RAM usage without any restart). The first 500.000 blocks can be synced without problems, then you will have to restart the sync process more often.

For me it works with every systems when I do it the following:

Start the defid.exe without the app

Open cmd and type
.\defid.exe -conf="%USERPROFILE%\.defi\defid.conf"

Observe the RAM consumption of the app in the taskmanager. When it exceeds 1 GB (with your amount of RAM, you can wait until it exceeds 2GB - 4GB) select the cmd window and press Ctrl + C to shutdown the node. Do not close the cmd window with "X", the blockchain can be corrupted
Wait until you can enter the next command and start from step 1. Check whether it syncs from your last synced block number.

Remember you need to start it with the parameter -reindex the first time ( .\defid.exe -conf="%USERPROFILE%\.defi\defid.conf" -reindex ) like described in the release notes.
Why it helps?
defid.exe will need about 300 MB of RAM when you restart the process, but can already use the last blockchain state. Then it will increase again (I could see increasing it to almost 14GB RAM usage without any restart). The first 500.000 blocks can be synced without problems, then you will have to restart the sync process more often.

Hi @rene-sch , thanks for your feedback. Please give us time to investigate this issue. Thank you.

Hi @rene-sch , thanks for your feedback. Please give us time to investigate this issue. Thank you.

Hi everyone,
I have the same problem with 2.1.10.
Maybe it's important to point out that this message appears when the synchronisation is almost finished.

Hi everyone,
I have the same problem with 2.1.10.
Maybe it's important to point out that this message appears when the synchronisation is almost finished.

Hi @BenThrottle85 , thanks for your feedback. This error occur because there's a potential bug, not because synchronisation is almost complete. The expected behaviour should be that it syncs to 100% with no error. Could you send your logs please?

Hi @BenThrottle85 , thanks for your feedback. This error occur because there's a potential bug, not because synchronisation is almost complete. The expected behaviour should be that it syncs to 100% with no error. Could you send your logs please?

Of course...
AppLogs.zip
BinaryLogs.zip

Of course...
AppLogs.zip
BinaryLogs.zip

Hi @BenThrottle85 , well received, please give us time to investigate. Thank you.

Hi @BenThrottle85 , well received, please give us time to investigate. Thank you.

I have the same problem with 2.1.10.
App Logs.zip
App Config.zip

I have the same problem with 2.1.10.
App Logs.zip
App Config.zip

Hi @kingsersch , well received. The team is currently investigating this issue.

Hi @kingsersch , well received. The team is currently investigating this issue.

Hello,
so i had the same problem on windows since 2.1.6. i think.
The connection to node stuff or the app only showing a white screen.
Two or three times it helped to delete the folder "C:\Users\YOUR_USERNAME\AppData\Roaming\DeFi Blockchain".
With 2.1.9 i was not able to get the App running again. I deleted every folder with defi stuff, but not working.
My solution now is, i have the wallet app running permanently on a virtual machine with linux mint.
Running like a charm, i had a power shortout once. And the NAS with the VM failed.
 So restored the last Snapshot from 2 days before.
App synced without problems.
Setting up a VM on a Windows PC is not a problem for someone that can google and it seems the linux version is not so buggy.
Please Team before you add any features or whatever debug that damn App.
Nevertheless DFI is great!!!
Love you guys :-D

Hello,
so i had the same problem on windows since 2.1.6. i think.
The connection to node stuff or the app only showing a white screen.
Two or three times it helped to delete the folder "C:\Users\YOUR_USERNAME\AppData\Roaming\DeFi Blockchain".
With 2.1.9 i was not able to get the App running again. I deleted every folder with defi stuff, but not working.
My solution now is, i have the wallet app running permanently on a virtual machine with linux mint.
Running like a charm, i had a power shortout once. And the NAS with the VM failed.
 So restored the last Snapshot from 2 days before.
App synced without problems.
Setting up a VM on a Windows PC is not a problem for someone that can google and it seems the linux version is not so buggy.
Please Team before you add any features or whatever debug that damn App.
Nevertheless DFI is great!!!
Love you guys :-D

I had the samel isse and for me it was simply the hardware space had 12 GB free but this was not enough i have now 150 GB and now it works

I had the samel isse and for me it was simply the hardware space had 12 GB free but this was not enough i have now 150 GB and now it works

@maaze19 i agree that restarting node will fix some higher ram usage, which version you use? I can't see more than 1 GiB in use during sync?

@maaze19 i agree that restarting node will fix some higher ram usage, which version you use? I can't see more than 1 GiB in use during sync?

Same issue synchronizing with 2.1.11 update.

Apps-log.zip

Same issue synchronizing with 2.1.11 update.

Apps-log.zip

How much memory you have?

How much memory you have?

8G of RAM, 312G disk space.

8G of RAM, 312G disk space.

Can you confirm that you have a bit more wallet addresses ?

Can you confirm that you have a bit more wallet addresses ?

This was my wallet address in the version 2.1.10. I could not get past the synchronization issue in version 2.1.11 to see my wallet.

This was my wallet address in the version 2.1.10. I could not get past the synchronization issue in version 2.1.11 to see my wallet.

I updated to 2.1.12, but the issue still not resolved. Can't view my wallet, can't do anything.
App_logs.zip

I updated to 2.1.12, but the issue still not resolved. Can't view my wallet, can't do anything.
App_logs.zip

@thiencdam can provide screenshot with open task manager around 580k block how much memory is used? From the logs you don't have enough memory.

@thiencdam can provide screenshot with open task manager around 580k block how much memory is used? From the logs you don't have enough memory.

I had the same problem and i have 32 Gb of Ram, so in my opinion there is a bug that causes the ram to run out.
Not the other way round.
Right now it's running smootly in an Linux VM with 4 GB Ram, no problems so far.

I had the same problem and i have 32 Gb of Ram, so in my opinion there is a bug that causes the ram to run out.
Not the other way round.
Right now it's running smootly in an Linux VM with 4 GB Ram, no problems so far.

Here is the snap shot, with task manager, where synchronizing stalled.

Here is the snap shot, with task manager, where synchronizing stalled.

Updated to 2.1.13 on Windows 10. No improvement. Synchronizing stalled at block 591203.
App_binary.zip

Updated to 2.1.13 on Windows 10. No improvement. Synchronizing stalled at block 591203.
App_binary.zip

Hi @rene-sch @BenThrottle85 @kingsersch @FrostyEraser can I check that you've updated to v2.1.13? Please send new information (include screenshots) and app logs. Thank you.
Hi @thiencdam well received, please give us time to investigate the issue.

Hi @rene-sch @BenThrottle85 @kingsersch @FrostyEraser can I check that you've updated to v2.1.13? Please send new information (include screenshots) and app logs. Thank you.
Hi @thiencdam well received, please give us time to investigate the issue.

Link to #372 and #352

Link to #372 and #352

same issue with Windows 8.1 and  V 2.1.13- see logs attached
DEFI-Chain.zip

same issue with Windows 8.1 and  V 2.1.13- see logs attached
DEFI-Chain.zip

@goldi147  it's needed at least 1GiB of RAM to run node.
@thiencdam did you disable pagefile (swap) or virtual memory?

@goldi147  it's needed at least 1GiB of RAM to run node.
@thiencdam did you disable pagefile (swap) or virtual memory?

Maybe I ran out of memory, but 4,6 gb is a huge number?
Is it better to install an older version? All my friends also have problems with 2.1.13.
If so please sen last stabil exe as link.
I also tried to run it in the cmd box in batches. After hours I got synchron, but after the next restart database (or synonym) has been corrupt. Now I try the Sync via programm again. It is running for 3-4 hours now.
Many thanks for your help!
<EMAIL_ADDRESS>Von: Anthony Fieroni<EMAIL_ADDRESS>Gesendet: Donnerstag, 28. Jänner 2021 08:53
An: DeFiCh/app<EMAIL_ADDRESS>Cc: Mario Gold<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Betreff: Re: [DeFiCh/app] Windows: sync stops at 99%+ > node is disconnected (#325)
@goldi147https://github.com/goldi147 it's needed at least 1GiB of RAM to run node.
@thiencdamhttps://github.com/thiencdam did you disable pagefile (swap) or virtual memory?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/DeFiCh/app/issues/325#issuecomment-768869057, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASJYEUKFGOVQYM4DZ64NPRDS4EJUDANCNFSM4WGMOCMA.

Maybe I ran out of memory, but 4,6 gb is a huge number?
Is it better to install an older version? All my friends also have problems with 2.1.13.
If so please sen last stabil exe as link.
I also tried to run it in the cmd box in batches. After hours I got synchron, but after the next restart database (or synonym) has been corrupt. Now I try the Sync via programm again. It is running for 3-4 hours now.
Many thanks for your help!
<EMAIL_ADDRESS>Von: Anthony Fieroni<EMAIL_ADDRESS>Gesendet: Donnerstag, 28. Jänner 2021 08:53
An: DeFiCh/app<EMAIL_ADDRESS>Cc: Mario Gold<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Betreff: Re: [DeFiCh/app] Windows: sync stops at 99%+ > node is disconnected (#325)
@goldi147https://github.com/goldi147 it's needed at least 1GiB of RAM to run node.
@thiencdamhttps://github.com/thiencdam did you disable pagefile (swap) or virtual memory?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/DeFiCh/app/issues/325#issuecomment-768869057, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASJYEUKFGOVQYM4DZ64NPRDS4EJUDANCNFSM4WGMOCMA.

I switched to V2.1.8 and got synchronised. But now when I try the add liquidity I get following massage:
<EMAIL_ADDRESS>Attached you find my logs as well.
Von: Anthony Fieroni<EMAIL_ADDRESS>Gesendet: Donnerstag, 28. Januar 2021 08:53
An: DeFiCh/app<EMAIL_ADDRESS>Cc: Mario Gold<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Betreff: Re: [DeFiCh/app] Windows: sync stops at 99%+ > node is disconnected (#325)
@goldi147https://github.com/goldi147 it's needed at least 1GiB of RAM to run node.
@thiencdamhttps://github.com/thiencdam did you disable pagefile (swap) or virtual memory?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/DeFiCh/app/issues/325#issuecomment-768869057, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASJYEUKFGOVQYM4DZ64NPRDS4EJUDANCNFSM4WGMOCMA.

I switched to V2.1.8 and got synchronised. But now when I try the add liquidity I get following massage:
<EMAIL_ADDRESS>Attached you find my logs as well.
Von: Anthony Fieroni<EMAIL_ADDRESS>Gesendet: Donnerstag, 28. Januar 2021 08:53
An: DeFiCh/app<EMAIL_ADDRESS>Cc: Mario Gold<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Betreff: Re: [DeFiCh/app] Windows: sync stops at 99%+ > node is disconnected (#325)
@goldi147https://github.com/goldi147 it's needed at least 1GiB of RAM to run node.
@thiencdamhttps://github.com/thiencdam did you disable pagefile (swap) or virtual memory?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/DeFiCh/app/issues/325#issuecomment-768869057, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASJYEUKFGOVQYM4DZ64NPRDS4EJUDANCNFSM4WGMOCMA.

I did not disable pagefile (swap) or virtual memory. Yesterday, I uninstalled DefiApp and increased virtual memory to 32Gb, then reinstalled V2.1.13.  I don't know which step fixed it but it get synched and is now running beautifully.  I backed-up my wallet just in case. It did take a while to synchronize around block 591248.

Good luck to everyone.

I did not disable pagefile (swap) or virtual memory. Yesterday, I uninstalled DefiApp and increased virtual memory to 32Gb, then reinstalled V2.1.13.  I don't know which step fixed it but it get synched and is now running beautifully.  I backed-up my wallet just in case. It did take a while to synchronize around block 591248.

Good luck to everyone.

One thing I forgot to mention, I did restart the computer to clear the registry before reinstalling the app.

One thing I forgot to mention, I did restart the computer to clear the registry before reinstalling the app.

when I sync via cmd:
cd "%localappdata%\Programs\defi-app\resources\binary\win"
.\defid.exe -conf="%USERPROFILE%.defi\defid.conf"
and stop with "ctrl+c" when I ran out of 4 gb memory I get synchonized.
but after the first or second restart I get errors again.

when I sync via cmd:
cd "%localappdata%\Programs\defi-app\resources\binary\win"
.\defid.exe -conf="%USERPROFILE%.defi\defid.conf"
and stop with "ctrl+c" when I ran out of 4 gb memory I get synchonized.
but after the first or second restart I get errors again.

I reindexed 4 times no improvement. I am stuck. How to fix it. It just stops at 99% and says node disconnected.

I reindexed 4 times no improvement. I am stuck. How to fix it. It just stops at 99% and says node disconnected.

Hi @rene-sch @BenThrottle85 @kingsersch @goldi147 @jibesmaster , update v2.1.15 is out.
 Thank you.

Hi @rene-sch @BenThrottle85 @kingsersch @goldi147 @jibesmaster , update v2.1.15 is out.
 Thank you.

Node disconnected issue continues to occur unabated.
Logs attached as requested, as is confirmatory screenshot.
binary_log.log
app_log.log

Node disconnected issue continues to occur unabated.
Logs attached as requested, as is confirmatory screenshot.
binary_log.log
app_log.log

Hi @izzycsy
I updated from Version 2.1.13 to 2.1.15 and got a "Error: Corrupted block database detected" as described in #390 . During the following synchronizing I get the issue described above. I retried  3 times already and will keep doing it. Please find Log files and screenshot attached.

NodeDiscoLogs.zip

Hi @izzycsy
I updated from Version 2.1.13 to 2.1.15 and got a "Error: Corrupted block database detected" as described in #390 . During the following synchronizing I get the issue described above. I retried  3 times already and will keep doing it. Please find Log files and screenshot attached.

NodeDiscoLogs.zip

@izzycsy
After several tries (lazy Sunday) I found a solution for me:
The procedure is similar to the one from user maaze19 without the use of cmd. I have a 16 GB RAM machine with SSD and 2x2,4 GHz CPUs and version is 2.1.15. While synchronizing I monitored the RAM and closed the app when 4 GB or 45-50% of total RAM usage was reached.. I had to do this two times and synchronizing started with the blocks from previous closing and lower RAM usage. Now I'm fully synchronized and all works fine so far.

@izzycsy
After several tries (lazy Sunday) I found a solution for me:
The procedure is similar to the one from user maaze19 without the use of cmd. I have a 16 GB RAM machine with SSD and 2x2,4 GHz CPUs and version is 2.1.15. While synchronizing I monitored the RAM and closed the app when 4 GB or 45-50% of total RAM usage was reached.. I had to do this two times and synchronizing started with the blocks from previous closing and lower RAM usage. Now I'm fully synchronized and all works fine so far.

Hi @Ausafari , well received, please give us time to investigate this issue. Do you have screenshot of your Activity Monitor?

Hi @Ausafari , well received, please give us time to investigate this issue. Do you have screenshot of your Activity Monitor?

I use a PC, not a Mac, so the Activity Monitor doesn't provide the information a Mac does. However I can take a screenshot of Task Manager, if that will help? I am assuming you want such a screenshot from when the error occurs?

I use a PC, not a Mac, so the Activity Monitor doesn't provide the information a Mac does. However I can take a screenshot of Task Manager, if that will help? I am assuming you want such a screenshot from when the error occurs?

Hi @Ausafari , yes for windows PC it's your task manager.

Hi @Ausafari , yes for windows PC it's your task manager.

OK, but you didn't answer as to when you want the screenshot taken, as asked previously. Is it when the app is trying to sync or when the app fails to sync?

OK, but you didn't answer as to when you want the screenshot taken, as asked previously. Is it when the app is trying to sync or when the app fails to sync?

The app failed again with the "Node is disconnected" error being thrown again before I received an answer from you @izzycsy . There's no point sending you the Task Manager output at this time as it's showing zero activity from the defi app of course.

The app failed again with the "Node is disconnected" error being thrown again before I received an answer from you @izzycsy . There's no point sending you the Task Manager output at this time as it's showing zero activity from the defi app of course.

Hi guys, I was facing the same issue (Node is disconnected). Then I removed all files and folders in the folder DeFi Blockchain. Now the problem is solved, my wallet is synchronized (at block 611616), but I lost my balance of DFI coins. What should I do in order to see my previous balance, please? Thanks in advance!

Hi guys, I was facing the same issue (Node is disconnected). Then I removed all files and folders in the folder DeFi Blockchain. Now the problem is solved, my wallet is synchronized (at block 611616), but I lost my balance of DFI coins. What should I do in order to see my previous balance, please? Thanks in advance!

Hi guys, I was facing the same issue (Node is disconnected). Then I removed all files and folders in the folder DeFi Blockchain. Now the problem is solved, my wallet is synchronized (at block 611616), but I lost my balance of DFI coins. What should I do in order to see my previous balance, please? Thanks in advance!

Have you restored your wallet with your seed phrase?

Hi guys, I was facing the same issue (Node is disconnected). Then I removed all files and folders in the folder DeFi Blockchain. Now the problem is solved, my wallet is synchronized (at block 611616), but I lost my balance of DFI coins. What should I do in order to see my previous balance, please? Thanks in advance!

Have you restored your wallet with your seed phrase?

Hi guys, I was facing the same issue (Node is disconnected). Then I removed all files and folders in the folder DeFi Blockchain. Now the problem is solved, my wallet is synchronized (at block 611616), but I lost my balance of DFI coins. What should I do in order to see my previous balance, please? Thanks in advance!

Have you restored your wallet with your seed phrase?

No, I didn`t, how should I do that?

Hi guys, I was facing the same issue (Node is disconnected). Then I removed all files and folders in the folder DeFi Blockchain. Now the problem is solved, my wallet is synchronized (at block 611616), but I lost my balance of DFI coins. What should I do in order to see my previous balance, please? Thanks in advance!

Have you restored your wallet with your seed phrase?

No, I didn`t, how should I do that?

After reindexing still the same error "Node is disconnected"
Now I try to delete the .defi folder.

After reindexing still the same error "Node is disconnected"
Now I try to delete the .defi folder.

Hi guys, I was facing the same issue (Node is disconnected). Then I removed all files and folders in the folder DeFi Blockchain. Now the problem is solved, my wallet is synchronized (at block 611616), but I lost my balance of DFI coins. What should I do in order to see my previous balance, please? Thanks in advance!

Have you restored your wallet with your seed phrase?

No, I didn`t, how should I do that?

Maybe you can upload a screenshot from your current screen?!
Have you made a backup of your wallet?

Hi guys, I was facing the same issue (Node is disconnected). Then I removed all files and folders in the folder DeFi Blockchain. Now the problem is solved, my wallet is synchronized (at block 611616), but I lost my balance of DFI coins. What should I do in order to see my previous balance, please? Thanks in advance!

Have you restored your wallet with your seed phrase?

No, I didn`t, how should I do that?

Maybe you can upload a screenshot from your current screen?!
Have you made a backup of your wallet?

hi there, I am facing the same issue.
I'm using macbook and version 2.1.15 of DeFi Chain app
It syncs up to over 99% then it gives the error
How can I fix it?

hi there, I am facing the same issue.
I'm using macbook and version 2.1.15 of DeFi Chain app
It syncs up to over 99% then it gives the error
How can I fix it?

Hi @izzycsy . I have attached two screenshots for you of the Task Manager output when the app fails. Each shows the different failure notices down the bottom right hand corner. From Task Manager you can see the system resources are not overloaded - the app simply fails - each and every time.

Hi @izzycsy . I have attached two screenshots for you of the Task Manager output when the app fails. Each shows the different failure notices down the bottom right hand corner. From Task Manager you can see the system resources are not overloaded - the app simply fails - each and every time.

hi there, I am facing the same issue.
I'm using macbook and version 2.1.15 of DeFi Chain app
It syncs up to over 99% then it gives the error
How can I fix it?

Update or fresh install?

hi there, I am facing the same issue.
I'm using macbook and version 2.1.15 of DeFi Chain app
It syncs up to over 99% then it gives the error
How can I fix it?

Update or fresh install?

I (and I suspect others in this thread) @BenThrottle85 have tried both methods numerous times - all without success. :(

I (and I suspect others in this thread) @BenThrottle85 have tried both methods numerous times - all without success. :(

Hi guys, I was facing the same issue (Node is disconnected). Then I removed all files and folders in the folder DeFi Blockchain. Now the problem is solved, my wallet is synchronized (at block 611616), but I lost my balance of DFI coins. What should I do in order to see my previous balance, please? Thanks in advance!

Have you restored your wallet with your seed phrase?

No, I didn`t, how should I do that?

Maybe you can upload a screenshot from your current screen?!
Have you made a backup of your wallet?

I have made a backup of my wallet, but unfortunately I cannot find the file.. But I have my recovery phase.. Im 100% synchronized now, how can I get back my coins? Im sure, I can find the address where I sent a coins from exchange. Does it help?

Hi guys, I was facing the same issue (Node is disconnected). Then I removed all files and folders in the folder DeFi Blockchain. Now the problem is solved, my wallet is synchronized (at block 611616), but I lost my balance of DFI coins. What should I do in order to see my previous balance, please? Thanks in advance!

Have you restored your wallet with your seed phrase?

No, I didn`t, how should I do that?

Maybe you can upload a screenshot from your current screen?!
Have you made a backup of your wallet?

I have made a backup of my wallet, but unfortunately I cannot find the file.. But I have my recovery phase.. Im 100% synchronized now, how can I get back my coins? Im sure, I can find the address where I sent a coins from exchange. Does it help?

I have just found a solution and it worked, it might not be an efficient one but I think for those whom are worried about their funds it can be a solution
I opened the app and once the percentage was going up I closed the app then open again for syncing, it took me a while more to finish the process but it did sync.

I have built on the work @UserMaxL did earlier and seem to have found a reliable method of making this app (the most unreliable DeFI app it has EVER been my sorry misfortune to use) sync.
1: Note the approximate block the app reaches when it throws the "Node disconnected" error (still occurring for all of we poor unfortunates after 6 days! Hang your heads in shame, app devs!)
2:  Launch the app.
3: Monitor it as it approaches the previous failure point. The block periods seem to oscillate around an (approximately) 20 second cycle.
4: As soon as you notice the block count stop advancing (give it say 30 seconds, just to be sure), close the app out. DO NOT wait for the app to fail.
5: Wait for the app to close completely before proceeding.
6: Relaunch the app - it may take a few minutes to relaunch, but be patient. Once it does relaunch, it should start again near or at the block where it left off.
7: Repeat steps 4-6 as necessary. (I had to close and re-launch the app three times before it finally synced).
8: When the app finally syncs, give a great big sigh of relief and continue on with the tasks you should reasonably have expected to be able  to have completed 6 DAYS AGO!!!!

OK, so my issue is solved. I have removed the app and deleted both folders (DeFi Blockchain and defi-app). Installed a newest version 2.1.15, wait for synchronizing and recovered with my seed phrase. All works! Thanks for your help.

I deleted the ".defi" folder on my Lubuntu VM:

syncing starts --> 100% with no problems. used memory was about 500MB all the time but it was much slower than the last times. but that's not the problem
app shows me my wallets but no coins displayed. but how does the app know what coins i had? i deleted the ".defi" folder!
Wallet -> Reset Wallet -> entered my seed phrase -> restoring wallet. my coins are back
opened DFI wallet -> "Include rewards" -> no rewards shown :-(
Reindexing ==> same issue again. the used memoy at about 90% goes up to the air.

I'm really frustrated right now!!!

OK, so my issue is solved. I have removed the app and deleted both folders (DeFi Blockchain and defi-app). Installed a newest version 2.1.15, wait for synchronizing and recovered with my seed phrase. All works! Thanks for your help.

Can you see your rewards in the dfi wallet?

OK, so my issue is solved. I have removed the app and deleted both folders (DeFi Blockchain and defi-app). Installed a newest version 2.1.15, wait for synchronizing and recovered with my seed phrase. All works! Thanks for your help.

Can you see your rewards in the dfi wallet?

Yes, I can.

For me it works with every systems when I do it the following:

Start the defid.exe without the app

Open cmd and type
cd "%localappdata%\Programs\defi-app\resources\binary\win"
.\defid.exe -conf="%USERPROFILE%\.defi\defid.conf"

Observe the RAM consumption of the app in the taskmanager. When it exceeds 1 GB (with your amount of RAM, you can wait until it exceeds 2GB - 4GB) select the cmd window and press Ctrl + C to shutdown the node. Do not close the cmd window with "X", the blockchain can be corrupted
Wait until you can enter the next command and start from step 1. Check whether it syncs from your last synced block number.

Remember you need to start it with the parameter -reindex the first time ( .\defid.exe -conf="%USERPROFILE%\.defi\defid.conf" -reindex ) like described in the release notes.
Why it helps?
defid.exe will need about 300 MB of RAM when you restart the process, but can already use the last blockchain state. Then it will increase again (I could see increasing it to almost 14GB RAM usage without any restart). The first 500.000 blocks can be synced without problems, then you will have to restart the sync process more often.

Hi @rene-sch @BenThrottle85 @kingsersch @FrostyEraser can I check that you've updated to v2.1.13? Please send new information (include screenshots) and app logs. Thank you.
Hi @thiencdam well received, please give us time to investigate the issue.

sorry for my late reply but i'm recovering from corona.
I still have problems finding the passage as described by maaze19 "Remember you need to start it with the parameter -reindex the first time (. \ defid.exe -conf ="% USERPROFILE% . defi \ defid.conf "-reindex ) like described in the release notes. "
I would then go through the path he described

@rene-sch
Try my procedure - I didn't fully get the maaze19 description, but I tried something similar:
While synchronizing I monitored the RAM usage in the Task Manager and closed the app when 4 GB or 45-50% of total RAM usage was reached ( I have a 16 GB RAM machine with SSD and 2x2,4 GHz CPUs and version is 2.1.15.). I had to do this two times and synchronizing started with the blocks from previous closing and lower RAM usage. It fully synchronized and all works fine so far.

@UserMaxL Thank you for your quick response. I've tried your approach. but when I close the app the synchronization starts at 0.
Do you close the app normally with "X"? and do you restart by double-clicking?

I deleted the ".defi" folder on my Lubuntu VM:

syncing starts --> 100% with no problems. used memory was about 500MB all the time but it was much slower than the last times. but that's not the problem
app shows me my wallets but no coins displayed. but how does the app know what coins i had? i deleted the ".defi" folder!
Wallet -> Reset Wallet -> entered my seed phrase -> restoring wallet. my coins are back
opened DFI wallet -> "Include rewards" -> no rewards shown :-(
Reindexing ==> same issue again. the used memoy at about 90% goes up to the air.

I'm really frustrated right now!!!

I created a new Virtual Machine and installed a fresh copy of the Defichain-App 2.1.15.
The Virtual Machine only has 4GB RAM but synchronisation works without any issues.

@UserMaxL Thank you for your quick response. I've tried your approach. but when I close the app the synchronization starts at 0.
Do you close the app normally with "X"? and do you restart by double-clicking?

Hi @rene-sch , yes, I just closed with "X" and opened with double click. Sorry it didn't work for you, but strange it starts at zero.

@UserMaxL Thank you for your quick response. I've tried your approach. but when I close the app the synchronization starts at 0.
Do you close the app normally with "X"? and do you restart by double-clicking?

Hi @rene-sch , yes, I just closed with "X" and opened with double click. Sorry it didn't work for you, but strange it starts at zero.
Hi @UserMaxL do you always wait until the background processes of the app are closed before restarting it?

@UserMaxL Thank you for your quick response. I've tried your approach. but when I close the app the synchronization starts at 0.
Do you close the app normally with "X"? and do you restart by double-clicking?

Hi @rene-sch , yes, I just closed with "X" and opened with double click. Sorry it didn't work for you, but strange it starts at zero.
Hi @UserMaxL do you always wait until the background processes of the app are closed before restarting it?
Hi, yes, you should always wait until the background processes of the app are closed.

@UserMaxL Thank you for your quick response. I've tried your approach. but when I close the app the synchronization starts at 0.
Do you close the app normally with "X"? and do you restart by double-clicking?

Hi @rene-sch , yes, I just closed with "X" and opened with double click. Sorry it didn't work for you, but strange it starts at zero.
Hi @UserMaxL do you always wait until the background processes of the app are closed before restarting it?
Hi, yes, you should always wait until the background processes of the app are closed.

@UserMaxL okay now it seems to be working. Does the synchronization then start exactly at the % at which you closed the app?

@UserMaxL Thank you for your quick response. I've tried your approach. but when I close the app the synchronization starts at 0.
Do you close the app normally with "X"? and do you restart by double-clicking?

Hi @rene-sch , yes, I just closed with "X" and opened with double click. Sorry it didn't work for you, but strange it starts at zero.
Hi @UserMaxL do you always wait until the background processes of the app are closed before restarting it?
Hi, yes, you should always wait until the background processes of the app are closed.

@UserMaxL okay now it seems to be working. Does the synchronization then start exactly at the % at which you closed the app?

Yes, was the case for me...what version of the app are you using?

@UserMaxL Thank you for your quick response. I've tried your approach. but when I close the app the synchronization starts at 0.
Do you close the app normally with "X"? and do you restart by double-clicking?

Hi @rene-sch , yes, I just closed with "X" and opened with double click. Sorry it didn't work for you, but strange it starts at zero.
Hi @UserMaxL do you always wait until the background processes of the app are closed before restarting it?
Hi, yes, you should always wait until the background processes of the app are closed.

@UserMaxL okay now it seems to be working. Does the synchronization then start exactly at the % at which you closed the app?

Yes, was the case for me...what version of the app are you using?

I'm currently trying to install version 2.1.16 from 2.1.6

Can you see in C:\Users\Username\AppData\Roaming\DeFi\blocks what's the size of first blkXXXXX.dat?

PS: You can't use older version, not because it's not recommended, you just can't.

PS: You can't use older version, not because it's not recommended, you just can't.

@bvbfan the problem is that currently no version works. So I'm trying to install version 2.1.16.

They work it's related to -reindex can you check C:\Users\Username\AppData\Roaming\DeFi\blocks and write here which are the size of of top 3 items?

They work it's related to -reindex can you check C:\Users\Username\AppData\Roaming\DeFi\blocks and write here which are the size of of top 3 items?

@bvbfan Do you mean these?

Delete only 3 blk file (first 3 after folder index) restart the app.

Delete only 3 blk file (first 3 after folder index) restart the app.

normal with double click or should I download it again from github?

Latest version 2.17, just start it.

okay i'm trying. thank you

Latest version 2.17, just start it.

@bvbfan
I finally managed to synchronize version 2.1.17. Then I went to "restore wallet" but now no coins etc. are displayed. Do I have to restore the wallet with my seed?
Thanks for your endless help.

I got the 2.1.17 windows version but still same issue. my wallet is not loading node disconected.

Now everything is working again. I installed version 2.1.17 (like @bvbfan told me) and entered my seed. Everything back. Many thanks for your help

Hi @BenThrottle85 @kingsersch @TTSIP @thiencdam @goldi147 @jibesmaster @Ausafari @monnyfx , thank you for your patience on this issue. Can you try installing the latest v2.1.17. If syncing is stalled, try the Reindex Option via Settings, click on Reindex after saving followed by delete peers and blocks. Does your app sync?

Is there a fix for this yet? My node syncs but it still says Sync disconnected.

Hi @MrStryker , we manage to reproduce this internally and the above solution worked, would appreciate if you can try the above solution and let me know the outcome. If it doesn't, kindly send your logs please. Thank you.

Hi @MrStryker , we manage to reproduce this internally and the above solution worked, would appreciate if you can try the above solution and let me know the outcome. If it doesn't, kindly send your logs please. Thank you.

Does not work. I've tried it several times.  What log files do you need?

Hi @MrStryker , we manage to reproduce this internally and the above solution worked, would appreciate if you can try the above solution and let me know the outcome. If it doesn't, kindly send your logs please. Thank you.

My issue was fixed.  My clock was manually adjusted on my PC.  When I put it to auto adjust the node was able to connect.

Hi @MrStryker , glad that your issue is fixed. Interesting so it was due to your time-clock on PC that's why you could not sync? What was your settings prior?

Hi @BenThrottle85 @kingsersch @TTSIP @thiencdam @goldi147 @jibesmaster @Ausafari @monnyfx @MrStryker , thank you for your patience on this issue. Update v2.3.0 is out.
Do give a thumbs up to this comment if your issue has been resolved, otherwise, you may reopen this issue with new information (include screenshots) and app logs. Thank you.


Add custom jinja template for slot documentation pages

Guidelines
Soft Schema Freeze
The nmdc-schema and berkeley-schema-fy24 schemas are under a soft freeze, which means changes should not be made that have any downstream implications. To ensure this, all PRs created creating during the freeze will be closely reviewed with every component of the NMDC system in mind.
Reviewers
To ensure no changes are made unexpectedly, PR creators will request reviews from all Berkeley Schema Roll Out task coordinators.
We expect task coordinators to review PRs and provide feedback/approval within 1 week of when they are identified as reviewers.
PRs will NOT be merged until all task coordinators (or one of their delegates) have approved.
Expedition, questions, and discussion can happen at any meeting.
Delays in review & merging should be addressed in meetings or with NMDC leadership.
PR Information
What type of PR is this? (check all applicable)

[ ] Refactor
[ ] Feature
[ ] Bug Fix
[ ] Optimization
[x] Documentation
[ ] Schema change: Structure and content

created, updated, or deleted a class, slot, or enum
changed whether a slot is multivalued
changed the way a slot is assigned to a class
changed the permissible_values of an enum
etc.

[ ] Schema change: Cleanup and preparation

updated the description of a class, slot, or enum
updated the mappings of a class, slot, or enum to an ontology
added an enum for future use (it is not in the range of any slot)
etc.

Description
Very trivial update adding custom jinja template for slot documentation pages to nmdc-schema repository.
Related Issues

All PRs should relate to or fix an issue(s). Please identify the issue(s) below.

Related Issue(s): #
Fixes: #

Did you add/update any tests?

[ ] Yes
[x] No (Add a justification below)
[ ] I need help with writing tests

Could this schema change make it so any valid data becomes invalid?

This is a question about what the schema allows. It is not a question about what happens to exists in the NMDC database right now.
Example: If, in this PR branch, you renamed a slot from foo to foo_bar, the answer to this question would be "yes," even if nothing in the NMDC database currently uses the foo slot.
More examples: slot or class name changes, changes to a slot's multivalued state, changes to a slot's range (e.g. string to integer), changes to slot assignments to classes, changes to an enum's permissible_values

[ ] Yes (A migrator is required)
[x] No
[ ] I need help determining this

If you answered "Yes", does this PR branch include that migrator?

[ ] Yes
[ ] No, this PR is incomplete and I need help writing the migrator

Does this PR have any downstream implications?

Examples: any change here that requires a change to workflows, workflow automation, the Mongo-to-Postgres ingest process, Jupyter notebooks, the Runtime, etc.

[ ] Yes (Explain below)
[x] No

To see an example of the main change that this PR is introducing compare these two documentation pages:

https://microbiomedata.github.io/nmdc-schema/gold_path_field/
https://microbiomedata.github.io/nmdc-schema/pr-preview/pr-2140/gold_path_field/

This PR is ready to be approved and merged.


feat(client/properties-provider): add details group

This aligns the Properties Panel with the one of the Camunda Modeler.
Closes #116

@pinussilvestrus Looks good! However one last question before merging this would be if we should add couple of more tests about changing the values of properties and check if businessObject of the related shape contains the correct fields. Maybe we cover this stuff in another tests, so wanted to make sure.

That should be completely covered by the unit tests for the different parts and implementations (cf. PropertiesProvider test suite)

Good job 🎉


httpx instrumentation: IndexError: tuple index out of range

Describe the bug
In some cases httpx passes request param to its httpx.Client.send() as keyword argument rather than positional one, like e.g. when using httpx.Client.stream() in a context manager, which is unacceptable by elastic-apm.
Please note same happens when using httpx.AsyncClient.stream() instead of sync httpx.Client.stream().
Current workaround
Disable httpx instrumenting (set environment variable SKIP_INSTRUMENT_HTTPX="true") prior elasticapm.instrument() is called.
To Reproduce
import elasticapm
import httpx

def main():
    elasticapm.instrument()

    client = httpx.Client()
    elastic_apm_client = elasticapm.Client()

    elastic_apm_client.begin_transaction("httpx-context-manager-client-streaming")
    try:
        with client.stream("GET", url="https://example.com") as response:
            assert response
    finally:
        client.close()
        elastic_apm_client.end_transaction("httpx-context-manager-client-streaming")

if __name__ == "__main__":
    main()

Expected result
No exception is raised.
Actual result
Traceback (most recent call last):
  File "<...>/test.py", line 21, in <module>
    main()
  File "<...>/test.py", line 13, in main
    with client.stream("GET", url="https://example.com") as response:
  File "<...>/venv/lib/python3.8/site-packages/httpx/_client.py", line 1799, in __enter__
    self.response = self.client.send(
  File "<...>/venv/lib/python3.8/site-packages/elasticapm/instrumentation/packages/base.py", line 210, in call_if_sampling
    return self.call(module, method, wrapped, instance, args, kwargs)
  File "<...>/venv/lib/python3.8/site-packages/elasticapm/instrumentation/packages/httpx.py", line 42, in call
    request = kwargs.get("request", args[0])
IndexError: tuple index out of range

See similar results for httpx.AsyncClient.stream()
To Reproduce
import asyncio

import elasticapm
import httpx

async def main():
    elasticapm.instrument()

    client = httpx.AsyncClient()
    elastic_apm_client = elasticapm.Client()

    elastic_apm_client.begin_transaction("httpx-async-context-manager-client-streaming")
    try:
        async with client.stream("GET", url="https://example.com") as response:
            assert response
    finally:
        await client.aclose()
        elastic_apm_client.end_transaction(
            "httpx-async-context-manager-client-streaming",
        )

if __name__ == "__main__":
    asyncio.run(main())

Actual result
Traceback (most recent call last):
  File "<...>/test.py", line 23, in <module>
    asyncio.run(main())
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "<...>/test.py", line 15, in main
    async with client.stream("GET", url="https://example.com") as response:
  File "<...>/venv/lib/python3.8/site-packages/httpx/_client.py", line 1821, in __aenter__
    self.response = await self.client.send(
  File "<...>/venv/lib/python3.8/site-packages/elasticapm/instrumentation/packages/asyncio/httpx.py", line 42, in call
    request = kwargs.get("request", args[0])
IndexError: tuple index out of range

Environment

OS: Ubuntu 18.04.5 LTS
Python version: 3.8.9
Framework and version: httpx==0.17.1 (also reproducible with current latest httpx==0.19.0)
APM Server version: --
Agent version: elasticapm==6.4.0

Hey @Tayum! Great find, thanks for the detailed bug report! I opened #1337 with a fix.


Show encrypted msg in transaction details

Issue #811

A new build of 2f142b4f630aed152a7b9d2f522fa2217e548f68 has completed succesfully!
Binaries available at https://build.nav.community/binaries/encrypted-msg-in-tx-details


Publish on tastejs/todomvc ?

Is there a reason why this TodoMVC example is not published on the main repo that hosts all examples ( https://github.com/tastejs/todomvc )?
It would be a good way for cyclejs to advertise itself among the other options people have when trying to choose a technology.

@vkammerer the main reason is we haven't really been proud of this particular codebase, enough to put it on that repository. Like we are experimenting still with different kinds of state management and list management tools, and we are slowly converging to a solution. We're actually close enough to getting there. But I think also at this point including the example there won't dramatically help with adoption, there are also things nowadays like https://github.com/gothinkster/realworld and hackernews clones as modern TodoMVC alternatives. That said, yes, when we're proud enough of our TodoMVC code, we'll put it there. :)


Wrong merged_DT from  merge_finemapping_results()

Hi,
I have couple of GWAS summary statisitics. I have completed running finemap_loci(). It has generated locus specific results. That is fine. Now I wanted to run codes given in Summarise vignette  (https://rajlabmssm.github.io/echolocatoR/articles/summarise_vignette.html). I set the "dataset " argument of merge_finemapping_results() to the parent folder where all the locus specific sub-folder are present.
I ran the following code:
merged_DT <- merge_finemapping_results(dataset=dataset_dir, minimum_support=1, include_leadSNPs=T)
I notice the following things:
(1) it gives me the following error: Error in merge_finemapping_results(dataset = dataset_dir, minimum_support = 1,     include_leadSNPs = T): + No multi-finemap files found., though printing
(2) when I print merged_DT , the "Dataset" column is Nalls23andMe_2019!!!. Though I have not that result stored in my machine.
I tried with "dataset" argument set to different dataset_dir, but merged_DT  is EXACTLY the same.
Could you please help? Thank You.
dataset set to one finemap_loci() dir:

dataset set to ANOTHER finemap_loci() dir:

:

Here is the output of utils.session()
R version 4.1.2 (2021-11-01)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS/LAPACK: /home/sislam3/anaconda3/envs/echoR/lib/libopenblasp-r0.3.18.so
locale:
[1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8       LC_NAME=C
[9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base
other attached packages:
[1] echolocatoR_0.1.2
loaded via a namespace (and not attached):
[1] colorspace_2.0-2            rjson_0.2.21
[3] ellipsis_0.3.2              class_7.3-20
[5] biovizBase_1.42.0           htmlTable_2.4.0
[7] XVector_0.34.0              GenomicRanges_1.46.1
[9] base64enc_0.1-3             gld_2.6.4
[11] dichromat_2.0-0             rstudioapi_0.13
[13] proxy_0.4-26                bit64_4.0.5
[15] AnnotationDbi_1.56.2        fansi_1.0.2
[17] mvtnorm_1.1-3               xml2_1.3.3
[19] splines_4.1.2               ggbio_1.42.0
[21] cachem_1.0.6                rootSolve_<IP_ADDRESS>
[23] knitr_1.37                  Formula_1.2-4
[25] Rsamtools_2.10.0            dbplyr_2.1.1
[27] cluster_2.1.2               png_0.1-7
[29] graph_1.72.0                BiocManager_1.30.16
[31] compiler_4.1.2              httr_1.4.2
[33] backports_1.4.1             assertthat_0.2.1
[35] Matrix_1.4-0                fastmap_1.1.0
[37] lazyeval_0.2.2              prettyunits_1.1.1
[39] htmltools_0.5.2             tools_4.1.2
[41] gtable_0.3.0                glue_1.6.0
[43] lmom_2.8                    GenomeInfoDbData_1.2.7
[45] reshape2_1.4.4              dplyr_1.0.7
[47] rappdirs_0.3.3              Rcpp_1.0.8
[49] Biobase_2.54.0              vctrs_0.3.8
[51] Biostrings_2.62.0           rtracklayer_1.54.0
[53] xfun_0.29                   stringr_1.4.0
[55] lifecycle_1.0.1             ensembldb_2.18.3
[57] restfulr_0.0.13             XML_3.99-0.8
[59] zlibbioc_1.40.0             MASS_7.3-55
[61] scales_1.1.1                BSgenome_1.62.0
[63] VariantAnnotation_1.40.0    ProtGenerics_1.26.0
[65] hms_1.1.1                   MatrixGenerics_1.6.0
[67] RBGL_1.70.0                 parallel_4.1.2
[69] SummarizedExperiment_1.24.0 expm_0.999-6
[71] AnnotationFilter_1.18.0     RColorBrewer_1.1-2
[73] curl_4.3.2                  yaml_2.2.1
[75] Exact_3.1                   memoise_2.0.1
[77] gridExtra_2.3               ggplot2_3.3.5
[79] biomaRt_2.50.2              rpart_4.1-15
[81] reshape_0.8.8               latticeExtra_0.6-29
[83] stringi_1.7.6               RSQLite_2.2.9
[85] S4Vectors_0.32.3            BiocIO_1.4.0
[87] e1071_1.7-9                 checkmate_2.0.0
[89] filelock_1.0.2              GenomicFeatures_1.46.3
[91] BiocGenerics_0.40.0         boot_1.3-28
[93] BiocParallel_1.28.3         GenomeInfoDb_1.30.0
[95] rlang_0.4.12                pkgconfig_2.0.3
[97] matrixStats_0.61.0          bitops_1.0-7
[99] lattice_0.20-45             purrr_0.3.4
[101] GenomicAlignments_1.30.0    htmlwidgets_1.5.4
[103] bit_4.0.4                   tidyselect_1.1.1
[105] GGally_2.1.2                plyr_1.8.6
[107] magrittr_2.0.1              R6_2.5.1
[109] IRanges_2.28.0              DescTools_0.99.44
[111] generics_0.1.1              Hmisc_4.6-0
[113] DelayedArray_0.20.0         DBI_1.1.2
[115] pillar_1.6.4                foreign_0.8-82
[117] survival_3.2-13             KEGGREST_1.34.0
[119] RCurl_1.98-1.5              nnet_7.3-17
[121] tibble_3.1.6                crayon_1.4.2
[123] utf8_1.2.2                  OrganismDbi_1.36.0
[125] BiocFileCache_2.2.0         jpeg_0.1-9
[127] progress_1.2.2              grid_4.1.2
[129] data.table_1.14.2           blob_1.2.2
[131] digest_0.6.29               stats4_4.1.2
[133] munsell_0.5.0

Hi @saifulislampharma thank you for bringing this to my attention. I think couple things are going on here.
1. Unable to find files
I just went through and realized this was a typo in which I accidentally only searched for files with .multi_finemap.csv (an convention from older versions of echolocatoR). I've updated it so that it now searches now the old and the new .Multi-finemap.tsv conventions. Apologies for not catching this earlier.
2. Conflicting merged_DT variables
I think when the merged_DT fails to be constructed from the local files, an error is not thrown because of some internal tryCatch functions. Then, the dataset of the same variable name (merged_DT) is called instead.
I've resolved this by renaming the merged_DT internal variable so echolocatoR can't confuse it with the PD dataset merged_DT.
Just reran with the Getting started vignette example and confirmed this gives different results than merged_DT


BCCM health dashboard

[x] Export playback errors from npaw to metabase
[x] Create Dashboard(s)
[x] Make sure they are finadable and available to the team

A dashboard that gives insight on general issues
e.g. reference to bmm dashboard

Addition from Metabase here: https://metabase.bcc.media/dashboard/57-bccm-platfrom-errors-db

Central page: https://www.notion.so/bccmedia/Stream-Troubleshooting-14a563c0cccf800e9e57ed14d1153247
Linked from: https://www.notion.so/bccmedia/a71370072a2e4ea98af9df4eb6eac098?v=9c8c864a49b44619a530fd6556605c2f

@kkuepper @sifferhans can you guys confirm that you have access to the links in: https://www.notion.so/bccmedia/Stream-Troubleshooting-14a563c0cccf800e9e57ed14d1153247
NPAW, Metabase, AWS

I don't have AWS login. Matjaz will help me set it up


C++ Compiler Test (Don't Pull)

This branch was used to test changes to mamba-workflow and changes in this branch will be integrated in #133


maim: add missing libSM dependency

Description of changes

Added a missing dependency called libSM
Added myself as a maintainer of the package
Formatted the file modified files nixpkgs-fmt

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 182506 run on x86_64-linux 1

3 packages built:

maim
pb_cli
yaxg

slop was also recently broken by libSM dependency. Since maim depends on slop, I think it would make sense to also fix it for slop in this PR. In fact, I suspect fixing the dependency for slop will also fix it for maim.

Result of nixpkgs-review pr 182506 run on x86_64-linux 1

7 packages built:

capture
giph
maim
pb_cli
recapp
slop
yaxg


Develop validation toolkit for lens and SNe generation

Validate that the input catalogs and rates for the SL and SNe are self consistent (what is input from the model is output from the catalogs). Develop the tools for these comparisons

@jbkalmbach @rbiswas4 I am wondering if this should become an epic, like the Basic Validation one and the Light Curve Analysis ones, just because it's about validating the contents of the images without looking at the light curves. Comments welcome!


runtime(java): Improve the recognition of the "style" method declarations

Request the new regexp engine (v7.3.970) for [:upper:] and
[:lower:].

Recognise declarations of in-line annotated methods.

Recognise declarations of strictfp methods.

Establish partial order for method modifiers as shown in
the MethodModifier production; namely, public and
friends should be written the leftmost, possibly followed
by abstract or default, or possibly followed by other
modifiers.

Stop looking for parameterisable primitive types (void<?>,
int<Object>, etc., are malformed).

Stop looking for arrays of void.

Acknowledge the prevailing convention for method names to
begin with a small letter and for class/interface names to
begin with a capital letter; and, therefore, desist from
claiming declarations of enum constants and constructors
with javaFuncDef.
Rationale:

Constructor is distinct from method:

its (overloaded) name is not arbitrary;
its return type is implicit;
its throws clause depends on indirect vagaries of
instance (variable) initialisers;
its invocation makes other constructors of its type
hierarchy invoked one by one, concluding with the
primordial constructor;
its explicit invocation, via this or super, can
only appear as the first statement in a constructor
(not anymore, see JEP 447); else, its super call
cannot appear in constructors of record or enum;
and neither invocation is allowed for the primordial
constructor;
it is not a member of its class, like initialisers,
and is never inherited;
it is never abstract or native.

Constructor declarations tend to be few in number and
merit visual recognition from method declarations.
Enum constants define a fixed set of type instances
and more resemble class variable initialisers.

Note that the code duplicated for @javaFuncParams is written
keeping in mind for g:java_highlight_functions a pending 3rd
variant, which would require none of the :syn-cluster added
groups.

// VIM_TEST_SETUP set encoding=utf-8 termencoding=utf-8

as they are only meaningful for visual inspection of created
screen dump files?

Yeah I believe so. But when trying to inspect debug the test, a nice visual representation is helpful instead of using ??
But isn't encoding=utf-8 not already default encoding? At least  that's what I thought for Linux.

Thanks, I didn't notice that vim still uses latin1 encoding in this case. I guess this is fine.

libvterm: Unhandled CSI t

I have seen this a few times, especially with debug builds, but those appear to be harmless.
Thanks!

I have also looked at the output of xxd for an input/ file
and a *_00* screen dump.  The relevant multibyte sequences
match.
We can proceed.

okay thanks!

Note that the code duplicated for @.***is written   keeping in mind forg:java_highlight_functionsa pending 3rd   variant, which would require none of the:syn-cluster` added
groups.

some weird on g:java_highlight_functions, not exactly what's that for..
for example following:
    public static Date StrToDate(String date) {  // <== why this would not highlight whatever its value (or unlet)?
        try {
            return SIMPLE_DATE_FORMAT.parse(date);
        } catch (ParseException e) {
            e.printStackTrace();
        }
        return null;
    }

    public static String DateToString(Date date) {  // <== why this would not highlight whatever its value (or unlet)?
        return SIMPLE_DATE_FORMAT.format(date);  // <-- would highlight this if it set to 'indent' and why hl this?
    }

    public static java.sql.Date utilDateToSqlDate(Date date) {  // <~~ would highlight this if it set to 'style'
        return new java.sql.Date(date.getTime());  // <-- would highlight this if it set to 'indent' and why hl this?
    }

--
shane.xb.qian

Why, :h ft-java-syntax says what g:java_highlight_functions
is for.
With style assigned to g:java_highlight_functions, all
supplied examples are correctly highlighted: StrToDate and
DateToString do not follow naming conventions for method
names.  Please read :h ft-java-syntax and code conventions
from Sun.  Hint: try strToDate and dateToString.  This PR
hasn't changed that.
With indent assigned to g:java_highlight_functions, all
supplied examples are outrightly irrelevant: your code uses
4-space indents (not supported), when that value is doubled,
any code construct that looks like a method declaration is
highlighted as such; here, return ... ().  Try :setl sw=2
or :setl sw=8 and reformat all lines with =.  Again, this
PR hasn't changed anything related to this behaviour.
(Maybe "consistently indented by either" instead of "always
indented by either" would be a better wording.  Anyway, file
an issue if that change in the documentation is needed.)
In general, I would suggest to instead consider using style
(or signature that is planned for the next PR).  The simple
syntax pattern of indent works when you can afford flouting
conventions or when there are no nested types with their own
method declarations.

Actually I was thinking why it not just highlight the methods in class which is a similar cfg like the one in c/cpp.
// I did not say your PR changed something to it, but just felt that cfg originally was weird, perhaps make one or default to make the method be highlighted was good.
--
shane.xb.qian

Eh, similar to this:
https://github.com/vim/vim/blob/3f821d6de2586d921fb23e2facb4764ef9eb3294/runtime/syntax/c.vim#L447-L449
and that:
https://github.com/vim/vim/blob/3f821d6de2586d921fb23e2facb4764ef9eb3294/runtime/syntax/cpp.vim#L16-L17
?
Since I'm not the original syntax file author, I can only
speculate.  A 1998 version of java.vim has only indent
disguised as style.  When in 1997 the support for inner
classes in Java had been introduced, style was added in
1999.
There is no method related keyword in Java.
There are no class or interface :syn-regions defined for
javaFuncDef to be included in.  If we had them, I suspect,
:syn-sync would demand a large value from g:java_minlines
for humongous classes, e.g. java.lang.Character.
As to why the method regexp patterns are complicated, it is
because Java is very thrifty with its keywords and permits
complex types that can contain methods that can contain
types that can contain methods that can contain types and so
on.  So, there are package-private methods (that is, neither
public, nor protected, nor private); there are abstract or
final or strictfp methods and classes.  In addition,
else if ( or new [Ff]oo( are not helping either.

yes, i meant c_functions,
and as above example, regardless the keywords, but just hl following:
StrToDate
DateToString
utilDateToSqlDate

or if it is a var: public static final SimpleDateFormat SIMPLE_DATE_FORMAT = new xxxxxx; then hl SIMPLE_DATE_FORMAT as well // tho maybe different hl color.
i am not sure if that make sense, or make sense to you, perhaps useful, or let it be~ thanks anyway.
// ps. yes, i hear java changed a lot, i knew a bit java approx in 2003~2008, looked it changed a lot later, :-)

You do realise that g:c_functions highlights both function
call sites and their declarations from about v9.1.0047,
whereas g:java_highlight_functions only highlights method
declarations from about v5.0.  Could you let me in on this
cunning plan of yours? :)
With methods that bear camelCase and PascalCase names
indifferently, I wonder how did you name your own Java types
back in ‘2003-8’?
Steer clear from "indent", its simultaneous support of 2-
or 8-space or tab indented methods is a mistake: with just
4x 2-space indents, there will be misidentified constructs
highlighted as method declarations.  (Before v5.2, it was
strictly 2 spaces or a tab.)
Follow naming conventions and use whatever indentation you
like with "style".  Else, you are free to exercise the third
option yourself:
https://github.com/vim/vim/blob/38b9f45253f582ab63174376e321092f8a9a7808/runtime/doc/syntax.txt#L2023-L2026

You do realise that g:c_functions highlights both function
call sites and their declarations from about v9.1.0047,
whereas g:java_highlight_functions only highlights method
declarations from about v5.0.  Could you let me in on this
cunning plan of yours? :)

donot understand what're you trying to say,
c_functions i recalled it was adding (by Luca) near after https://github.com/vim/vim/issues/13641 which we corrected something for c/cpp syntax at that time around.

With methods that bear camelCase and PascalCase names
indifferently, I wonder how did you name your own Java types
back in ‘2003-8’?

if you were arguing the naming or something of that example, ok, yes, i copied some piece somewhere just for AS IT IS an example, it definitely was not me wrote it at 2003~2008.
so or whatever, it's on you, you can think about this idea, or let it be, but for now to me i think this g:java_highlight_functions looks a bit difficult or confused to use, :-)
--
shane.xb.qian


Syntax highlighting in Create Types / Change Type dialogs

What is the feature you'd like to have?
When editing very large type definitions (e.g. COM vtables), having no syntax highlighting in the text box makes the process a bit harder than it needs to be. It would be nice to have some basic syntax highlighting here. I don't think it has to be a full on language server situation - even just highlighting keywords and well-known types would go a long way.
Are any alternative solutions acceptable?
Editing the type in VS Code and copying it into Binary Ninja, I suppose.

So this is probably going to block on a more general "support syntax highlighting in a native UI element" type of feature. Since we'd likely use it in other places too such as source display, the scripting console, snippets, etc. It would be nice though!

So this is probably going to block on a more general "support syntax highlighting in a native UI element" type of feature.

Is this a separate feature from the syntax highlighting seen in e.g. the snippet plugin's editor and Sidekick's index editor?

Is this a separate feature from the syntax highlighting seen in e.g. the snippet plugin's editor and Sidekick's index editor?

Those plugins are using pygments for highlighting which is a python-specific package. Since the Binja UI is written using Qt/C++ there isn't an easy way to import those. An implementation of this would need to investigate either plumbing that package through the api (yuck) or using some other parser to highlight the text.


Support for Colab

I'm struggling to run the notebooks in both binder and local. It would be really nice if there were Colab examples of those notebooks as well...

What is the issue you have running the notebooks in MyBinder or locally? I hope we can help you get them working! Colab support for all notebooks is not immediately planned. We do hope to have another cloud option soon where you can run all the notebooks.
If you're experiencing issues with a local install, it would be helpful if you can paste the output of python check_install.py, run from the root of the repository directory, with openvino_env activated.

I tried to use 202-vision-superresolution-image but as it didn't have access to GPU it got stuck. In local, I also don't have access to GPU. As Colab has access to free GPU it would've been lot easier to run those notebooks.

OpenVINO is not optimized for NVIDIA GPUs, so having the GPU in Colab does not make inference faster. The superresolution notebook is a bit slow - but it should still work, even on Binder. Inference on the 202 notebook took less than a minute for me there (see screenshot). Binder is a free resource though - so at times resources may be more limited than usual.
Please try it locally if you want! You do not need a GPU to use OpenVINO.

Looking forward to GPU support someday. Running a Deep Learning model on CPU might be consuming after all .....

OpenVINO is optimized to run on CPU, and if you have a recent Intel Core CPU with Intel Integrated Graphics Card, it can use that as well. It can also use both at the same time, for even better performance. See the 104-model-tools notebook for some quick performance checks with Open Model Zoo models. You'll see that you get quite good performance with many models on CPU (using benchmark_app to check the performance). In the 102 notebook you can also see how running the converted model on CPU or iGPU is much faster than the original PyTorch model. In the PaddleGAN notebook the difference is very large - the PaddlePaddle model is too heavy for MyBinder, but the converted OpenVINO IR model runs well there.


[Kosmos-2] Will depth information be incorporated in the future?

Hi,
I wonder has anyone successfully incorporated Kosmos-2 with depth information? Will this be the future goal to make the model gain more spatial awareness?
Model I am using Kosmos-2

Hi, @quantingxie, Thanks for the attention!
It sounds like a promising goal, but unfortunately we have not achieved it yet. If you need help, we're happy to provide it.

Hi Zhiliang,
It would be great if you guys could provide a Kosmos-2 model that can also output depth information in addition to bounding boxes! Looking forward to it! If you have any progress, please let me know, thanks!


chore: bdd-spring-1578938033 to 0.0.1

chore: Promote bdd-spring-1578938033 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Bumping OLM from v0.19.1 to v0.20.0 after hitting a bug making OLM st…

…op working
Signed-off-by: Nico Schieder<EMAIL_ADDRESS>

/lgtm


Increasing per-process limit of core file size to infinity.

Hello,
I finished the installation successfully by running the Setup.sh file. Now I am trying to run the rebuild file, following this command "UE4_ROOT=~/UnrealEngine_4.18 ./Rebuild.sh"
I receive the following error:
Performing 11 actions (8 in parallel)
[2/11] Compile Module.Carla.cpp
[3/11] Compile Module.Carla.gen.cpp
[1/11] Compile PCH.CarlaUE4.h
[4/11] Compile CarlaUE4GameModeBase.gen.cpp
[5/11] Compile CarlaMapGenerator.cpp
[6/11] Compile CarlaUE4.cpp
[7/11] Compile CarlaUE4GameModeBase.cpp
[8/11] Compile CarlaMapGenerator.gen.cpp
[9/11] Compile CarlaUE4.init.gen.cpp
[10/11] Link libUE4Editor-Carla.so
[11/11] Link libUE4Editor-CarlaUE4.so
Total build time: 72.53 seconds (Local executor: 20.96 seconds)
Launching UE4Editor...
Increasing per-process limit of core file size to infinity.
setrlimit() failed with error 22 (Invalid argument)

Max per-process value allowed is 0 (we wanted infinity).
Could not adjust core file size, consider changing "core" in /etc/security/limits.conf and relogin.
error(22): Invalid argument
Alternatively, pass -nocore if you are unable or unwilling to do that.
Could not set desired per-process limits, consider changing system limits.
Exiting abnormally (error code: 1)

what do I have to do now  ? @nsubiron

I don't know why this would happen, but it seems that passing the -nocore argument when launching the editor should work around it. Maybe you can search in the Unreal Engine forums for more info.

Hello
thanks for your response.
During the past days, I searched a lot online, and I didn't find definitive answers, but the -nocore option was one of the solutions I found.
I tried doing this by adding this flag in the Rebuild.sh file in the command which launches UE4editor, however, this is the error I got afterwards:

It already started to load the editor, and it reached over 95% then it stopped and I got this above error

and when i try to run it by:
~/UnrealEngine_4.18/Engine/Binaries/Linux$ ./UE4Editor -nocore
I receive the following error

oh it seems there is something wrong with your UE4 installation, maybe you can try reinstalling it. You can find here more details and troubleshooting for compiling UE4 on Linux.

I reinstalled UE4, but I still get the same problem
It is so weird, and I still can't know how to solve it. I am just trying to figure out


Error with record.py

I can run run attack.py and generate adversarial examples without issue but afterwards, when I run record.py, I get the following errors. I am wondering what is record.py actually doing if the adversarial examples have been created and stored in the results directory already?
/home/asd/robust/local/lib/python2.7/site-packages/librosa/core.py:25: UserWarning: Could not import scikits.samplerate. Falling back to scipy.signal
'Falling back to scipy.signal')
ALSA lib pcm_dsnoop.c:606:(snd_pcm_dsnoop_open) unable to open slave
ALSA lib pcm_dmix.c:1029:(snd_pcm_dmix_open) unable to open slave
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib pcm_dmix.c:1029:(snd_pcm_dmix_open) unable to open slave
TypeError: argument 1 must be string or read-only buffer, not list
Traceback (most recent call last):
File "record.py", line 68, in
main(sys.argv[1], sys.argv[2])
File "record.py", line 38, in main
outdata = np.array(librosa.load(out_path, fs)[0], dtype=np.float32)
File "/home/asd/robust/local/lib/python2.7/site-packages/librosa/core.py", line 76, in load
with audioread.audio_open(os.path.realpath(path)) as input_file:
File "/home/asd/robust/local/lib/python2.7/site-packages/audioread/init.py", line 111, in audio_open
return BackendClass(path)
File "/home/asd/robust/local/lib/python2.7/site-packages/audioread/rawread.py", line 62, in init
self._fh = open(filename, 'rb')
IOError: [Errno 2] No such file or directory: '/home/asd/robust/robust_audio_ae1/ae.wav'
Exception TypeError in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored

The record.py is intended for playing generated adversarial examples to evaluate their attack success rate. You don't need to run this script if you just want to generate adversarial examples.


Add additional CoreServices dependency when building on macOS

Needed to build on macOS

I assume this is now needed because of the recently added watch option? Perhaps worth putting behind a feature flag then to make this dependency optional 🤔? It shouldn't be a problem for wasm & the like since it only lives in aiken (and not the lib crates), but still 🤷


MLA prefill w/o weight absorption

Motivation
For large batch sizes, not using weight absorption in the MLA prefill phase will be more efficient.
python3 -m sglang.bench_one_batch --batch-size 128 --input 512 --output 1 --model neuralmagic/DeepSeek-Coder-V2-Instruct-FP8 --trust-remote-code --tp 8 --disable-cuda-graph

# prefill w/o absorb (this PR)
Prefill. latency: 2.17135 s, throughput:  30182.18 token/s

# prefill w/ absorb (main)
Prefill. latency: 3.29770 s, throughput:  19873.24 token/s

prefill w/o absorb (this PR)

prefill w/ absorb (main)

ref: https://github.com/sgl-project/sglang/pull/2203
Checklist

[ ] Format your code according to the Contributor Guide.
[ ] Add unit tests as outlined in the Contributor Guide.
[ ] Update documentation as needed, including docstrings or example tutorials.

QQ Why disable-cuda-graph and not enable-dp-attention?  And what is the throughput and latency like in small batch size?

Why disable-cuda-graph and not enable-dp-attention? And what is the throughput and latency like in small batch size?

We only test prefill here, while cuda graph is only used for decoding.
The prefill performance difference becomes significantly noticeable only with large batch sizes. However, in our serving  benchmarks, we employed chunked prefill, with default chunk sizes of 8192 for TP and 4096 for Data Parallel (DP) attention( due to the workload issues with FusedMoE). For the sake of this test, I evaluated only one batch, which I believe is sufficient to illustrate the performance differential. Once the pull request at https://github.com/sgl-project/sglang/pull/2203 is merged, I will present results using larger chunk sizes on DP + EP configurations.

Why disable-cuda-graph and not enable-dp-attention? And what is the throughput and latency like in small batch size?

We only test prefill here, while cuda graph is only used for decoding.
The prefill performance difference becomes significantly noticeable only with large batch sizes. However, in our serving  benchmarks, we employed chunked prefill, with default chunk sizes of 8192 for TP and 4096 for DP attention (due to the workload issues with FusedMoE), so we cannot test large prefill batch. For this test, I evaluated only one batch, which I believe is sufficient to illustrate the performance difference. Once MoE Expert Parallel Impl #2203 merged, I will present results using larger chunk sizes on DP + EP configurations.

What I mean here is not only performance, but also compatibility, all features can be compatible.


Click v8 support Resolve #280

Does not attempt autocompletion yet: #278

https://github.com/graham33/typer/commit/856a954c18dc02aea5795000257e96780a06cdaf adds some fixes for the autocompletion stuff, tests still failing though.

Thanks for your interest! :cake:
This was handled in https://github.com/tiangolo/typer/pull/317, making Typer compatible with both Click 7 and Click 8, and is now released in Typer version 0.4.0. 🎉
Given that, I'll close this one now. :coffee:

Thank you so much, @tiangolo ! typer is great!


Bottle caps

Bottle caps are an important part of the game and should be near the top of the todo list. They affect event Pokemon, transforming, moves that copy other moves (Copycat -> Hidden Power), and possibly other stuff that don't come to mind.
Client: I think the best thing is for the teambuilder to have a column next to IVs to set bottle caps. There just needs to be a check box next to each IV because you can only maximize them.

What specifically is wrong with the current implementation? Can you be much more specific?
Also: I was told that you could set IVs to anything. You can only maximize them?

Bottle caps don't change IVs. They just trigger a flag which makes stat calculations use 31. Your IVs remain intact and bottle caps only maximize.

Are your original IVs relevant to anything in-battle other than Hidden Power?

From https://www.smogon.com/forums/threads/hidden-type.3591194/#post-7168256 -

Is this really relevant to non-mods, though? Just set any 30 IV to 31 for your OM. I don't really think something that will never come up outside of mods (i.e. this isn't even relevant to AG or Hackmons) should be worth modding the teambuilder for.

I think I'd rather have a Hidden Power type selector than a checkbox next to every IV, anyway.

Sounds good to me. However, because bottle caps only max the stat, we need to update the validator to check hidden power legalities if the IV is below 31

We already do...?

They are only relevant to Hidden Power as far as I know. The Hidden Power type selector sounds fine.

@Zarel no, we don't. I can absolutely set a Pokemon to have 30 in all IVs (IVs should make it HP Fighting) and still have any other Hidden Power when I waltz into OU. This link is using HP Fire but with 30-all. http://replay.pokemonshowdown.com/gen7pokebankou-510856243

Also event Pokémon with fixed IVs don't work correctly because of your current implementation. Once you have this HP type selector implemented, I will update the validator to fix events.

What's wrong with my current implementation re: event pokemon?

You allow any IVs with any Hidden Power... Fixed IVs means fixed Hidden Power.

I thought this was Fixed though. I have fought with Pokemon with certain Iv's and it all turns out to be Dark HP even though in team builder it said HP Electric for example.

Yeah ive seen that working, but for bottle caps this could be a bit tricky. A few ideas:

Like TI said have a way to mark a IV as maxed by bottle cap. When validating HP type, legends needing 3 perfect IVs use the selected IVs, otherwise use 31 (possibly just switch the IV to 31 after validating)

Refactor the HP validator to check to see if the HP is legal with the IVs marked 31 allowed to be any number (0-31). If the HP type selected is legal, use it, if not use the current setup to select a valid HP type (asusme no bottle caps)

Okay, so, for the packed team:
I kind of want to implement it the dumb way, by adding HP type and ball after the EVs.
So the EVs which are currently, say,
|4,252,,,,252|
would become
|4,252,,,,252,flying,quickball|
The main advantage of doing it this way is to be backwards-compatible with old official implementations of fastUnpackTeam. The main disadvantage is that it's weird to look at, but honestly, packed teams will always look weird.

We could also have them as comma-separated after happiness, or even |-separated after happiness, which would still be backwards compatible but would require more code for backwards compatibility.

Comma-separated after happiness would be relatively simple and readable code, backwards compatible, and not being complete "wtf" material the way stuffing things after EVs would be, while trading off a small amount of performance that's probably unnoticeable. I think I'll do it that way.

402bb7bd84b7d
I ended up doing comma-separated after happiness. Feel free to code-review.

Mostly done in 2d6bd87c0bcca657

I think this issue is fixed with the oras validator + allowing 31 in any stat right? Or am I missing something here?

This issue is fixed for all official formats. It's not fixed for OMs, where someone might want to specify a Hidden Power type separately from IVs.


Move the delay timer to the highest possible number

The delay timer and PWM devices are both currently specified as device 0
meaning that any use of delay will interrupt the PWM the output. By
moving the delay timer to the upper range that will increase the number
of PWM devices that can be used before conflict occurs.
In future we may want to consider a dedicated delay timer in hardware.

Codecov Report

Merging #1202 (99f9e26) into image_v2.6.2 (cce36fd) will not change coverage.
The diff coverage is n/a.

@@              Coverage Diff              @@
##           image_v2.6.2    #1202   +/-   ##
=============================================
  Coverage         47.66%   47.66%
=============================================
  Files                25       25
  Lines              4618     4618
  Branches            767      767
=============================================
  Hits               2201     2201
  Misses             2364     2364
  Partials             53       53

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cce36fd...99f9e26. Read the comment docs.


Bump Sentry Javascript to fix incompatibility with Sentry Tracing

Version bump in order to be compatible with the latest version.
Sample dependency update.

@lucas-zimerman v7.9.0 is now out https://github.com/getsentry/sentry-javascript/releases


Flatpaks: hybrid modularity, container.yaml, package set restriction

This PR has three main groups of changes:

Support hybrid modularity by including the repository for the 'koji' plugin in the package set that we install in the Flatpak Dockerfile. (Also requires a osbs-client change)
Merge flatpak.json into container.yaml, and actually implement some tags from flatpak.json that were in the examples, but never hooked up.
Restrict the set of packages that are available to installation upfront using the dnf includepkgs option instead of checking after the fact that nothing extra was installed.

It would be hard to do three independent pull requests, but if it is preferred I can do serial pull requests for the three topics filing each one after the previous is merged. (The patches are cleanly split up, it's only the PR that is for everything together.) I've only done end-to-end testing with real content on all three sets of changes together.

Merge conflicts.

Rebased, comments addressed.

Looks like some of the coverage fell?
https://coveralls.io/builds/15883069/source?filename=atomic_reactor%2Fplugins%2Fprepub_flatpak_create_oci.py#L42

Looks like some of the coverage fell?
https://coveralls.io/builds/15883069/source?filename=atomic_reactor%2Fplugins%2Fprepub_flatpak_create_oci.py#L42

Am I reading it correctly that coverage fell for the CentOS 7 build, but not other builds? Certainly those lines are tested on Fedora 27 both under python2 and python3.

Am I reading it correctly that coverage fell for the CentOS 7 build, but not other builds? Certainly those lines are tested on Fedora 27 both under python2 and python3.

I think you're right.

@owtaylor, ok to merge?

Dug into the coverage situation:

What @lcarva noticed was actually just me deleting some covered lines - prepub_flatpak_create_oci went from 243/255 - 95.33% covered to 213/225 - 94.66% covered - I removed 30 covered lines - so the coverage for this file decreased a bit even though I didn't add any new uncovered lines.
But it was confusing to understand, because since #861 the Flatpak code isn't tested on CentOS (The code could be tested on CentOS - it isn't tested on CentOS so that the code paths for missing modulemd are tested...) and Coveralls counts those lines as covered in some cases and not others!  If you look at https://coveralls.io/builds/15883069 then you see those lines as covered for the percentages and in the "Source Files on build 5324" section, but listed as uncovered for "new lines missed in diff"...

So I think this is fine to merge.

Updated release notes draft:

Flatpaks: hybrid modularity, container.yaml, package set restriction


[gen_l10n] .packages generate wrong host

Steps to Reproduce

add generate: true in pubspec.yaml
flutter pub get

Expected results:
# Generated by pub on 2020-10-20 14:21:04.002596.
animations:file:///Users/liaojianxun/.pub-cache/hosted/pub.dartlang.org/animations-1.1.2/lib/
html_text:file:///Users/liaojianxun/.pub-cache/hosted/http%2558%2547%2547<IP_ADDRESS>%25588083/html_text-0.0.3/lib/

Actual results:
# generated by package:package_config at 2020-10-20 14:19:29.675468
animations:../../.pub-cache/hosted/pub.dartlang.org/animations-1.1.2/lib/
html_text:../../.pub-cache/hosted/httpXGG<IP_ADDRESS>X8083/html_text-0.0.3/lib/

my pubspec:
name: flutter_app
description: A new Flutter project.

# The following line prevents the package from being accidentally published to
# pub.dev using `pub publish`. This is preferred for private packages.
publish_to: "none" # Remove this line if you wish to publish to pub.dev

# The following defines the version and build number for your application.
# A version number is three numbers separated by dots, like 1.2.43
# followed by an optional build number separated by a +.
# Both the version and the builder number may be overridden in flutter
# build by specifying --build-name and --build-number, respectively.
# In Android, build-name is used as versionName while build-number used as versionCode.
# Read more about Android versioning at https://developer.android.com/studio/publish/versioning
# In iOS, build-name is used as CFBundleShortVersionString while build-number used as CFBundleVersion.
# Read more about iOS versioning at
# https://developer.apple.com/library/archive/documentation/General/Reference/InfoPlistKeyReference/Articles/CoreFoundationKeys.html
version: 1.0.0+1

environment:
  sdk: ">=2.8.0 <3.0.0"

dependencies:
  flutter:
    sdk: flutter
  # The following adds the Cupertino Icons font to your application.
  # Use with the CupertinoIcons class for iOS style icons.
  cupertino_icons: ^1.0.0
  flutter_localizations:
    sdk: flutter
  flutter_cupertino_localizations: ^1.0.1
  intl: ^0.16.1
  html_text:
    hosted:
      name: html_text
      url: http://<IP_ADDRESS>:8083
    version: ^0.0.3
  animations: ^1.1.2

dev_dependencies:
  flutter_test:
    sdk: flutter
  build_runner: ^1.10.0
  mockito: ^4.1.1
  hive_generator: ^0.7.1

# For information on the generic Dart part of this file, see the
# following page: https://dart.dev/tools/pub/pubspec

# The following section is specific to Flutter.
flutter:
  # The following line ensures that the Material Icons font is
  # included with your application, so that you can use the icons in
  # the material Icons class.
  uses-material-design: true
  generate: true
  # To add assets to your application, add an assets section, like this:
  assets:
    - images/intro_logo.png
  # An image asset can refer to one or more resolution-specific "variants", see
  # https://flutter.dev/assets-and-images/#resolution-aware.
  # For details regarding adding assets from package dependencies, see
  # https://flutter.dev/assets-and-images/#from-packages
  # To add custom fonts to your application, add a fonts section here,
  # in this "flutter" section. Each entry in this list should have a
  # "family" key with the font family name, and a "fonts" key with a
  # list giving the asset and other descriptors for the font. For
  # example:
  # fonts:
  #   - family: Schyler
  #     fonts:
  #       - asset: fonts/Schyler-Regular.ttf
  #       - asset: fonts/Schyler-Italic.ttf
  #         style: italic
  #   - family: Trajan Pro
  #     fonts:
  #       - asset: fonts/TrajanPro.ttf
  #       - asset: fonts/TrajanPro_Bold.ttf
  #         weight: 700
  #
  # For details regarding fonts from package dependencies,
  # see https://flutter.dev/custom-fonts/#from-packages

Logs

[✓] Flutter (Channel master, 1.24.0-2.0.pre.86, on Mac OS X 10.15.7 19H2 darwin-x64, locale zh-Hant-TW)
    • Flutter version 1.24.0-2.0.pre.86 at /Users/liaojianxun/flutter
    • Framework revision dde20448f7 (89 minutes ago), 2020-10-20 00:59:56 -0400
    • Engine revision ea2aea1c61
    • Dart version 2.11.0 (build 2.11.0-234.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at /Users/liaojianxun/Library/Android/sdk
    • Platform android-30, build-tools 30.0.2
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.0.1, Build version 12A7300
    • CocoaPods version 1.9.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[✓] IntelliJ IDEA Ultimate Edition (version 2020.2.3)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.50.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.15.1

[✓] Connected device (4 available)
    • Pixel 5 (mobile) • 09091FDD4000Q6 • android-arm64  • Android 11 (API 30)
    • macOS (desktop)  • macos          • darwin-x64     • Mac OS X 10.15.7 19H2 darwin-x64
    • Web Server (web) • web-server     • web-javascript • Flutter Tools
    • Chrome (web)     • chrome         • web-javascript • Google Chrome 86.0.4240.80

• No issues found!

flutter doctor -v
[✓] Flutter (Channel master, 1.24.0-2.0.pre.86, on Mac OS X 10.15.7 19H2
    darwin-x64, locale en-GB)
    • Flutter version 1.24.0-2.0.pre.86 at /Users/tahatesser/Code/flutter_master
    • Framework revision dde20448f7 (5 hours ago), 2020-10-20 00:59:56 -0400
    • Engine revision ea2aea1c61
    • Dart version 2.11.0 (build 2.11.0-234.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at /Users/tahatesser/Code/sdk
    • Platform android-30, build-tools 30.0.2
    • ANDROID_HOME = /Users/tahatesser/Code/sdk
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.0.1, Build version 12A7300
    • CocoaPods version 1.10.0.rc.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.50.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.15.1

[✓] Connected device (5 available)
    • RMX2001 (mobile)   • EUYTFEUSQSRGDA6D                     • android-arm64
      • Android 10 (API 29)
    • iPhone 11 (mobile) • 193EBBC9-CC2A-4477-B8A0-A5F12BB51400 • ios
      • com.apple.CoreSimulator.SimRuntime.iOS-14-0 (simulator)
    • macOS (desktop)    • macos                                • darwin-x64
      • Mac OS X 10.15.7 19H2 darwin-x64
    • Web Server (web)   • web-server                           • web-javascript
      • Flutter Tools
    • Chrome (web)       • chrome                               • web-javascript
      • Google Chrome 86.0.4240.80

• No issues found!

CC @shihaohong

I quickly glanced at the synthetic package generation code -- When using flutter:generate: true, it takes the existing package_config.json and .packages files and inserts a new reference to flutter_gen. Here is the line that generates and overwrites .packages and .dart_tool/package_config.json.
Could you describe the exact problems you're facing because of this modification? I'm not as familiar with how package config works, so what I see is the use of relative uris and an actual difference to the html_text path.
cc/ @jonahwilliams who wrote the synthetic package generation code.

@shihaohong IDE can't access html_text package.

Fixed via https://github.com/flutter/flutter/commit/178c87cc27cbcb28971b96ff61fafaaa8feda000


Add state code 14

Description:
Following an error in log when updating my xiaomi vacuum: "STATE not supported: Updating, state_code: 14"
Related issue (if applicable): fixes #
Pull request in home-assistant.io with documentation (if applicable): home-assistant/home-assistant.io#<home-assistant.io PR number goes here>
Example entry for configuration.yaml (if applicable):
vacuum:
  - platform: xiaomi_miio
    host: 192.168.1.XX
    token: !secret vacuum_token

Checklist:

[x] The code change is tested and works locally.
[ ] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[ ] New dependencies have been added to the REQUIREMENTS variable (example).
[ ] New dependencies are only imported inside functions that use them (example).
[ ] New or updated dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[ ] New files were added to .coveragerc.

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

State 14 means that the device is updating itself, see https://github.com/rytilahti/python-miio/blob/master/miio/vacuumcontainers.py#L68 . Therefore I'm not sure if it should be marked as idle (as the device will not likely be accepting commands at that point).


docs(tracer): Fix hl_lines

Issue #, if available:
Description of changes:
Fix line highlights in examples
Checklist

[x] Meet tenets criteria
[x] Update tests
[x] Update docs
[x] PR title follows conventional commit semantics

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@heitorlessa pity you can't use comments on the line to set the highlight.