How to get probability values from the Detectron model

Hello Harsh,
First of all, I want to say thank you very much for sharing your work about this background removal! One of my current projects is quite related to it. I have read both your articles and your code repo and they are quite helpful to me! Thanks a lot!  :)
I am using Detectron2 to do instance segmentation (human face + hair) on images and achieved 91% precision. My next step is to do background removal on those segmentations. I understand from your article that a trimap is needed, and you mentioned, " However, with a few tweaks, we can get the probability values from the Detectron model also.", I am wondering whether you can share those codes as well? I tried to track down the raw code of detectron2 and found it hard to get it out.   Thanks a lot!!
I did look at your detectron_seg.py script, but I see the outputs (using out.pred_mask as below code shows) look like still just binary instead of pixel-level probability for the segmentation.
` def get_seg_output(self,image:np.array):
out = self.model(image)['instances']
    outputs = [(out.pred_masks[i],out.pred_classes[i]) for i in  range(len(out.pred_classes)) if out.pred_classes[i]==0]

    return outputs`

You need to look inside the detectron library and grab the input before it
is applying the threshold. I did that a year back, so I don't exactly
remember where you will find that. But there certainly is a way.
On Tue, 13 Jul, 2021, 7:33 am bokelai1989, @.***> wrote:

Hello Harsh,
First of all, I want to say thank you very much for sharing your work
about this background removal. One of my current projects is quite related
to it. I have read both your articles and your code repo and they are quite
helpful to me! Thanks a lot!
I am using Detectron2 to do instance segmentation (human face + hair) on
images and achieved 91% precision. My next step is to do background removal
on those segmentations. I understand from your article
https://heartbeat.fritz.ai/changing-backgrounds-with-image-segmentation-deep-learning-code-implementation-d848648ac477
that a trimap is needed, and you mentioned, " However, with a few
tweaks, we can get the probability values from the Detectron model also.",
I am wondering whether you can share those codes as well? I tried to
track down the raw code of detectron2 and found it hard to get it out.
Thanks a lot!!
I did look at your detectron_seg.py script, but I see the outputs (using
out.pred_mask as below code shows) look like still just binary instead
of pixel-level probability for the segmentation.
` def get_seg_output(self,image:np.array):
out = self.model(image)['instances']
outputs = [(out.pred_masks[i],out.pred_classes[i]) for i in  range(len(out.pred_classes)) if out.pred_classes[i]==0]

return outputs`

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/harsh2912/Background-removal/issues/3, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AENHXF2XQT7KQS4EABRYH7LTXONG7ANCNFSM5AH7NODQ
.

Hello Harsh, thanks for your fast reply! I will try to track it down further and see then.

On Wed, 14 Jul, 2021, 5:52 am bokelai1989, @.***> wrote:

Hello Harsh, thanks for your fast reply! I will try to track it down
further and see then.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/harsh2912/Background-removal/issues/3#issuecomment-879488845,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AENHXF4UTNPC2PEB6T4WOQLTXTKE5ANCNFSM5AH7NODQ
.


Restore-DBADatabase without recovery

Is this a feature OR bug:
Bug
System Details

Operating system name and version:
Output from $PSVersionTable:

Name                           Value
----                           -----
PSVersion                      5.1.14393.953
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14393.953
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>

Output of dbatools version:

Major  Minor  Build  Revision
-----  -----  -----  --------
0      8      943    -1
0      8      942    -1
0      8      933    -1
0      8      901    -1
0      8      900    -1
0      8      692    -1
0      8      65     -1

SQL Server version for source/target
12.0.4502.0/12.0.4502.0

Steps to Reproduce
Restore-DbaDatabase -SqlServer 'Name' -path \\server\Path-MaintenanceSolutionBackup
-RestoreTime $restoretime -UseDestinationDefaultDirectories
-DatabaseName db_Powershellrestore -DestinationFilePrefix 'test_'
-OutputScriptOnly -WithReplace -verbose

Action Results
Restore without recovery
Expected Results
Restore with recover
Attached Logs or any exception errors

Use of Start-Transcript can help collection of console output and exceptions
Certain commands will generate an exception log that you can find in following directory: dir $env:USERPROFILE\Documents\dbatools-exceptions.txt
I didnt add all the verbose when it just scanned all the files :)

VERBOSE: Get-FilteredRestoreFile - 804 Files to filter
VERBOSE: Get-FilteredRestoreFile - 1 database to process
VERBOSE: Get-FilteredRestoreFile - Find Newest Full backup
VERBOSE: Get-FilteredRestoreFile - Got a Full backup, now to find diffs if they exist
VERBOSE: Get-FilteredRestoreFile - we have at least one diff so look for tlogs after the last one
VERBOSE: Get-FilteredRestoreFile - Got a Full/Diff backups, now find all Tlogs needed
VERBOSE: Get-FilteredRestoreFile - Filtered 768 down to 2
VERBOSE: Get-FilteredRestoreFile - 3 Transaction Log backups found
VERBOSE: Get-FilteredRestoreFile - Returning Results to caller
VERBOSE: Restore-DbaDatabase -  dbs to restore
VERBOSE: Restore-DbaDatabase - Starting FileSet
VERBOSE: Test-DbaLsnChain - Testing LSN Chain
VERBOSE: Test-DbaLsnChain - Passed LSN Chain checks
VERBOSE: Test-DbaRestoreVersion - RestoreVersion is 12
VERBOSE: Restore-DBFromFilteredArray - Starting
VERBOSE: RPcount = 5
VERBOSE: Restore-DBFromFilteredArray - Restoring DB_Powershellrestore backup starting at order 1 - LSN<PHONE_NUMBER>42010500145 in \sodra.com\sql-backup\Produktion\vax-vs563\DB\FULL\VAX-VS563_DB_FULL_20170226_210000.b
ak
VERBOSE: Restore-DBFromFilteredArray - Beginning Restore of DB_Powershellrestore
VERBOSE: Restore-DBFromFilteredArray - restoring to 03/26/2017 00:00:00
VERBOSE: Restore-DBFromFilteredArray - restore action = Database
VERBOSE: Restore-DBFromFilteredArray - More files to restore, NoRecovery set
VERBOSE: Restore-DBFromFilteredArray - Adding device
VERBOSE: Restore-DBFromFilteredArray - Performing restore action
RESTORE DATABASE [DB_Powershellrestore] FROM  DISK = N'\sodra.com\sql-backup\Produktion\vax-vs563\DB\FULL\VAX-VS563_DB_FULL_20170226_210000.bak' WITH  FILE = 1,  MOVE N'DBPDT' TO N'G:\Data\test_DBPDT.mdf',  M
OVE N'DBPDT_log' TO N'H:\Data\test_DBPDT_log.LDF',  NORECOVERY,  NOUNLOAD,  REPLACE,  STATS = 1,  STOPAT = N'03/26/2017 00:00:00'
VERBOSE: Restore-DBFromFilteredArray - Succeeded, Closing Server connection
VERBOSE: Restore-DBFromFilteredArray - Restoring DB_Powershellrestore backup starting at order 2 - LSN<PHONE_NUMBER>24462100167 in \share\olahallengren\DIFF\VAX-VS563_DB_DIFF_20170325_211000.b
ak
VERBOSE: Restore-DBFromFilteredArray - Beginning Restore of DB_Powershellrestore
VERBOSE: Restore-DBFromFilteredArray - restoring to 03/26/2017 00:00:00
VERBOSE: Restore-DBFromFilteredArray - restore action = Database
VERBOSE: Restore-DBFromFilteredArray - More files to restore, NoRecovery set
VERBOSE: Restore-DBFromFilteredArray - Adding device
VERBOSE: Restore-DBFromFilteredArray - Performing restore action
RESTORE DATABASE [DB_Powershellrestore] FROM  DISK = N'\share\olahallengren\DIFF\VAX-VS563_DB_DIFF_20170325_211000.bak' WITH  FILE = 1,  MOVE N'DBPDT' TO N'G:\Data\test_DBPDT.mdf',  M
OVE N'DBPDT_log' TO N'H:\Data\test_DBPDT_log.LDF',  NORECOVERY,  NOUNLOAD,  REPLACE,  STATS = 1,  STOPAT = N'03/26/2017 00:00:00'
VERBOSE: Restore-DBFromFilteredArray - Succeeded, Closing Server connection
VERBOSE: Restore-DBFromFilteredArray - Restoring DB_Powershellrestore backup starting at order<PHONE_NUMBER>23479400001 - LSN<PHONE_NUMBER>23479400001 in \share\olahallengren\LOG\VAX-VS563_DB_LOG
_20170325_220000.trn
VERBOSE: Restore-DBFromFilteredArray - Beginning Restore of DB_Powershellrestore
VERBOSE: Restore-DBFromFilteredArray - restoring to 03/26/2017 00:00:00
VERBOSE: Restore-DBFromFilteredArray - restore action = Log
VERBOSE: Restore-DBFromFilteredArray - More files to restore, NoRecovery set
VERBOSE: Restore-DBFromFilteredArray - Adding device
VERBOSE: Restore-DBFromFilteredArray - Performing restore action
RESTORE LOG [DB_Powershellrestore] FROM  DISK = N'\share\olahallengren\LOG\VAX-VS563_DB_LOG_20170325_220000.trn' WITH  FILE = 1,  MOVE N'DBPDT' TO N'G:\Data\test_DBPDT.mdf',  MOVE N'A
xxosPDT_log' TO N'H:\Data\test_DBPDT_log.LDF',  NORECOVERY,  NOUNLOAD,  STATS = 1,  STOPAT = N'03/26/2017 00:00:00'
VERBOSE: Restore-DBFromFilteredArray - Succeeded, Closing Server connection
VERBOSE: Restore-DBFromFilteredArray - Restoring DB_Powershellrestore backup starting at order<PHONE_NUMBER>29313800001 - LSN<PHONE_NUMBER>29313800001 in \share\olahallengren\LOG\VAX-VS563_DB_LOG
_20170325_230000.trn
VERBOSE: Restore-DBFromFilteredArray - Beginning Restore of DB_Powershellrestore
VERBOSE: Restore-DBFromFilteredArray - restoring to 03/26/2017 00:00:00
VERBOSE: Restore-DBFromFilteredArray - restore action = Log
VERBOSE: Restore-DBFromFilteredArray - More files to restore, NoRecovery set
VERBOSE: Restore-DBFromFilteredArray - Adding device
VERBOSE: Restore-DBFromFilteredArray - Performing restore action
RESTORE LOG [DB_Powershellrestore] FROM  DISK = N'\share\olahallengren\LOG\VAX-VS563_DB_LOG_20170325_230000.trn' WITH  FILE = 1,  MOVE N'DBPDT' TO N'G:\Data\test_DBPDT.mdf',  MOVE N'A
xxosPDT_log' TO N'H:\Data\test_DBPDT_log.LDF',  NORECOVERY,  NOUNLOAD,  STATS = 1,  STOPAT = N'03/26/2017 00:00:00'
VERBOSE: Restore-DBFromFilteredArray - Succeeded, Closing Server connection
VERBOSE: Restore-DBFromFilteredArray - Restoring DB_Powershellrestore backup starting at order<PHONE_NUMBER>35130800001 - LSN<PHONE_NUMBER>35130800001 in \share\olahallengren\LOG\VAX-VS563_DB_LOG
_20170326_000001.trn
VERBOSE: Restore-DBFromFilteredArray - Beginning Restore of DB_Powershellrestore
VERBOSE: Restore-DBFromFilteredArray - restoring to 03/26/2017 00:00:00
VERBOSE: Restore-DBFromFilteredArray - restore action = Log
VERBOSE: Restore-DBFromFilteredArray - Doing Recovery on last file
VERBOSE: Restore-DBFromFilteredArray - Adding device
VERBOSE: Restore-DBFromFilteredArray - Performing restore action
RESTORE LOG [DB_Powershellrestore] FROM  DISK = N'\share\olahallengren\LOG\VAX-VS563_DB_LOG_20170326_000001.trn' WITH  FILE = 1,  MOVE N'DBPDT' TO N'G:\Data\test_DBPDT.mdf',  MOVE N'A
xxosPDT_log' TO N'H:\Data\test_DBPDT_log.LDF',  NOUNLOAD,  STATS = 1,  STOPAT = N'03/26/2017 00:00:00'
VERBOSE: Restore-DBFromFilteredArray - Succeeded, Closing Server connection
VERBOSE: Database DB_Powershellrestore restored successfully

I get the same script (without Recovery) when I create it manualy in ssms so this might not be an issue with the restore command.

Hmm, that's interesting. The T-SQL generated is correct:
RESTORE LOG [DB_Powershellrestore] FROM DISK = N'\share\olahallengren\LOG\VAX-VS563_DB_LOG_20170326_000001.trn' WITH FILE = 1, MOVE N'DBPDT' TO N'G:\Data\test_DBPDT.mdf', MOVE N'A
xxosPDT_log' TO N'H:\Data\test_DBPDT_log.LDF', NOUNLOAD, STATS = 1, STOPAT = N'03/26/2017 00:00:00'
should do recovery (it's the default if there's no NORECOVERY in the with block).
Out of interest could you try it against a different SQL instance? Just wondering if there's some reason why this (and SSMS) aren't doing the recovery at the end of the restore.

I have run it succesfully after this on different servers. I Believe something went wrong at the server level.

@PsPsam - do you consider this issue resolved? If so, I will close it.
Thanks!

Yea it can be closed.

🤘


Wolvenkit.WolvenKit.Installer version <IP_ADDRESS>

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.2 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

@MSFTbot Dependencies

For context, see https://github.com/microsoft/winget-pkgs/discussions/88963

This package has WindowsAppRuntime.1.2 as a dependency, and this is available through the MS Store as 9P7VS8XHG9G5. But maybe MS Store packages don't work as dependencies - the logs end in this error:
2022-12-22 10:21:22.457 [CLI ] Manifest fields: Name [WindowsAppRuntime.1.2], Version [Unknown]
2022-12-22 10:21:22.491 [CLI ] Get user entitlement.
2022-12-22 10:21:22.497 [CLI ] Get device entitlement.
2022-12-22 10:21:22.854 [CLI ] Get entitlement succeeded. Server error. ProductId: 9P7VS8XHG9G5
2022-12-22 10:21:22.854 [CLI ] Terminating context: 0x8a15001e at D:\a\_work\1\s\external\pkg\src\AppInstallerCLICore\Workflows\MSStoreInstallerHandler.cpp:7a
2022-12-22 10:21:22.854 [CLI ] Terminating context: 0x8a150110 at D:\a\_work\1\s\external\pkg\src\AppInstallerCLICore\Workflows\InstallFlow.cpp:231

The Microsoft.WindowsAppRuntime.1.2 package is now available through Winget. With the dependency above it installs successfully. But when running the installer, it gives this error:

So it appears that Microsoft.DotNet.DesktopRuntime.6 is also a dependency. Currently we only support installing the first dependency:

https://github.com/microsoft/winget-cli/issues/2202

Unfortunately, Winget is only able to install the first dependency at this time. This package can be made available through Winget only after Winget can install multiple dependencies.

Unfortunately, Winget is only able to install the first dependency at this time. This package can be made available through Winget only after Winget can install multiple dependencies.

Thank you for this information. Although, it seems it’s still failing on the WindowsApp runtime. Even though it is the first package dependency in the manifest. I assumed winget would install the first dependency, but it seems this is incorrect…


Problem with Epson ET-2550

Firts i got Error 426:
25.2.2021, 23:24:05 [PrinterPlatform] Error: Received unexpected response status 426 from the printer.
at ClientRequest. (/home/hoobs/.hoobs/node_modules/@sealsystems/ipp/lib/request.js:85:12)
at Object.onceWrapper (events.js:422:26)
at ClientRequest.emit (events.js:315:20)
at HTTPParser.parserOnIncomingClient (_http_client.js:641:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
at Socket.socketOnData (_http_client.js:509:22)
at Socket.emit (events.js:315:20)
at addChunk (_stream_readable.js:309:12)
at readableAddChunk (_stream_readable.js:284:9)
at Socket.Readable.push (_stream_readable.js:223:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
After a while it changed to:
25.2.2021, 23:38:56 [PrinterPlatform] Error: Received unexpected response status 404 from the printer.
at ClientRequest. (/home/hoobs/.hoobs/node_modules/@sealsystems/ipp/lib/request.js:85:12)
at Object.onceWrapper (events.js:422:26)
at ClientRequest.emit (events.js:315:20)
at HTTPParser.parserOnIncomingClient (_http_client.js:641:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
at Socket.socketOnData (_http_client.js:509:22)
at Socket.emit (events.js:315:20)
at addChunk (_stream_readable.js:309:12)
at readableAddChunk (_stream_readable.js:284:9)
at Socket.Readable.push (_stream_readable.js:223:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
in Apple Home see only the switch and Contactsnsor , when i print no changes !
my config fiele:
[
{
"name": "Epson ET-2550",
"address": "http://<IP_ADDRESS>:631/ipp/printer",
"polling": 10,
"manufacturer": "Epson",
"model": "ET-2550",
"serialNumber": "12345"
}
]
try as well with ipp instead of http and with lower and larger polling time  No changes
mabe it is important i use Hoobs

HOOBS 3 Version
3.3.2

Node Version
14.15.1

@ash123-germany
It seems there is an protocol error
Can you try following config pls
[
{
"name": "Epson ET-2550",
"address": "ipp://<IP_ADDRESS>:631/ipp/printer",
"polling": 10,
"manufacturer": "Epson",
"model": "ET-2550",
"serialNumber": "12345"
}
]

Hi SeydX,
thanks for the fast Responce , i try this as well after reading some other comments here . And i did it today too , same responce:
27.2.2021, 19:49:35 [PrinterPlatform] Epson ET-2550: An error occured during getting state
27.2.2021, 19:49:35 [PrinterPlatform] Error: Received unexpected response status 404 from the printer.
at ClientRequest. (/home/hoobs/.hoobs/node_modules/@sealsystems/ipp/lib/request.js:85:12)
at Object.onceWrapper (events.js:422:26)
at ClientRequest.emit (events.js:315:20)
at HTTPParser.parserOnIncomingClient (_http_client.js:641:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
at Socket.socketOnData (_http_client.js:509:22)
at Socket.emit (events.js:315:20)
at addChunk (_stream_readable.js:309:12)
at readableAddChunk (_stream_readable.js:284:9)
at Socket.Readable.push (_stream_readable.js:223:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
Config:
[
{
"name": "Epson ET-2550",
"address": "ipp://<IP_ADDRESS>:631/ipp/printer",
"polling": 10,
"manufacturer": "Epson",
"model": "ET-2550",
"serialNumber": "12345"
}
]

After update same:
27.2.2021, 19:57:06 [PrinterPlatform] Epson ET-2550: An error occured during getting state
27.2.2021, 19:57:06 [PrinterPlatform] Error: Received unexpected response status 404 from the printer.
at ClientRequest. (/home/hoobs/.hoobs/node_modules/@sealsystems/ipp/lib/request.js:85:12)
at Object.onceWrapper (events.js:422:26)
at ClientRequest.emit (events.js:315:20)
at HTTPParser.parserOnIncomingClient (_http_client.js:641:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
at Socket.socketOnData (_http_client.js:509:22)
at Socket.emit (events.js:315:20)
at addChunk (_stream_readable.js:309:12)
at readableAddChunk (_stream_readable.js:284:9)
at Socket.Readable.push (_stream_readable.js:223:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
as well with http

wrong buttom

@ash123-germany
are u sure that the given ipp printer of your printer is correct?
404 means not found

Hi , yes im sure

out of Fritz Box:

BR
Arnd

Gesendet: Samstag, 27. Februar 2021 um 20:01 Uhr
Von: "Seyd"<EMAIL_ADDRESS>An: "SeydX/homebridge-printer"<EMAIL_ADDRESS>Cc: "ash123-germany"<EMAIL_ADDRESS>"Mention"<EMAIL_ADDRESS>Betreff: Re: [SeydX/homebridge-printer] Problem with Epson ET-2550 (#10)

@ash123-germany
are u sure that the given address printer of your printer is correct?
404 means not found
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

@ash123-germany
You need the address to the ipp endpoint
read this
https://github.com/SeydX/homebridge-printer/issues/1

Wheen i type in chrome or Edge http://<IP_ADDRESS>:631/   i get a side calling : http://<IP_ADDRESS>:631/PRESENTATION/HTML/TOP/INDEX.HTML
when i tery to use  http or ipp ://<IP_ADDRESS>:631//ipp/printer
i get error 404
i look in config side from Printer : '        Erlaubt is marked
IPP aktivieren
Nicht sichere Kommunikation erlauben :
Erlaubt Nicht erlaubt
Kommunikationszeitüberschreitung (s) :
300
URL(Netzwerk) :
http://<IP_ADDRESS>:631/ipp/print
https://<IP_ADDRESS>:631/ipp/print

ipp ://<IP_ADDRESS>:631/ipp/printer
How can i find out this adress on printer itself ?

@ash123-germany
read this issue
https://github.com/SeydX/homebridge-printer/issues/1
i wrote a guide how to find the address

Mhm its written in the config , this is funny

when i cancel ipp/printer ,
i get this:
27.2.2021, 20:26:14 [PrinterPlatform] Epson ET-2550: {"version":"2.0","statusCode":"successful-ok","id":3240418,"operation-attributes-tag":{"attributes-charset":"utf-8","attributes-natural-language":"en-us"},"printer-attributes-tag":{"printer-is-accepting-jobs":true,"printer-make-and-model":"EPSON ET-2550 Series","printer-state":"idle","printer-up-time":568,"queued-job-count":0,"marker-names":["Black ink","Cyan ink","Magenta ink","Yellow ink"],"marker-levels":[99,100,100,100]}}
Do you expect something like this ?

@ash123-germany
yes thats the correct result
It should work now!

27.2.2021, 20:34:42 [PrinterPlatform] Epson ET-2550: {"version":"2.0","statusCode":"successful-ok","id":89079657,"operation-attributes-tag":{"attributes-charset":"utf-8","attributes-natural-language":"en-us"},"printer-attributes-tag":{"printer-is-accepting-jobs":false,"printer-make-and-model":"EPSON ET-2550 Series","*printer-state":"processing*","printer-up-time":1067,"queued-job-count":1,"marker-names":["Black ink","Cyan ink","Magenta ink","Yellow ink"],"marker-levels":[99,100,100,100]}}
And it reconised Printer state ! an in HOME i see a swich   and a kontaktsender which is open when Printer is printing !
GREAT work
P.S.:   can you make the Switch working   ?  Printer on/off   ?

Mean i see once in Log , when i switch off the Printer:
27.2.2021, 20:38:14 [PrinterPlatform] Error: Received unexpected response status 500 from the printer.
at ClientRequest. (/home/hoobs/.hoobs/node_modules/@sealsystems/ipp/lib/request.js:85:12)
at Object.onceWrapper (events.js:422:26)
at ClientRequest.emit (events.js:315:20)
at HTTPParser.parserOnIncomingClient (_http_client.js:641:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
at Socket.socketOnData (_http_client.js:509:22)
at Socket.emit (events.js:315:20)
at addChunk (_stream_readable.js:309:12)
at readableAddChunk (_stream_readable.js:284:9)
at Socket.Readable.push (_stream_readable.js:223:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
And then 27.2.2021, 20:38:43 [PrinterPlatform] Epson ET-2550: Can not reach printer!
But in Home the switch is still on ON

Yep its already mentioned here
https://github.com/SeydX/homebridge-printer/issues/11
will fix it in the next version, closing this issue now

oK GREAT work , thanks  a lot.
One Idea at the end . i menan in real live im a tester from new products in mobile sector.
the Polling request you need are written in the normal log protocol.
I think it would be better to seperate them in a special log .
while to analyse other errors is hard to do while you have so many polling requests.
Thsi is only an idea , i do notz know if it is possible here .
And its not a criticism, you did a great usefull work.
Br
Arnd

@ash123-germany
thanks, i ll put this in my todo list :)


cask-repair "failed to change to directory of [cask]"

I am unable to reproduce this issue with other casks (at least, I haven't found one that reproduces this issue), however it happens every time I attempt to use the cask-repair script to update this cask, chirp-daily. I've tried running cask-repair -e chirp-daily but the same error is produced.

Fixed. There was a duplicate in the repos (see how it appears in both homebrew-cask and homebrew-versions, in the messages).


Update for Swift 5.

Updates source and Package.swift for Swift 5 compatibility.

To avoid breaking any old existing clients, it's probably best to have this as a<EMAIL_ADDRESS>manifest in addition to the existing one

Done – this may break Swift 3 anyway due to the code changes, so it'd probably be best to release as a new major version.

Ah yeah - what are you using this project for by the way? I'm not sure whether a release will happen as this is a Vapor 2 package that is long past it's EOL

I'm using it for a project with a Google CloudSQL database.
I just noticed https://github.com/PerfectlySoft/Perfect-PostgreSQL, which might also work for me.

Yeah the whole server-side Swift industry is moving to libraries based on NIO - this is the one that's been pitched to the SSWG

PS, we can make the changes non-breaking by using compiler directives to switch on the version of Swift

Done.

This is released as 2.2.0

Thank you!
On Mon, May 18, 2020 at 8:20 AM Tim Condon<EMAIL_ADDRESS>wrote:

This is released as 2.2.0
https://github.com/vapor-community/postgresql/releases/tag/2.2.0
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/vapor-community/postgresql/pull/79#issuecomment-630253512,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOHH7XYNH7JUH7SSHG62SZLRSFG2JANCNFSM4MZGISSA
.


NCZarr - Netcdf Support for Zarr

I am moving the conversation about NCZarr to its own issue. See Issue https://github.com/zarr-developers/zarr/issues/317 for initial part of this discussion.

Surfacing here notes on the NetCDF NCZarr implementation, thanks @DennisHeimbigner for sharing.

Also relevant here, documentation of xarray zarr encoding conventions, thanks @rabernat.

@DennisHeimbgner: It looks like Unidata's Netcdf C library can now read data with the xarray zarr encoding conventions, right?
@rabernat, should I raise an issue for xarray to also support the Unidata NcZarr conventions?

The ability to read xarray is in the main branch, and will be in the upcoming 4.8.1 release. I am shaving the yak to get our automated regression and integration test infrastructure back up and running but we hope to have 4.8.1 out shortly.

@rabernat, should I raise an issue for xarray to also support the Unidata NcZarr conventions?

I see this as very difficult. The reason is that the ncZarr conventions use files outside of the zarr hierarchy. We would probably need to implement a compatibility layer as a third-party package, similar to h5netcdf.

One thing I'll note on Xarray's convention for the Zarr is that we will likely change things in the near future to always write and expect "consolidated metadata" (see https://github.com/pydata/xarray/issues/5251). This is almost completely backwards compatible, but if NcZarr writes these consolidated metadata fields in Xarray compat mode we could load these Zarr stores much quicker in Xarray.
Consolidated metadata would probably be a nice feature for NcZarr, too, because it reduces the number of files that need to be queried for metadata down to only one. I think there was a similar intent behind the .nczgroup JSON field. Consolidated metadata is sort of a super-charged version of that.

NCZarr get similar improvement by doing lazy read of metadata objects. That is one problem with _ARRAY_DIMENSIONS -- it requires us to read all attributes even if otherwise unneeded. NCZarr avoids this by keeping the dimension names separate.
As for consolidated metadata, I assume you are NOT saying that any pure zarr container that does not contain the consolidated metadata will be unreadable by Xarray.

NCZarr get similar improvement by doing lazy read of metadata objects. That is one problem with _ARRAY_DIMENSIONS -- it requires us to read all attributes even if otherwise unneeded. NCZarr avoids this by keeping the dimension names separate.

In Xarray, we have to read nearly all the metadata eagerly to instantiate xarray.Dataset objects.

As for consolidated metadata, I assume you are NOT saying that any pure zarr container that does not contain the consolidated metadata will be unreadable by Xarray.

This is correct, you don't need to write consolidated metadata. But if you do, Xarray will be able to read the data much faster.
As for whether netCDF users would notice a difference with consolidated metadata, I guess it would depend on their use-cases. Lazy metadata reads are great, but for sure it is faster to download a single small file than to download multiple files in a way that cannot be fully parallelized, even if they add up to the same total size.

faster to download a single small file than to download multiple files

true, but we have use cases where the client code is walking a large set of netcdf files and reading a few pieces of information out of each of them and where the total metadata is large (14 megabytes). This can occur when one has a large collection of netcdf files covering some time period and each netcdf file is a time slice (or slices).
Perhaps Rich Signell would like to comment with his experience.

https://github.com/zarr-developers/zarr-specs/issues/41#issuecomment-833024978 I see this as very difficult. The reason is that the ncZarr conventions use files outside of the zarr hierarchy. We would probably need to implement a compatibility layer as a third-party package, similar to h5netcdf.

For what it's worth, I could see making some movement (June-ish?) on https://github.com/zarr-developers/zarr-specs/issues/112#issuecomment-825690209 to permit the additional files. But either way, certainly https://github.com/ome/ngff/pull/46#pullrequestreview-652899174 (related issue) would suggest hammering out a plan for this difference before another package introduces a convention.

https://github.com/zarr-developers/zarr-specs/issues/41#issuecomment-833036094 One thing I'll note on Xarray's convention for the Zarr is that we will likely change things in the near future to always write and expect "consolidated metadata" (see pydata/xarray#5251). This is almost completely backwards compatible, but if NcZarr writes these consolidated metadata fields in Xarray compat mode we could load these Zarr stores much quicker in Xarray.

Having gone through https://github.com/pydata/xarray/issues/5251 I'm slightly less worried about this then when I first read itI, but having just spent close to 2 months trying to get dimension_separator "standardized", I'd like to raise a flag that consolidated metadata is a similar gray area. It'd be nice to get it nailed down.

@DennisHeimbigner, just a quick comment that I too always use consolidated metadata when writing Zarr.   Here's a recent example of some ocean model output we are publishing, where consolidated metadata is an order of magnitude faster to open:
https://nbviewer.jupyter.org/gist/rsignell-usgs/bd10b623a90982303a4c0010e5db997d

Note that the issue for me is: for what use-cases is lazy metadata download better than consolidated metadata. The latter is better in the cases where you know that you need to access almost all of the meta-data or where the total size of the metadata is below some (currently unknown) size threshold. My speculation is that the access patterns vary all over the place and are highly domain dependent.  I infer that Rich's use case is one where all the metadata is going to be accessed.
In any case, once .zmetadata is well-defined (see Josh's previous comment) I will be adding it to nczarr. However, we will probably give the user the choice to use it or not if lazy download makes more sense for their use-case.
On the other side, it seems to me that zarr-python might profitably explore lazy download of the metadata.

Note that the issue for me is: for what use-cases is lazy metadata download better than consolidated metadata. The latter is better in the cases where you know that you need to access almost all of the meta-data or where the total size of the metadata is below some (currently unknown) size threshold. My speculation is that the access patterns vary all over the place and are highly domain dependent. I infer that Rich's use case is one where all the metadata is going to be accessed.

Agree! I'm sure there are cases where using consolidated metadata is not a great idea, though my guess is that they are relatively rare.

In any case, once .zmetadata is well-defined (see Josh's previous comment) I will be adding it to nczarr. However, we will probably give the user the choice to use it or not if lazy download makes more sense for their use-case.

Sounds great, thanks!

On the other side, it seems to me that zarr-python might profitably explore lazy download of the metadata.

As I understand it, this is already the case in Zarr-Python (if not using consolidated metadata). It's just that lazy metadata does not work for Xarray.
In particular, I think there is definitely a place for including an explicit "index" of arrays in a group that doesn't require potentially expensive directory listing. Hopefully this is already in the draft v3 spec (I haven't checked).

Can you elaborate on what you mean by this?
I think there is definitely a place for including an explicit "index" of arrays in a group that doesn't require potentially expensive directory listing.

Can you give an example?

Rich- are you using zarr-python directly or using xarray?

Can you elaborate on what you mean by this?
I think there is definitely a place for including an explicit "index" of arrays in a group that doesn't require potentially expensive directory listing.

Can you give an example?

I was thinking of exactly the information you store in .nczgroup.

Rich- are you using zarr-python directly or using xarray?

It looks like Rich is using Xarray here.

ok

I get the impression from various conversations that people do not like the nczarr convention of using extra objects like .nczarray.
As an alternative, we can store the same netcdf specific metadata as extra keys inside the various zarr standard objects. Thus most of the stuff in .nczarray could be inserted into .zarray with, say, a key named _nczarr_array.
My impression is that adding extra keys might be more acceptable than adding extra objects, but opinions welcome.

Thinking about

I get the impression from various conversations that people do not like the nczarr convention of using extra objects like .nczarray.

and

@shoyer https://github.com/zarr-developers/zarr-specs/issues/41#issuecomment-833184536 But if you do, Xarray will be able to read the data much faster.

I wonder if a .xarray file back in the day wouldn't have led to the need for consolidated metadata? i.e. are the two intertwined?

I'd definitely be for having a clear relationship between the consolidated metadata and the extra keys (https://github.com/zarr-developers/zarr-specs/issues/112#issuecomment-825690209) but I'll point out that they are guaranteed to be JSON, so we'd either need to detect JSON and ingest or have a naming scheme for detectable "extension" files.

Let me first state clearly that the technical discussion here and my comments below are not intended as a personal or institutional criticism of anyone; this discussion is a necessary part of the (sometimes slow / frustrating) community open source development process. I thank everyone for engaging with such patience and feel nothing but respect for everyone in the conversation.
The need for .zmetadata is most definitely not unique to Xarray. The ability of stores to "list" a directory was assumed in the V2 spec. However, certain types of stores (e.g. vanilla HTTP) are inherently unlistable. Other stores are slow to list, such as s3 buckets with hundreds of thousands of objects or directories on certain HPC filesystems.
.zmetadata mitigates these problems in two ways:

It explicitly enumerates all of the relevant Zarr metadata objects (.zarray, .zattrs, .zgroup) within a directory. This means that you never need to explicitly "list" the store.
It actually duplicates the contents of those files within a nested json structure. This reduces the number of distinct read operations you need to open a dataset (to one). This is important for high-latency stores, although the problem could also be mitigated with async.

The original consolidated metadata PR (https://github.com/zarr-developers/zarr-python/pull/268) contains more useful and relevant discussion.

".xarray" -- I like it. Too bad it wasn't used. Still, it is not too late.

I don't like this idea and think it would have been a mistake. Xarray is not a data / metadata standard. It's an analysis library. Xarray's data model is explicitly based on netCDF: "[xarray.Dataset] is designed as an in-memory representation of the data model from the netCDF file format." Xarray made different choice from Unidata about how to encode netCDF into Zarr, but the end goal is the same.

I get the impression from various conversations that people do not like the nczarr convention of using extra objects like .nczarray.

I am admittedly one of those people. However, I understand completely that when developing something new, certain technical decisions have to be made in order to bootstrap a project. I am familiar with this challenge, because I am part of the group (together with @jhamman and @shoyer) who made the original decision about how to encode / decode Xarray Datasets into Zarr.  Reviewing that original PR (https://github.com/pydata/xarray/pull/1528) is instructive. I have learned a lot since then, and probably would have made different choices.
At the time, we thought that the only thing we had to solve was: how to we encode named dimensions in Zarr?. We made the ad hoc choice of using the _ARRAY_DIMENSIONS key within the standard array metadata (.zattrs). A different choice would have been to create an extra metadata object (a la nczarr); however, that would have been much harder to implement because zarr-python itself would not know about such objects, so we would have had to go around zarr-python. As it is, xarray's zarr implementation sits completely on top of zarr python and never interacts directly with the lower-level stores. In general, I still think this is a good model for higher-level libraries to follow.
Hindsight is 20/20. In retrospect, a better (but slower) path, would have been to work within the Zarr community to standardize an official convention for named dimensions. Maybe it would have been an extra metadata object. I personally think it would fit well in .zarray as an optional key. Maybe it would have been _ARRAY_DIMENSIONS in .zattrs. In any case, it would have been an accepted standard.
Now with the release of nczarr, we have even more ad hoc conventions that were developed outside of the Zarr community process. For Unidata, there was never a choice to build on top of an existing base Zarr implementation, because you were writing everything from scratch to begin with, so this distinction probably seemed academic; why not just add new features to Zarr while you're at it? However, I do wonder whether it would have been best to first build a Zarr C library with feature parity with zarr-python, rather than going directly to nczarr. (Similar to the relationship between HDF5 and netCDF.)
In any case, in both cases the implementer (either Xarray devs or Unidata devs) went forward with a reasonable but unilateral technical decision about how to encode stuff on top of the base Zarr data model.  It's a cliche to post this cartoon, but...

The problem we all face now is: how do implementations support an expanding set of ad hoc conventions?  Is it too late too deprecate some of the conventions that are out there? There are petabytes of data in the wild using the xaray _ARRAY_DIMENSIONS convention, so I think that ship has sailed. And it's not very problematic, since the data are 100% compatible with vanilla Zarr. But what about nczarr? Are any data providers serving this format yet?
Going forward, I still think the best path is to add optional named dimensions to the Zarr spec, and then have downstream data models leverage this, rather than coming up with new conventions outside of the spec. Overall, it would be great to absorb the lessons of nczarr into the V3 spec process and provide a standard, community-consensus-based way to extend the spec in an open and transparent. way.

A final comment on process.
The nczarr development has occurred openly on GitHub via PRs: https://github.com/Unidata/netcdf-c/pulls?q=is%3Apr+nczarr. There was an opportunity almost a year ago for us (Zarr + Xarray python community) to engage with the nczarr decisions (e.g. https://github.com/Unidata/netcdf-c/pull/1769) and provide feedback before a release of nczarr. We missed this opportunity. It was a difficult year for everyone (to say the least! 😷), and a lot of balls got dropped. But going forward, let's try to find the bandwidth for this type of engagement and collaboration, particularly around the V3 spec.

First let me be clear that I have no problem with .zmetadata and Unidata will release  in NCZarr it as soon as it is standardized.
Second, it is still not too late to modify the NCZarr extensions since it is unlikely that it has been used for significant amounts of data.

One thing that surprises me is that the HDF5 community has not weighed in on this. Mapping HDF5 to Zarr is also going to need similar extensions as required by NetCDF-4.

One thing that surprises me is that the HDF5 community has not weighed in on this. Mapping HDF5 to Zarr is also going to need similar extensions as required by NetCDF-4.

I don't think that HDF5 / Zarr interoperability is a priority for HDFGroup. I think they have their hands full supporting their own format and products, and they have no incentive to support Zarr.
From the Zarr perspective, we have tried hard to make Zarr-python API-compatible with h5py, so users can easily swap one for another. And we provide utilities to copy to / from HDF5.

New Representation of NCZarr extended metadata
After much discussion inside Unidata, and taking account of comments here, Unidata would like to propose a new representation for the NCZarr extensions to the Zarr format.
The new NCZarr extensions are stored as standard Zarr attributes in the .zattrs object. They use a naming convention to mark those attributes as NCZarr specific. The proposed naming convention is to prefix each attribute with the string _nczarr. Unrecognized attributes prefixed with this string are ignored. But developers should be aware that Unidata will probably add new attributes with this prefix as support for NetCDF-4 is enhanced.
The value of these attributes is intended to be an arbitrary JSON object. But Zarr only supports simple values or arrays of such. To bypass this constraint, an nczarr attribute's value consists of a single string value. This value is assumed to encode an arbitrary JSON object so the string value can be parsed to obtain the actual extended metadata. As a rule, the parsed value is a dictionary. Unrecognized keys are ignored to allow for future changes.
Proposed extension Attributes
The NCZarr extension metadata is stored as follows.
Extended .zgroup Attributes
_nczarr_version
The root group stores the NCZarr version number. This is stored only in the root group.
Example:
"_nczarr_version": "2.0.0"

_nczarr_group
This contains a dictionary listing the names of variables (arrays), and subgroups contained (non-transitively) in a group. It also contains both the names and sizes of dimensions defined in that group. Note that eventually, the size field will allow some mechanism for specifying that the dimension has an "unlimited" size (in the NetCDF/HDF5 sense of unlimited).
Example:
"_nczarr_group": "{
\"dimensions\": {\"d1\": \"1\", \"d2\": \"1\",...}
\"variables\": [\"v1\", \"v2\", ...]
\"groups\": [\"g1\", \"g2\", ...]
}"

Extended .zarray Attributes
_nczarr_array
This contains a dictionary listing the dimension references as fully qualified names. Ideally, we would use the XArray "_ARRAY_DIMENSIONS" convention, but it does not support fully qualified names.
As an alternate representation, the "_ARRAY_DIMENSIONS" attribute could be extended to allow the storage of fully qualified names. As long as XArray is restricted to a single, root group, there would be no conflict. That is, a multigroup Zarr file is not XArray compatible, so adapting the contents of "_ARRAY_DIMENSIONS" presents no conflict. Using this alternative would, of course, require permission from the XArray group. It also depends on what other implementations do when they encounter "_ARRAY_DIMENSIONS" in a multi-group Zarr file.
Example:
"_nczarr_array": "{
\"dimensions\": [\"/g1/g2/d1\", \"/d2\",...]
}"

_nczarr_attrs
This contains a dictionary listing the attribute types.
Example:
"_nczarr_attrs": "{
\"types\": {\"attr1\": \"<i4\", \"attr2\": \"<i1\",...}
}"

Last Revised: 5/15/2021

The value of these attributes is intended to be an arbitrary JSON object. But Zarr only supports simple values or arrays of such.

I don't think that's the case? I agree it's a little vague in the spec, but I'm pretty sure intent is to allow for storing arbitrary JSON documents in values, e.g., I'm pretty sure this in valid in the Zarr Python client. "Simple" is not a subset of JSON, it's a description of JSON itself 😀
Assuming I'm right I would definitely suggest storing normal JSON documents. This is much more readable than JSON encoded strings.

Thanks for the info. If and when the spec clarifies, then it is worth revisiting.

This contains a dictionary listing the dimension references as fully qualified names. Ideally, we would use the XArray "_ARRAY_DIMENSIONS" convention, but it does not support fully qualified names.
As an alternate representation, the "_ARRAY_DIMENSIONS" attribute could be extended to allow the storage of fully qualified names. As long as XArray is restricted to a single, root group, there would be no conflict. That is, a multigroup Zarr file is not XArray compatible, so adapting the contents of "_ARRAY_DIMENSIONS" presents no conflict. Using this alternative would, of course, require permission from the XArray group. It also depends on what other implementations do when they encounter "_ARRAY_DIMENSIONS" in a multi-group Zarr file.

I'm not entirely opposed to fully qualified names, but I'm not sure I agree we need them.
First, to clarify: Xarray can actually load datasets from Zarr groups, but only from a single group at a time.
As I understand the netCDF4 data model, dimensions are visible to the arrays in a group and all sub-groups. Duplicate dimensions at the same level are not allowed. It is allowed to override dimensions in a sub-group. So far, this is all compatible with unqualified dimension names, with the "nearest" name taking precedence.
Are you concerned about cases where a dimension has a different size in a sub-group, but a variable in that sub-group uses a dimension of the same name from a parent group? If this is indeed possible in netCDF-C, I sincerely hope that there are no files that use this feature out in the wild. In my mind, this violates an implicit assumption of the netCDF data model, which is that uses of a dimension name among variables in a group have the same size. These files would be incompatible with Xarray and I suspect many other tools for working with netCDF data. As far as I can tell, it is not possible to create such netCDF files using netCDF4-Python.

This is incorrect. The search upwards rule only applies to the CDL representation.
It is perfectly legal to use any dimension for a variable, including one  arbitrarily
deep in some other group. So this is why FQNs are required.

This is incorrect. The search upwards rule only applies to the CDL representation.
It is perfectly legal to use any dimension for a variable, including one arbitrarily
deep in some other group. So this is why FQNs are required.

From the netCDF documentation on groups: https://www.unidata.ucar.edu/software/netcdf/docs/groups.html
"Dimensions are scoped such that they are visible to all child groups. For example, you can define a dimension in the root group, and use its dimension id when defining a variable in a sub-group."
In my opinion, this is a good constraint to enforce, for the sake of user sanity :). Even if netCDF4 doesn't enforce it, we could enforce it for NCZarr.

I was just pointed to this discussion by a helpful colleague. Regarding dimensions that are not defined in groups that are direct ancestors of the group(s) where they are used: As Dennis says, netCDF4 allows it. Moreover, NASA stores a large amount of data in the HDF5-EOS format. Some of those datasets (e.g., the OMI instrument on the Aura satellite) put the geophysical fields (e.g., ozone) in groups (e.g., sub-groups) that are cousins (not children) of the groups where the spatial/temporal dimensions are defined. I hope a solution is found that is both user/developer-friendly and ensures that NCZarr has the flexibility it needs to represent and access such datasets.
I do agree it is a poor practice to for data producers to organize data in ways that make direct ancestor searches insufficient to resolve all the metadata values.
FYI: The CF Metadata Convention formally adopted Groups as part of CF-1.8 in 2019. However, CF-compliant datasets may only use dimensions defined in direct ancestors of the groups where they are referenced. In other words, CF does not support the full generality of dimension references supported by netCDF4, such as the HDF5-EOS datasets mentioned above.

@czender thanks for joining us! Interesting to hear that non-hierarchical dimensions are in fact widely used. So it sounds like NCZarr needs this feature.
It sounds like this needs a new extension point like the proposed _nczarr_array attribute. The problem is that there are likely already Xarray datasets that have been written to Zarr groups without fully qualified names, and there is also other application code (e.g., neuroglancer) that expects the entries in _ARRAY_DIMENSIONS  to be plain names.
That said, I do still think there is value in having a "domain agnostic" way to represent dimension names on arrays, without requiring associated dimension objects. Non-geoscience libraries like Neuroglancer may not care enough to implement the full NCZarr standard, but they would be willing to handle basic dimension names.
Ideally NCZarr could be a super-set of this domain agnostic functionality, e.g., perhaps by writing  the _ARRAY_DIMENSIONS attribute even if it is redundant with _nczarr_array.

Another thought: perhaps it would make sense to name these NCZarr attributes in all capitals like _NCZARR_ARRAY (rather than lowercase). This would be a little more consistent with Xarray's _ARRAY_DIMENSIONS naming convention, and would distinguish them more clearly from typical user provided attributes.

I have been working on how to allow nczarr to interoperate with zarr numcodec JSON filter definitions. I have a blog post of a proposal. It is  a bit netcdf-c centric, but some in the zarr community might find it interesting.
https://www.unidata.ucar.edu/blogs/developer/en/entry/nczarr-support-for-zarr-filters

I've given a try at netcdf-c master with the Version 2 NCZarr Extented Meta-data, and I think it is slightly non conformant with the Zarr V2 specification
I've converted a netcdf file to the new NCZarr format with nccopy and I see in a .zarray file

{
  "zarr_format": 2,
  "shape": [
    20,
    20
  ],
  "dtype": "<i1",
  "chunks": [
    20,
    20
  ],
  "order": "C",
  "compressor": null,
  "filters": null,
  "_NCZARR_ARRAY": {
    "dimrefs": [
      "/y",
      "/x"
    ],
    "storage": "chunked"
  }
}

The presence of the _NCZARR_ARRAY key seems to contradict this point of the spec in https://zarr.readthedocs.io/en/stable/spec/v2.html#metadata : "Other keys SHOULD NOT be present within the metadata object and SHOULD be ignored by implementations."

There was a long conversation about this on the Zarr spec github.
I pointed out that the new "dimension_separator" key violated
this constraint. The consensus seemed to be that extra keys would be allowed,
but must be ignored if they are not recognized by the implementation.
I have not checked to see if that change has made it into the spec yet.

I agree, I think any reasonable implementation should ignore unrecognized
keys or files. Hopefully this will be codified in zarr v3.
On Sun, Aug 1, 2021 at 11:30 AM Dennis Heimbigner @.***>
wrote:

There was a long conversation about this on the Zarr spec github.
I pointed out that the new "dimension_separator" key violated
this constraint. The consensus seemed to be that extra keys would be
allowed,
but must be ignored if they are not recognized by the implementation.
I have not checked to see if that change has made it into the spec yet.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/zarr-developers/zarr-specs/issues/41#issuecomment-890566349,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJJFVXS6D3GDCMQPJULL4DT2WHDLANCNFSM4H5L7P7A
.

The consensus seemed to be that extra keys would be allowed, but must be ignored if they are not recognized by the implementation.

ok, thanks for the clarification

for reference: https://github.com/zarr-developers/zarr-python/pull/715#issuecomment-821094669
That didn't make it into a zarr-specs issue (neither v2 nor v3) as far as I can tell. Anyone up for shepherding that?

See the related conversation in https://github.com/pydata/xarray/issues/6374

@DennisHeimbigner does NCZarr support any filter now?

Yes although there are some complications because the code uses HDF5 filters to perform
the actual filter code, but it needs extra code to convert a Zarr codec JSON format
to the HDF5 unsigned integer parameters.
What specific filter(s) do you need?

@DennisHeimbigner Do you have documentations about how to enable and use the filter through NCZarr, and we have a new codec which is not binding to any codec yet. Do you have suggestion on how to enable it in your NCZarr?

@DennisHeimbigner @halehawk  Maybe I should jump in now ;)
I have a lossy compressor product (SPERR: https://github.com/shaomeng/SPERR) that I'm looking at paths to integrate into the Zarr format. I haven't spent too much time on it, but my understanding is that I need to make it a Zarr filter. Our immediate application of it, an ASD run of MURam, has decided to use NCZarr to  output Zarr files, so the question arises that if Zarr filters are supported by NCZarr.
I guess the most direct question to @DennisHeimbigner as the NCZarr developer is, what approach do you recommend to integrate a lossy compressor to an NCZarr output?

If the compressor is (or easily could be) written in python, then see the NumCodecs web page.
If the compressor is in C or C++, and you decide to use netcdf-c NCZarr, then you need
to build an HDF5 compressor wrapper plus the corresponding codecs API.
I have attached the relevant documentation. If this compressor is similar to some existing
compressor such as bzip2 or zstandard, then you can copy and modify the corresponding
wrapper in netcdf-c/plugins directory -- H5Zzstd.c, for example.
filters.md

That's super helpful, thanks for the pointer! One more question: can the
compression-enabled NCZarr output read by zarr tools in the python
ecosystem?
On Fri, Apr 8, 2022, 3:56 PM Dennis Heimbigner @.***>
wrote:

If the compressor is (or easily could be) written in python, then see the
NumCodecs web page.
If the compressor is in C or C++, and you decide to use netcdf-c NCZarr,
then you need
to build an HDF5 compressor wrapper plus the corresponding codecs API.
I have attached the relevant documentation. If this compressor is similar
to some existing
compressor such as bzip2 or zstandard, then you can copy and modify the
corresponding
wrapper in netcdf-c/plugins directory -- H5Zzstd.c, for example.
filters.md
https://github.com/zarr-developers/zarr-specs/files/8455495/filters.md
—
Reply to this email directly, view it on GitHub
https://github.com/zarr-developers/zarr-specs/issues/41#issuecomment-1093392829,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGG6JNSZGN6IUUZAPCW6ODVECTQTANCNFSM4H5L7P7A
.
You are receiving this because you commented.Message ID:
@.***>

Yes, IF the filters are available in NumCodecs.

Does it mean the compressor would be better to be integrated in numcodecs if it wants to be used by nczarr and zarr/xarray?
Haiying
Sent from my iPhone

On Apr 8, 2022, at 5:03 PM, Dennis Heimbigner @.***> wrote:

Yes, IF the filters are available in NumCodecs.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.

Sorry, I wasn't clear.
Suppose you use nczarr to write a Zarr file where some of its arrays apply a filter.
Then you can obviously read that file with nczarr.
However, suppose you write the array with nczarr and then want
others to read it using python-zarr. In that case, you will need to create
a NumCodecs compatible version of your filter written in python so that
the python-zarr users can read the data written by nczarr.

There are two pieces here and I am sorry I was unclear.
The first piece is the declaration of the compressor for a variable
in the Zarr metadata, This is specified in the "compressor" key for
the .zarray metadata object for the variable. The format for this
is defined by NumCodecs and generally has the form
{"id": "<compressor name>", "parameter1": <value>, ... "parametern": <value>}

So for zstd, we might have this:  {"id": "zstd", "level:": 5}
The second part is the actual code that implements the compressort.
NCZarr supports the first part so that it can read/write legal Zarr metadata.
BUT, NCZarr requires its filter code to be written in C (or C++).
More specifically, it does not support the Python compressor code implementations.
Sorry for the confusion

@Dennis Heimbigner @.***> I still need you to clarify something
here. So I looked at your H5Zzstd.c which is a HDF5 plugin for zstd and
supports numcodec zstd read/write. Then I got this idea, if Samuel's new
compressor need not get a formal HDF5 filter ID, but should add a similar
H5Zsperr.c to your netcdf-c if he need not write to HDF5/netcdf file now.
But he needs to be in numcodecs to get Zarr/numcodecs support.
On Tue, Apr 12, 2022 at 12:20 PM Dennis Heimbigner @.***>
wrote:

There are two pieces here and I am sorry I was unclear.
The first piece is the declaration of the compressor for a variable
in the Zarr metadata, This is specified in the "compressor" key for
the .zarray metadata object for the variable. The format for this
is defined by NumCodecs and generally has the form
{"id": "", "parameter1": , ... "parametern": }
So for zstd, we might have this: {"id": "zstd", "level:": 5}
The second part is the actual code that implements the compressort.
NCZarr supports the first part so that it can read/write legal Zarr
metadata.
BUT, NCZarr requires its filter code to be written in C (or C++).
More specifically, it does not support the Python compressor code
implementations.
Sorry for the confusion
—
Reply to this email directly, view it on GitHub
https://github.com/zarr-developers/zarr-specs/issues/41#issuecomment-1097044935,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACAPEFEAYJPY2GZOFJWM6ODVEW5F5ANCNFSM4H5L7P7A
.
You are receiving this because you were mentioned.Message ID:
@.***>

NCZarr requires its filter code to be written in C (or C++).

Just to clarify, did you mean that NCZarr requires its filter code to be in C AND also exposed to NCZarr as HDF5 filters? E.g., NumCodecs filters won't work.
Sorry for the back and forth in this github thread. I think this is my last try and if there's still confusion, I'll try to set up a meeting and resolve it more directly :)

I don't know the details of how codecs are defined for NCZarr, but in general you will need to provide a separate implementation of a codec for each zarr implementation that you want to support it.
Zarr-python provides a mechanism by which codecs can be registered --- numcodecs defines many codecs, and zarr-python pulls in numcodecs as a dependency, but it is actually possible to define a codec for zarr-python outside of the numcodecs package --- see for example the imagecodecs Python package.

@dennis Heimbigner @.***> I still need you to clarify something here. So I looked at your H5Zzstd.c which is a HDF5 plugin for zstd and supports numcodec zstd read/write. Then I got this idea, if Samuel's new compressor need not get a formal HDF5 filter ID, but should add a similar H5Zsperr.c

That is correct. THe HDF group reserves ids 32768 to 65535 for unregistered use.
So Samuel can pick a number in that range for his filter; later, if desired, a forml
HDF5 filter id can be assigned.

First a big :100: for the discussion, since this is exactly what we want to see happening for cross-implementation support of codecs. @shaomeng & @halehawk, don't hesitate to keep asking.
I do wonder, @DennisHeimbigner, if we don't want to establish the channel you'd like for more nczarr questions. If so, I'd say we update the start and end of this thread with that URL and close this issue.
Others may want to express an opinion, but if it's useful, we can have a no-code location like github.com/zarr-developers/nczarr for people to find a README pointing to the netcdf-c implementation's resources.
cc: @WardF

21-050r1_Zarr_Community_Standard.pdf
Adding this here for reference in the convo.

Pertinent text from @briannapagan's link above...

Beginning with NetCDF-C version 4.8.0, Unidata introduced experimental Zarr support
into the NetCDF-C library. This was accomplished via creating a new specification -
NCZarr - which is “similar to, but not identical with the Zarr Version 2 Specification.”
Specifically, NCZarr adds two additional metadata files (“.nczarray" and ".nczattr”),
which are not part of the Zarr V2 Spec. Since NCZarr stores are not fully compatible and
interoperable with Zarr V2, this community standard excludes NCZarr. Work is ongoing
to reconcile NCZarr and the architectural reasons that motivated its development with the
forthcoming Zarr V3 Specification.
Fortunately, the NetCDF-C library also supports reading / writing of data using the
simpler Named Dimension convention described in 4.1.

That information is out-of-date in a couple of ways.

the metadata files  (“.nczarray" and ".nczattr”)  are no longer used; they were replaced with special dictionary entries.
I believe the spec was changed to specify that unrecognized elements (objects and dictionary entries) should be ignored by any implementation that does not recognize them.
With #2 in effect, nczarr created files can be read by pure zarr implementations and nczarr can read pure zarr files.

Thanks for calling that out, @DennisHeimbigner. This came out of a conversation over here. https://github.com/zarr-developers/geozarr-spec/issues/22
There are very few people who have a deep enough understanding of the moving parts here to answer all the questions. It's good to hear that we basically have interoperability.
Two questions:

Do you feel like we even need to worry about the distinction right now?
Is there a current document we should be using to learn about the nuances between "pure zarr" and "nczarr"?


request headers “Accept-Language” has a higher priority than domain path "/en"

Version
nuxt-i18n: 6.15.1
nuxt: 2.13.0
Nuxt configuration
mode:

[√] universal
[] spa

Nuxt-i18n configuration
i18n: {
    locales: ['zh', 'en'],
    differentDomains: false,
    defaultLocale: 'zh',
    vueI18n: {
      fallbackLocale: 'zh',
      messages: {
        en: require('../lang/en.json'),
        zh: require('../lang/zh.json'),
      },
    },
  }

Reproduction Link

Steps to reproduce
I apologize for not being able to provide an example to reproduce this problem. But maybe you can reproduce the problem with a simple command.
This project has two languages, Chinese and English， Chinese is the default language. So, when I visit "http://localhost:3000/en", I expect that I will open the English version of the website, But in fact it is not the case. Sometimes it returns 302 and redirects to the Chinese version of the website.
curl -I http://localhost:3000/en -H 'Accept-Language:zh-CN,zh'

The above command will return 302 Found
HTTP/1.1 302 Found
Location: /
Date: Sun, 20 Sep 2020 03:41:54 GMT
Connection: keep-alive

What is Expected?
What is actually happening?
I need to add a domain to my project. This domain only provides English version services. so I need to redirect the request to "/en" in nginx.
And then I find that the browser sometimes fails to visit the website because of too many redirects, that is to say sometimes I visit "/en", nuxt-i18n will redirect to "/", finally, I found that it was caused by the request header "Accept-Language", I had to reset the header: "Accept-Language" in nginx  to solve this problem.

Have you seen and tried the detectBrowserLanguage.onlyOnRoot option?

Yes, after I discovered this problem, I noticed this option in the document. This option is false by default. I did not declare this option before. After discovering this problem, I tried to proactively declare it as false, But I don’t know if the configuration is incorrect or other problems, the problem still exists, so I just reset the header "accept-language" in nginx

You should try setting it to true

Sorry
{
    locales: ['zh', 'en'],
    differentDomains: false,
    detectBrowserLanguage: {
      onlyOnRoot: true,
    },
    defaultLocale: 'zh',
    vueI18n: {
      fallbackLocale: 'zh',
      messages: {
        en: require('../lang/en.json'),
        zh: require('../lang/zh.json'),
      },
    },
  }

I configured it like this, but it still returns "302 Found" when I request "/en"

Without onlyOnRoot it's expected that you get redirected with that CURL command.
With onlyOnRoot it should not redirect and it doesn't here.
Make sure you are using the latest nuxt-i18n version and try to create a repro on codesanbox if you can still reproduce.

I currently do not reproduce this problem in codesandbox.
Is there an option to disable detectBrowserLanguage, if I do not want to redirect under the root path, what should I do?

detectBrowserLanguage: false

Thank you for your patient help.


Clean up update_message_flags events.

All events related to update_message_flags use operation instead of op, the latter being the key used in ALL other events.So modify the update_message_flags events so that they also use op. Backward compatibility should be taken care of.

@neiljp @gnprice thoughts on changing this?  I think we'd probably have to go with sending op and operation with the latter being deprecated, and then slowly update clients with the usual API_FEATURE_LEVEL process.

That seems reasonable to me :+1: ZT moved to a unified TypedDict for Events some time ago (though python 3.6 will make this easier & better), and seeing op and operation was something I remember noticing then.

OK.  I think we can safely add op alongside operation and mark operation as deprecated.  @orientor want to do the PR?

@zulipbot claim

@timabbott This issue can be closed now, right?

Yes, thanks @orientor!


Check for presence of pom.xml before packing

When the GitHubProjectMiner has found a project, it will then call run the projectPacker on that project. By default, the projectPacker is the constructor of the JavaMavenProject class.
To solve this, the projectPacker function has been changed to verify that there is a pom.xml before actually packing the project.

Codecov Report

Merging #387 into master will decrease coverage by 0.13%.
The diff coverage is 0%.

@@             Coverage Diff              @@
##             master     #387      +/-   ##
============================================
- Coverage      71.7%   71.56%   -0.14%
  Complexity      374      374
============================================
  Files            65       65
  Lines          1562     1565       +3
  Branches        195      196       +1
============================================
  Hits           1120     1120
- Misses          360      363       +3
  Partials         82       82

Impacted Files
Coverage Δ
Complexity Δ

...ngpipeline/miner/github/GitHubProjectDownloader.kt
75.47% <ø> (ø)
10 <0> (ø)
:arrow_down:

.../miningpipeline/miner/github/GitHubProjectMiner.kt
0% <ø> (ø)
0 <0> (ø)
:arrow_down:

...src/main/kotlin/org/cafejojo/schaapi/OptionSets.kt
58.62% <0%> (-0.88%)
0 <0> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ab78147...652bd66. Read the comment docs.


Start of the week

Hello. In my country we starts week from Monday not from Sunday. Please make this available.

Is it implemented on Calendar? Can you give me an example of how?


exposed to_time_point on the test scheduler

Exposed to_time_point on the test scheduler, updated skip_until and take_until tests,
please review.
Thank you,
Grigoriy

That looks nicer to me, what do you think?
Thank you!

agreed, that's definitely better and allows to write tests without relying on implementation details.


Use merge=union for package.json

Occasionally the v3 -> v2 mergeback script for releases fails due to a merge conflict in package.json.
Specifically, when the line below "@types/node" (currently "@types/semver") is bumped in the Action by Dependabot, there is a merge conflict between v3 and v2 because node should be version 20 for v3 and version 16 for v2.
Then the call to npm version in the release script fails because the file is no longer valid JSON due to the merge conflict; so the mergeback PR is not created at all and the release conductor needs to resolve the conflict and restart the script locally.
This change adds merge=union as a git attribute for package.json so that if there is a merge conflict, both changes will be included. This will prevent parsing errors in package.json — at least it will be valid JSON — when npm version is run (the release conductor will need to manually edit the file to resolve the merge conflict, but that's already part of the autogenerated checklist that's created in the associated PR.
Merge / deployment checklist

[x] Confirm this change is backwards compatible with existing workflows.
[x] Confirm the readme has been updated if necessary.
[x] Confirm the changelog has been updated if necessary.

Ah.. I hadn't realized that npm version would then swallow one of the keys. I'll close in favor of a textual replacement fix then. Thanks both for taking a look!


Addition of constructor

Added constructor to prevent default read of Group table.
See issue #6

Thanks again, I'll cut a release to pypi this weekend.

Closes #6

Please can you update here when the release is available? thanks

@matthewhegarty sorry for the delay - it's live now

@r1b :+1: thanks


[Property Wrappers] Fix disallowing dynamic Self for property wrapper wrappedValue/projectedValue to also cover optional.

This is a follow up to https://github.com/apple/swift/pull/31344 to also cover Self?.
I left the error message the same because any use of dynamic self other than Self and Self? will be diagnosed with the Covariant 'Self' can only appear at the top level of property type diagnostic.

@swift-ci please smoke test


Remove version: Bitnami.WordPress version 6.0.2-0

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

URL: https://downloads.bitnami.com/files/stacks/wordpress/6.0.2-0/bitnami-wordpress-6.0.2-0-windows-x64-installer.exe
OK
(Automated message - build 910)

Version still available.


Add the windows-static-egl Cargo feature

Fixes #997: this allows using Glutin together with https://crates.io/crates/mozangle

Should mozangle be mentioned in the doc as a possible way to use this feature?
Should mozangle be added as in dev-dependencies and the program below added under examples/?
#[cfg(not(windows))]
fn main() {
    println!("This example only runs on Windows.");
}

#[cfg(windows)] extern crate mozangle;
#[cfg(windows)] extern crate glutin;

#[cfg(windows)]
fn main() {
    use glutin::GlContext;
    use glutin::os::GlContextExt;
    use glutin::os::windows::RawHandle;
    use mozangle::egl::get_proc_address;
    use mozangle::gles::ffi as gl;

    let mut events_loop = glutin::EventsLoop::new();
    let window = glutin::WindowBuilder::new();
    let context = glutin::ContextBuilder::new()
        // Requesting OpenGlEs is required to make glutin try to use EGL.
        .with_gl(glutin::GlRequest::Specific(glutin::Api::OpenGlEs, (2, 0)));
    let gl_window = glutin::GlWindow::new(window, context, &events_loop).unwrap();

    let handle = unsafe {
        gl_window.context().raw_handle()
    };
    match handle {
        RawHandle::Egl(_) => {},
        RawHandle::Wgl(_) => panic!("Expected an EGL context, found a WGL one.")
    }

    assert_ne!(gl_window.get_proc_address("glClearColor"), 0 as _);
    assert_eq!(gl_window.get_proc_address("glClearColor"), get_proc_address("glClearColor") as _);

    unsafe {
        gl_window.make_current().unwrap();
        gl::ClearColor(0., 0.5, 0., 1.);
    }

    events_loop.run_forever(|event| {
        match event {
            glutin::Event::WindowEvent { event, .. } => match event {
                glutin::WindowEvent::Closed => return glutin::ControlFlow::Break,
                glutin::WindowEvent::Resized(w, h) => gl_window.resize(w, h),
                _ => (),
            },
            _ => ()
        }

        unsafe {
            gl::Clear(gl::COLOR_BUFFER_BIT);
        }

        gl_window.swap_buffers().unwrap();
        glutin::ControlFlow::Continue
    });
}

The example program in the previous comment is subject to https://github.com/tomaka/winit/issues/415. (Maybe Clear + swap_buffers is slow in debug mode? And re-rendering on non-resize events is unnecessary.)

I'm not fan of this feature but I guess I will merge anyway.
Please solve the conflict, and also please remove the example because for such a niche feature I don't think an example really makes sense.

I’m not really a fan of it either. I think it’s preferable for WebRender/Wrench to use winit + EGL instead of this.


terminal api CORS

As a developer, I want to send cross origin requests to terminal api endpoint when developing on my local machine.
On production, cors should be disabled.

Fulfilled by https://github.com/openh2labs/elastic-erga/pull/68


Add Xcode 14 beta (not RC)

Tool name
Xcode 14 beta
Tool license
none
Add or update?

[X] Add
[ ] Update

Desired version
latest beta (not RC)
Approximate size
12.6GB
Brief description of tool
It is the latest beta of Xcode.  The Xcode 14 release candidate doesn't include the macOS 13 SDK, but the latest beta version does.
URL for tool's homepage
https://developer.apple.com/download/applications/
Provide a basic test case to validate the tool's functionality.
No response
Platforms where you need the tool

[X] Azure DevOps
[ ] GitHub Actions

Virtual environments where you need the tool

[ ] Ubuntu 18.04
[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] macOS 10.15
[ ] macOS 11
[X] macOS 12
[ ] Windows Server 2019
[ ] Windows Server 2022

Can this tool be installed during the build?
no

Tool installation time in runtime
No response
Are you willing to submit a PR?
No response

@roballie Hi!
We have already done this in the scope of https://github.com/actions/runner-images/pull/6204
The new image will be deployed in the next two days.

@roballie @Brett-Best the new image with both Xcodes has been deployed.
I'm going to close the issue.
Thank you!


Create Swagger Object and Operation tags during conversion

Operations don't have tags when doing a conversion. This results in all operations being lumped into the "default" resource bucket in the swagger-ui for Swagger 2.0 instead of being nicely categorized like the swagger-ui for Swagger 1.2.
Basically, use a sanitized #/apis/*/path in the 1.2 schema as the operation tag and #/apis/*/description in the 1.2 schema as the Swagger Object tag description.

I don't think description can directly mapped to tags. @webron what do you think?

What we (kinda) do with swagger-core is take the path from the Resource Listing (such as http://petstore.swagger.io/api/api-docs) and use that as the tag for the converted operations (that's a 1:1 conversion). The description can indeed be mapped to the top level tags.

I have noticed that in the current Swagger UI version, tags for 1.2 are formed from each resource's resourcePath instead of resource_listing's path. I have raised a separate issue reg. this in swagger-ui. Meanwhile, if we want to keep 1-on-1 relationship between v1.2 and v2.0 then v1.2's resourcePath should be converted to v2.0's tags imo.

@analogue @prat0318  After 8918e59 commit, this lib try to use resourcePath for tag names and path as fallback. It also always correctly match description from Resource Listing.
It will be part of next major release, be aware that library API was changed.

@analogue @prat0318 1.0.0 release brought feature described in my previous comment.
I'm closing issue, but feel free to reopen in case if you expect different behaviour.


gpflow can not be imported

Hi,
I'm trying to install GPflow in my macOS Monterey. It's installed successfully but it always causing the dead kernel when I import the gpflow. ''' import gpflow'''

Could you help me see how to resolve this issue?
Thank you!

I have the same problem in Mac on M1 chip, but I don't know whether this problem has been solved or not.

Can you start by installing only tensorflow and tensorflow_probability (not yet gpflow!), and confirm whether that works for you (and what versions of tensorflow and tensorflow_probability you end up with)?

I'm closing this as I've come across lots of issues of people who struggle to get tensorflow to work on Mac M1, and I don't think this is gpflow-specific...


Extract maintain shell API externally

Hej there
I recently extracted the shell API into its own repo at ysndr/shell-rs for easier usage in my own projects.
I wonder if it's desired to maintain cargo's shell API externally as well in this repo as part of the rust-lang-nursery to offer the advantages of a multiplatform, colored, customizable stdoutwrapper to the whoel community.
Please excuse if this is the wrong place to ask for this I just did'nt know a better place.

I think this would be a great crate to split out! I'd prefer it live in this repository so we can make changes if necessary, but we could certainly publish it all and take prs for docs and such

I think as the project is strongly coupled with cargo by nature including it in the repo is certainly a good idea. Im just worried that new features might get ignored because of the lack of relevance in cargo which would be a point for putting it into it's own repo, to loosen the ties to cargo a bit.

True! Perhaps the API could be extracted in this repo, and if it goes well after a few weeks/months/etc we can extract it to its own repo?

Yes maybe its best to build a stable foundation in this repo.
Yet I think it should get its own repo as soon as it's considered kinda final (API-wise and usage wise inside cargo). Otherwise having to keep track with cargo just to contribute to shell could keep people from doing so.
It could also be considered too close to cargo to be used alone.
tldr; I think you right, stabilizing it's API usage and documentation/testing(?) should be done here be its main users. Let's give it some more room for itself when that is done

Closing in favor of #5088


golint ignores variable clobbering

Hi,
This is potentially unsafe code that is overwriting the variables:
import (
    "encoding/json"
    "fmt"
    "runtime"
)

type ver struct {
	App  string
	Os   string
}

var version ver

func main() {
    version.App = "1.0"
    version.Os = runtime.GOOS

    json, _ := json.Marshal(&version) // [1]
    fmt.Println(string(json))
}

[1] – At this point global json variable is overwritten and I cannot use json.* functions anymore.
It would be very convenient if golint detected and reported such situations.

This isn't a job for golint. It's not a matter of style, but correctness. It's not unsafe either; you'll simply get a compile error if you were trying to access the json package, at which point you'll fix it by renaming the var.


jsc.suchthat doesn't work with pairs

The following test
    it("shrinks a pair so predicate is true for all shrinked values still", function () {
      var smallAndBigPair = jsc.suchthat(jsc.pair(jsc.nat()), function (pair) {
        return pair[0] <= pair[1];
      });

      checkShrink([0, 0], jsc.forall(smallAndBigPair, function (pair) {
        return pair[0] < pair[1];
      }), 100);
    });

fails with:

shrink suchthat shrinks a pair so predicate is true for all shrinked values still:
TypeError: Object nil has no method 'filter'
at shrink (lib/suchthat.js:43:28)
at lib/shrink.js:28:12
at Object.curriedNInstance [as curried2] (lib/utils.js:102:14)
at Object.shrinkTuple [as tuple] (lib/shrink.js:147:16)
at shrinkResult (lib/jsverify.js:126:25)
at lib/jsverify.js:208:14
at Object.bind (lib/functor.js:55:12)
at test (lib/jsverify.js:187:20)
at lib/jsverify.js:214:13
at loop (lib/jsverify.js:276:13)
at Object.check (lib/jsverify.js:291:22)
at checkShrink (test/shrink.js:15:17)
at Context. (test/shrink.js:95:7)

This issue should be fixed (at least the test case passes). @cipix2000, could you verify?

Thank you @phadej, it seems to be fixed.
I really like what you are doing with jsverify ! keep the good work.

Thanks to you for spotting the bug!


Autocompletions carry over after pressing space

Autocompletions should be reset or refreshed after a user presses space

Looks like the same issue as #414 - but not specific to YCM - basically all Vim-style completion is affected. This would be really nice to fix because this feels really broken for some using Vim autocompletion...
It looks like we might need to grab the popupmenu_hide event from Vim and close in that case.

This should be a quick fix, I'm just trying to think if there's any weird cases where neovim says to close completions even though oni itself has completions... I think neovim is smarter than us on completions mostly though, so I'll just throw a few lines into the AutoCompletion Service and test it out to see if it's mostly good

Yep, it looks like at the very least we'd want to hook the popupmenu_hide event around where we pick up the popupmenu_show event. There could definitely be some corner cases since the language-service powered completion gets fired a bit differently from the neovim-powered integration, worth testing around a bit to see if there is any badness. I think the worst case would be the completions would be overwritten by one strategy, if, for example, neovim-completion kicks in after the language service completion.

Alright, at least in my generic testing it should be good to go. We also need to stop showing a completion that is the same as the word

Alright, at least in my generic testing it should be good to go.

Awesome! Look forward to picking up this PR.

We also need to stop showing a completion that is the same as the word, which I'm not sure how to do

I can help with this piece. We do this for TypeScript / LSP already - I was thinking that handling popupmenu_hide might handle this, but there might need to be some special handling in Oni's completion layer.

Fixed by #531 .


THIS IS NOT AN ISSUE!!

This is not an issue BUT I SERIOUSLY LOVE THIS APP!! THANK YOU SOOOO MUCH FOR MAKING IT!!!!

I'm closing this now :)


Fix example managed policy file

"KeyPermissions" has always been a 2-layered hash, but it looks like I made a mistake when generating the sample file. The previous format is rejected by Chrome as not matching the expected schema.
r? @carl-stripe
I couldn't find a direct citation for the format of this key, but https://www.chromium.org/administrators/policy-list-3#KeyPermissions refers it indirectly. Also I tested it today on my Chromebook.

👍


occamy: Transfer hartid via user signal

This PR

adds a user width converter to solder,
increases the user width of the narrow xbar from 1 to 9,
adds user width converters between the narrow and wide xbar,
sends hartid + 1 of all cores via the AXI user signal with each data request (user = 0 is invalid).

ToDo:

[x] Test
[x] Fix linting
[x] Configure amo adapter to use user signal (requires #397)

Ready to merge (if CI passes)


ch. 2: missing inplace=True in drop/dropna?

In Ch.  2 section on Data Cleaning,
housing.dropna(subset=["total_bedrooms"])    # option 1
housing.drop("total_bedrooms", axis=1)       # option 2
seems to need inplace=True? @ageron

(I see now, option 1 and option 2 are meant to be no-ops)

Hi @notestaff ,
Thanks for your feedback. Indeed, I just wanted to show various options, but I only actually applied one of them.
Closing this issue, but feel free to reopen if it's still unclear.

@ageron The confusing part is that two of the options shown can't be used as written, because as written they're no-ops.  Maybe, write each option functional-style (housing = housing.dropna(subset=["total_bedrooms"])) rather than in-place, then the last option still prevails but they can all be used verbatim?
Thanks for your excellent book.

Hi @ageron, @notestaff,
I'm interested in understanding this better. @notestaff provided that I'm fully getting your point, imo the way they are written poses no problems as the implicit inplace=False returns a copy of housing.
On the other end, housing["total_bedrooms"].fillna(median, inplace=True) is giving me a SettingWithCopyWarning due to chained indexing, that could be possibly avoided by writing housing["total_bedrooms"] = housing["total_bedrooms"].fillna(median) (though I do understand that this does not really affect what was intended to be shown here).
Thank you for the amazing book and for your answer.